Max

What about Yelp?

Stats from OpenRobotsTxt.org which says 600.5m hostnames scanned, 36,841 User-Agents found, and last updated on 15th May 2025.

So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!

openrobotstxt.org

Thank you :-)

Example:
<link href="https://www.example.mx/" rel="alternate" hreflang="es-mx" data-country="Mexico">

Hi @johnmu.com we'd like to extend our HREFlang integration to include data attributes in the source code. Can Google handle the possibility of additional information in the HTML element with hreflang links that Googlebot can simply ignore?

thank you very much, that helped me! :-)

Google crawls millions of unnecessary URLs, which only costs both sides unnecessary time and money, or rather, crawl budget. So, does that mean the removal tool doesn't treat the block in robots.txt after 6 months as a deletion and like a 404 or 410 status code?

Now, Google Search Console says the URLs are indexable bc the bot is stuck on the robots.txt rule & can't see the noindex. Can I delete all the affected URLs now using your removal tool, and after 6 months, Google will simply abide by the robots.txt + no longer crawl the page area unnecessarily?

Hi @johnmu.com , I've blocked a domain directory in my robots.txt, which Google is crawling unnecessarily because it's my site search and everything is set to noindex.

I'll send it to you.

More and more reporting the same crawling problem on the same day:

support.google.com/webmasters/t...
support.google.com/webmasters/t...
support.google.com/webmasters/t...
support.google.com/webmasters/t...
www.webmasterworld.com/google/51181...

@johnmu.com or @searchliaison.bsky.social can you please take a look at your logs to see what is causing the Googlebot to hang up or get stuck? Unfortunately, we cannot reproduce it without your help and we have been researching the problem for three weeks.

Google Search Console crawling stats

As @tentaclequing.bsky.social and @rustybrick.com has already reported, there are increasing problems with the Googlebot, which thinks a page is very slow and then dramatically reduces crawling.

Looking for some SEO peeps to follow on Bluesky?

Here's a Starter Pack for you: go.bsky.app/NK1jwq3

Happy blogging birthday 🎉🎈🤩

Google’s Index Size Revealed: 400 Billion Docs It’s a big index, but faces growing pains Google rarely discusses the size of its web index—at least publicly. What exactly is Google’s index? Simply put, it’s a digital record of web pages an...

Great article from @cyrusshepard.bsky.social about the true Google index size:

zyppy.com/seo/google-i...

Geht, der Kreuzberg wird dann zum ersten 5000er in Berlin.

Can't see any Fehler.