Google is planning to expand its documentation regarding unsupported robots.txt rules by analyzing real-world data from the HTTP Archive. Rather than adding directives arbitrarily, the team aims to identify and document the top 10 to 15 most commonly used unsupported tags found in the wild. Additionally, Google may broaden its tolerance for common misspellings of the disallow directive.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
Google has introduced Google-Agent, a new entity appearing in server logs, to differentiate between traditional search crawling (like Googlebot) and AI-driven content fetching triggered by user interactions. Unlike Googlebot which proactively crawls and indexes the web, Google-Agent operates reactively, only fetching content in direct response to user prompts within Google AI products. A key distinction is that Google-Agent ignores `robots.txt` directives, behaving more like a standard web browser due to its user-initiated nature. This shift necessitates that developers adapt their infrastructure to identify and manage Google-Agent traffic correctly, focusing on real-time request management rather than traditional crawl budgets.
This article provides a verified list of AI crawlers (GPTBot, ClaudeBot, Gemini, etc.) with user-agent strings, crawl rates, and IP verification information to help manage access and maintain inclusion in AI discovery.
A new protocol is emerging to give site owners control over how AI companies use their content, potentially integrated into robots.txt. The IETF AI Preferences Working Group is defining standardized rules for AI access and usage.
Perplexity defends its AI assistants against Cloudflare’s claims, arguing that they are not web crawlers but user-triggered agents.
Google’s John Mueller downplayed the usefulness of LLMs.txt, comparing it to the keywords meta tag, as AI bots aren’t currently checking for the file and it opens potential for cloaking.
The crawl-delay directive is an unofficial directive in robots.txt meant to communicate to crawlers to slow down crawling to not overload the web server. However, support for this directive varies among search engines.
