Google has introduced Google-Agent, a new entity appearing in server logs, to differentiate between traditional search crawling (like Googlebot) and AI-driven content fetching triggered by user interactions. Unlike Googlebot which proactively crawls and indexes the web, Google-Agent operates reactively, only fetching content in direct response to user prompts within Google AI products. A key distinction is that Google-Agent ignores `robots.txt` directives, behaving more like a standard web browser due to its user-initiated nature. This shift necessitates that developers adapt their infrastructure to identify and manage Google-Agent traffic correctly, focusing on real-time request management rather than traditional crawl budgets.
Real-world data from MERJ and Vercel examines patterns from top AI crawlers, showing significant traffic volumes and specific behaviors, especially with JavaScript rendering and content type priorities.
