Google's web.dev guidance now advises developers to treat AI agents as a distinct audience alongside human visitors. As more users delegate goal-oriented tasks to AI, websites with complex hover states or shifting layouts may become functionally broken for these automated entities. The guide highlights that optimization for agents aligns closely with existing accessibility and semantic HTML best practices, making sites better for both humans and machines.
* Treating agents as a distinct visitor type
* How agents interpret websites via screenshots, raw HTML, and the accessibility tree
* Recommendations for using semantic HTML elements and maintaining stable layouts
* Introduction to WebMCP, a proposed web standard for agent-website interaction
* Treating agents as a distinct visitor type
* How agents interpret websites via screenshots, raw HTML, and the accessibility tree
* Recommendations for using semantic HTML elements and maintaining stable layouts
* Introduction to WebMCP, a proposed web standard for agent-website interaction
Google is planning to expand its documentation regarding unsupported robots.txt rules by analyzing real-world data from the HTTP Archive. Rather than adding directives arbitrarily, the team aims to identify and document the top 10 to 15 most commonly used unsupported tags found in the wild. Additionally, Google may broaden its tolerance for common misspellings of the disallow directive.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
This post demonstrates how to use Cloudflare's Browser Rendering to easily crawl entire websites, even those with complex JavaScript. It simplifies web crawling by rendering pages with a single API call, bypassing the need for headless browsers and enabling efficient data extraction for tasks like SEO monitoring and content archiving.
Google has removed the "design for accessibility" section from within the Understand the JavaScript SEO basics documentation.
Google said this was removed because the information was "out of date and not as helpful as it used to be."
The old text advised that using JavaScript for page content "may be hard for Google to see," but Google now states this hasn't been true for many years.
While Google Search can handle JavaScript well, it's still important to double-check what Google Search sees using the URL inspection tool in Google Search Console.
Google said this was removed because the information was "out of date and not as helpful as it used to be."
The old text advised that using JavaScript for page content "may be hard for Google to see," but Google now states this hasn't been true for many years.
While Google Search can handle JavaScript well, it's still important to double-check what Google Search sees using the URL inspection tool in Google Search Console.
Cloudflare launched Markdown for Agents, converting HTML pages to markdown automatically when AI crawlers request it through content negotiation. This feature is available in beta at no additional cost for eligible paid plans.
This article provides a verified list of AI crawlers (GPTBot, ClaudeBot, Gemini, etc.) with user-agent strings, crawl rates, and IP verification information to help manage access and maintain inclusion in AI discovery.
A deep dive into the unseen systems behind Google Search – uncovering live experiments, entity-based infrastructure, AI agents, and more. The article details findings from a research project into Google's internal workings, including a list of nearly 1,200 experiments, the importance of entities and the Knowledge Graph, the development of AI agents, and how Google profiles users.
Here's a summary of the most interesting facts from the Search Engine Land article, presented as bullet points:
* **Extensive Experimentation:** Google is running approximately 1,200 experiments within its search system, with over 800 currently active as of June 2025. This highlights a continuous, iterative approach to search improvement.
* **Key Systems Still Active:** Systems previously revealed in leaks (Mustang, Twiddlers, QRewrite, Tangram, QUS) remain central to Google Search.
* **AI Agent Focus:** Google is developing a "constellation" of over 90 specialized AI agents (e.g., MedExplainer, Travel Agent) rather than a single all-purpose assistant, all under the "Project Magi" umbrella.
* **Knowledge Graph as Central Nervous System:** The Knowledge Graph isn’t just a side panel feature; it’s the core infrastructure powering many Google services, with a focus on data verification using a layered namespace hierarchy (kc, ss, hw).
* **Ghost Entities:** Google utilizes "ghost entities" – temporary, unanchored entities – to react quickly to emerging events and trends.
* **User Embedding (Nephesh):** Google creates mathematical embeddings representing user preferences and behaviors across all its products, influencing personalization.
* **Real-time Query Scoring:** Google employs a complex real-time scoring system for query terms, factoring in various elements like term placement and entity recognition.
* **Specialized Embeddings:** Beyond the main Knowledge Graph, Google uses specialized embeddings for verticals (shopping, travel) and temporal data.
* **AI Mode UI Changes:** Google is experimenting with integrating AI Mode into various UI elements, including the search bar and "I'm Feeling Lucky" button.
* **Focus on Entity Validation:** The article emphasizes the importance of brands establishing themselves as validated entities within Google’s Knowledge Graph for improved visibility.
Guide
Here's a summary of the most interesting facts from the Search Engine Land article, presented as bullet points:
* **Extensive Experimentation:** Google is running approximately 1,200 experiments within its search system, with over 800 currently active as of June 2025. This highlights a continuous, iterative approach to search improvement.
* **Key Systems Still Active:** Systems previously revealed in leaks (Mustang, Twiddlers, QRewrite, Tangram, QUS) remain central to Google Search.
* **AI Agent Focus:** Google is developing a "constellation" of over 90 specialized AI agents (e.g., MedExplainer, Travel Agent) rather than a single all-purpose assistant, all under the "Project Magi" umbrella.
* **Knowledge Graph as Central Nervous System:** The Knowledge Graph isn’t just a side panel feature; it’s the core infrastructure powering many Google services, with a focus on data verification using a layered namespace hierarchy (kc, ss, hw).
* **Ghost Entities:** Google utilizes "ghost entities" – temporary, unanchored entities – to react quickly to emerging events and trends.
* **User Embedding (Nephesh):** Google creates mathematical embeddings representing user preferences and behaviors across all its products, influencing personalization.
* **Real-time Query Scoring:** Google employs a complex real-time scoring system for query terms, factoring in various elements like term placement and entity recognition.
* **Specialized Embeddings:** Beyond the main Knowledge Graph, Google uses specialized embeddings for verticals (shopping, travel) and temporal data.
* **AI Mode UI Changes:** Google is experimenting with integrating AI Mode into various UI elements, including the search bar and "I'm Feeling Lucky" button.
* **Focus on Entity Validation:** The article emphasizes the importance of brands establishing themselves as validated entities within Google’s Knowledge Graph for improved visibility.
Guide
Google's new multi-vector retrieval algorithm (MUVERA) improves search speed and performs better on complex queries. It addresses limitations of previous multi-vector systems like ColBERT and could be a successor to Google’s RankEmbed technology. SEOs should focus on aligning with query context and intent rather than exact keyword matching.
Unlock the power of 301 redirects at scale LLMs to enhance user experience and optimize your website's SEO strategy.
Google’s John Mueller downplayed the usefulness of LLMs.txt, comparing it to the keywords meta tag, as AI bots aren’t currently checking for the file and it opens potential for cloaking.
