Google is planning to expand its documentation regarding unsupported robots.txt rules by analyzing real-world data from the HTTP Archive. Rather than adding directives arbitrarily, the team aims to identify and document the top 10 to 15 most commonly used unsupported tags found in the wild. Additionally, Google may broaden its tolerance for common misspellings of the disallow directive.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
Key points:
- Use of HTTP Archive data via BigQuery to identify prevalent unsupported rules.
- Potential expansion of documentation to include frequently used but ignored directives.
- Possible increase in typo tolerance for the disallow command.
- Recommendation for webmasters to audit robots.txt files for ineffective directives.
