0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag
The crawl-delay directive is an unofficial directive in robots.txt meant to communicate to crawlers to slow down crawling to not overload the web server. However, support for this directive varies among search engines.
First / Previous / Next / Last / Page 1 of 0