The crawl-delay directive is an unofficial directive in robots.txt meant to communicate to crawlers to slow down crawling to not overload the web server. However, support for this directive varies among search engines.
0 bookmark(s) - Sort by: Date ↓ / Title /
First / Previous / Next / Last
/ Page 1 of 0 