klotz: webmaster* + crawler*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

  1. The article discusses the author's experience with Amazon's FriendlyCrawler, which overloaded the author's website by crawling at a very high interval. The author criticizes the crawler's disregard for robots.txt and provides solutions for blocking such traffic using CloudFlare.
    2024-10-16 Tags: , , , by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0 SemanticScuttle - klotz.me: Tags: webmaster + crawler

About - Propulsed by SemanticScuttle