An open source project called Scrapling is gaining traction with AI agent users who want their bots to scrape sites without permission, and is being used to bypass anti-bot systems like Cloudflare Turnstile. Cloudflare is actively working to counter these efforts.
Cloudflare launched Markdown for Agents, converting HTML pages to markdown automatically when AI crawlers request it through content negotiation. This feature is available in beta at no additional cost for eligible paid plans.
