Google has introduced Google-Agent, a new entity appearing in server logs, to differentiate between traditional search crawling (like Googlebot) and AI-driven content fetching triggered by user interactions. Unlike Googlebot which proactively crawls and indexes the web, Google-Agent operates reactively, only fetching content in direct response to user prompts within Google AI products. A key distinction is that Google-Agent ignores `robots.txt` directives, behaving more like a standard web browser due to its user-initiated nature. This shift necessitates that developers adapt their infrastructure to identify and manage Google-Agent traffic correctly, focusing on real-time request management rather than traditional crawl budgets.
discrawl mirrors Discord guild data into a local SQLite database, allowing you to search, inspect, and query server history independently of Discord. It’s a bot-token crawler – no user-token hacks – and keeps your data local. It discovers accessible guilds, syncs channels, threads, members, and message history, maintains FTS5 search indexes for fast text search (including small attachments), records mentions, and tails Gateway events for live updates with repair syncs. It provides read-only SQL access for analysis and supports multi-guild schemas with a simple single-guild default. Search defaults to all guilds, while sync and tail default to a configured default guild or fan out to all discovered guilds if none is set.
An open source web crawler that searches the internet. It's a minimal, real-time web search CLI that searches the internet for you. Enter a query and get search results as JSON (title, url, published_date), sorted by recency.
