--- title: Bots on websites url: https://ftrlg.com/en/wiki/web-bots/ lang: en published: 2026-10-28 version: 1 summary: Over half of web traffic is programs. Who they are: search engines, AI crawlers for training and search, user-initiated fetchers, SEO bots, vulnerability scanners and fakes, and what they really do. cite_as: "future.log, «Bots on websites», https://ftrlg.com/en/wiki/web-bots/" license: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ --- # Bots on websites **Bots on websites** are programs that request pages on their own: search engine robots, data crawlers for AI, assistants that open a page at a person's request, analytics services and vulnerability scanners. By Imperva's estimate, in 2024 automated traffic exceeded human traffic for the first time in a decade, at 51% of all web traffic, and in 2025 it rose to 53%.[^1][^2] Bots that introduce themselves honestly are easy to recognize by their user agent and to verify by their address. The rest are harder: crawlers pass themselves off as browsers, scanners hunt for other people's passwords, and fakes call themselves Googlebot. There is one rule of conduct for all of them, the robots.txt file, but it is a request, not a lock.[^3][^4] ## History - **17.12.2024** Vercel and MERJ report that over one month GPTBot made 569 million requests across their network and ClaudeBot 370 million; none of the major AI crawlers other than Googlebot and Applebot executes JavaScript.[^5] - **15.04.2025** Imperva: automated traffic in 2024 made up 51% of all web traffic, and malicious bots 37%.[^1] - **01.07.2025** Cloudflare, by its own account the first infrastructure company to do so, begins to offer new sites, by default, the option of blocking AI crawlers, and launches a beta of payment for crawling (pay per crawl).[^6] - **04.08.2025** Cloudflare accuses Perplexity of stealth crawling: a blocked bot kept coming back disguised as an ordinary Chrome browser, switching addresses and networks, on tens of thousands of domains, with millions of requests a day.[^7] - **29.08.2025** Cloudflare: almost 80% of AI bot activity is data collection for training; in July, [Anthropic](https://ftrlg.com/en/models/anthropic/)'s crawlers loaded about 38,000 pages for every visitor they sent to a site.[^8] - **15.12.2025** Cloudflare's review of 2025: AI bots account for an average of 4.2% of HTML requests, and Googlebot, which collects data for both search and AI, for 4.5%; crawling at users' request grew more than 15-fold.[^9] - **26.03.2026** The Wikimedia Foundation reports (in figures updated in April) that it blocks or throttles about 30% of automated requests from crawlers that do not follow its rules, and those crawlers have moved to “residential proxies”: other people's home and mobile connections.[^10] - **29.04.2026** Imperva: bots account for more than 53% of all web traffic.[^2] ## Types of bots **Search.** Googlebot, Applebot and Bingbot build indexes for search. Their authenticity can be checked: Google advises running a reverse DNS lookup on the address, making sure the name ends in googlebot.com, google.com or googleusercontent.com, and confirming with a forward lookup that the name points back to the same address. Apple's Applebot powers Spotlight, Siri and Safari, but its data may also go into training Apple's models.[^11][^12] **AI training.** GPTBot from [OpenAI](https://ftrlg.com/en/models/openai/), ClaudeBot from Anthropic, Meta-ExternalAgent from [Meta](https://ftrlg.com/en/models/meta/), Amazonbot from Amazon and CCBot from Common Crawl collect the texts that models are trained on. They bring the site almost no visitors: hence the gap between pages loaded and click-throughs that Cloudflare measures.[^3][^4][^13][^14][^8] **AI search.** OAI-SearchBot, Claude-SearchBot, PerplexityBot and newer services (ShapBot from Parallel Web Systems, the company of former Twitter CEO Parag Agrawal, the Exa crawler and KeenableBot) build the indexes that AI assistants and agents answer from. Keenable was founded by Andrey Styskin, former head of search, AI and cloud at [Yandex](https://ftrlg.com/en/models/yandex/), and Matthias Petri; the company says its index holds more than 100 billion documents.[^3][^15][^31][^16][^17] **User-initiated.** ChatGPT-User, Claude-User and Perplexity-User open a page when a user has asked about it. OpenAI writes that because such an action is initiated by a person, “robots.txt rules may not apply” to it; Perplexity, that its fetcher “generally ignores robots.txt rules”; Anthropic, by contrast, promises that all its bots respect the restrictions.[^3][^18][^4] **SEO bots.** SemrushBot, AhrefsBot and MJ12bot (Majestic) collect the links between sites for search engine optimization services. According to Cloudflare, SEO bots generated more than 13% of requests from verified bots in 2025.[^19][^32][^33][^9] **Vulnerability scanners.** The most frequent “visitors” are looking not for texts but for weak spots: `.env` files with passwords, the `.git` folder, WordPress login pages and its old files, such as wlwmanifest.xml, which WordPress has not shipped since 2023.[^34] In the experiment of a SANS researcher who disallowed popular paths in robots.txt, the violators' most requested page over a year was /wp-login.php, at 1,140 times; in another researcher's log, /.env alone drew more than 22,000 requests. In April 2025 GreyNoise saw a spike in scanning for `.git/config`: on 20–21 April, nearly 4,800 addresses a day, far more than usual, and 95% of the addresses engaged in it over 90 days were malicious.[^20][^21][^22] **Fakes.** Someone else's name costs one line of text: Common Crawl warns outright that “we are aware of crawlers falsely identifying themselves as CCBot”. So a bot's user agent proves nothing without checking its address.[^14][^11] ## Data **How many bots: it depends who is counting** | Who and when | What they measured | What they found | |---|---|---| | Imperva, 29.04.2026[^2] | All web traffic in 2025 | Bots: more than 53%. | | Fastly, 19.08.2025[^23] | AI bot traffic on the Fastly network, April to July 2025 | Crawlers (rather than user-initiated fetchers): almost 80% of AI bot traffic; Meta's crawlers account for 52%, Google's for 23%, OpenAI's for 20%. | | Cloudflare, 15.12.2025[^9] | Share of HTML requests in 2025 | AI bots: an average of 4.2%; Googlebot: 4.5%; as of 2 December, humans generated 47% of HTML requests and non-AI bots 44%. | | Ahrefs, 15.06.2026[^24] | Server logs of 137,000 sites, May 2026 | 28% of domains publish llms.txt, but 97% of those files received not a single request. | | Read the Docs, 25.07.2024[^25] | A single AI crawler in May 2024 | It downloaded 73 TB of zipped HTML, almost 10 TB in a single day, downloading the same files over and over. | > **[EDITORIAL CONCLUSION]** When the same thing is measured, the figures agree: at Imperva, bots are more than half of all traffic; at Cloudflare, humans generated less than half of HTML requests. Gaps of several times appear when all bots are compared with AI bots alone, or different networks with each other. One thing is common to all of them: the share of programs is growing, and AI crawlers take a lot while sending few visitors back to the site. **llms.txt.** A Markdown file for language models was proposed in 2024 by Jeremy Howard. Ahrefs found that almost nobody reads it, and Google's John Mueller, as reported by Search Engine Journal, compared it to the keywords meta tag, which search engines have long ignored.[^26][^24][^27] ## Criticism **robots.txt is a request.** The big operators promise to follow it, but the rule is voluntary. Perplexity's stealth crawling, the “residential proxies” at Wikimedia and the complaints from SourceHut show that some crawlers get around it: SourceHut founder Drew DeVault wrote that AI crawlers come from tens of thousands of mostly residential addresses, with one request from each.[^7][^10][^28] **Load with nothing in return.** At Wikimedia, bandwidth for downloading media files has grown by 50% since January 2024, mostly because of bots, and bots generate at least 65% of the most expensive traffic while accounting for about 35% of pageviews.[^29] **Traps.** An old trick is to disallow in robots.txt a path that a hidden link leads to: an honest bot will not go there, and a violator gives itself away.[^35] A SANS researcher who disallowed popular paths in robots.txt concluded that “one might be able to catch a significant portion of interesting ‘bad’ bot behavior with just a single-page … honeypot”.[^20] ## Development **Signatures instead of names.** ShapBot and the Exa crawler sign their requests under the Web Bot Auth scheme, based on HTTP Message Signatures (RFC 9421): a site can verify that a request came from whoever the bot claims to be, without lists of addresses.[^15][^16] **Payment and blocking.** Cloudflare by default offers new sites the option of blocking AI crawlers, and lets them charge for crawling and feed violators a labyrinth of generated pages.[^6][^30] > **[EDITORIAL CONCLUSION]** The web is splitting its guests into two classes: those who introduce themselves and follow the rules, and those who hide behind someone else's name or browser. For the first, signatures, payment and rules are emerging; for the second, there remain traps, address checks and blocking, and an arms race in which crawlers are already moving to home addresses. ## See also [Dead internet theory](https://ftrlg.com/en/wiki/dead-internet-theory/) · [Death of CAPTCHA](https://ftrlg.com/en/wiki/death-of-captcha/) · [Behavioral biometrics](https://ftrlg.com/en/wiki/behavioral-biometrics/) · [Moving target defense](https://ftrlg.com/en/wiki/moving-target-defense/) · [Wikipedia after AI](https://ftrlg.com/en/wiki/wikipedia-after-ai/) · [Markdown](https://ftrlg.com/en/wiki/markdown/) ## References [^1]: 2025 Imperva Bad Bot Report: How AI is Supercharging the Bot Threat — Imperva, 15.04.2025. [imperva.com](https://www.imperva.com/blog/2025-imperva-bad-bot-report-how-ai-is-supercharging-the-bot-threat/) [^2]: Bad Bot Report 2026: Bots in the Agentic Age — Imperva, 29.04.2026. [imperva.com](https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age/) [^3]: Overview of OpenAI Crawlers — OpenAI. [developers.openai.com](https://developers.openai.com/api/docs/bots) [^4]: Does Anthropic crawl data from the web, and how can site owners block the crawler? — Claude Help Center, 07.04.2026. [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) [^5]: The rise of the AI crawler — Vercel, 17.12.2024. [vercel.com](https://vercel.com/blog/the-rise-of-the-ai-crawler) [^6]: Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large — Cloudflare, 01.07.2025. [cloudflare.com](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) [^7]: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — Cloudflare, 04.08.2025. [blog.cloudflare.com](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/) [^8]: The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — Cloudflare, 29.08.2025. [blog.cloudflare.com](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) [^9]: The 2025 Cloudflare Radar Year in Review — Cloudflare, 15.12.2025. [blog.cloudflare.com](https://blog.cloudflare.com/radar-2025-year-in-review/) [^10]: Quo Vadis, Crawlers? Progress and what's next on safeguarding our infrastructure — Wikimedia Foundation, 26.03.2026. [diff.wikimedia.org](https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/) [^11]: Verify Requests from Google Crawlers and Fetchers — Google, 20.03.2026. [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) [^12]: About Applebot — Apple, 04.09.2026. [support.apple.com](https://support.apple.com/en-us/119829) [^13]: Meta-Webcrawler — Meta for Developers. [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) [^14]: CCBot — Common Crawl. [commoncrawl.org](https://commoncrawl.org/ccbot) [^15]: Crawler — Parallel Web Systems. [docs.parallel.ai](https://docs.parallel.ai/resources/crawler) [^16]: Exa Search Crawler — Exa, 30.07.2026. [crawler.exa.ai](https://crawler.exa.ai/) [^17]: Accel-backed Keenable is indexing the web for AI agents — TechCrunch, 25.08.2026. [techcrunch.com](https://techcrunch.com/2026/08/25/accel-backed-keenable-is-indexing-the-web-for-ai-agents/) [^18]: Perplexity Crawlers — Perplexity. [docs.perplexity.ai](https://docs.perplexity.ai/guides/bots) [^19]: Semrush Bot — Semrush. [semrush.com](https://www.semrush.com/bot/) [^20]: What pages do bad bots look for? — SANS Internet Storm Center, 01.08.2020. [isc.sans.edu](https://isc.sans.edu/diary/26414) [^21]: Increased Number of Configuration File Scans — SANS Internet Storm Center, 03.05.2023. [isc.sans.edu](https://isc.sans.edu/diary/29806) [^22]: Spike in Git Configuration Crawling Highlights Risk of Codebase Exposure — GreyNoise, 28.04.2025. [greynoise.io](https://www.greynoise.io/blog/spike-git-configuration-crawling-risk-codebase-exposure) [^23]: New Fastly Threat Research Reveals AI crawlers make up almost 80% of AI bot traffic — Fastly, 19.08.2025. [fastly.com](https://www.fastly.com/press/press-releases/new-fastly-threat-research-reveals-ai-crawlers-make-up-almost-80-of-ai-bot) [^24]: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Ahrefs, 15.06.2026. [ahrefs.com](https://ahrefs.com/blog/llmstxt-study/) [^25]: AI crawlers need to be more respectful — Read the Docs, 25.07.2024. [about.readthedocs.com](https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/) [^26]: The /llms.txt file — Jeremy Howard (llmstxt.org), 03.09.2024. [llmstxt.org](https://llmstxt.org/) [^27]: Google Says LLMs.Txt Comparable To Keywords Meta Tag — Search Engine Journal, 17.04.2025. [searchenginejournal.com](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) [^28]: Please stop externalizing your costs directly into my face — Drew DeVault, 17.03.2025. [drewdevault.com](https://drewdevault.com/blog/Stop-externalizing-your-costs-on-me/) [^29]: How crawlers impact the operations of the Wikimedia projects — Wikimedia Foundation, 01.04.2025. [diff.wikimedia.org](https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/) [^30]: Trapping misbehaving bots in an AI Labyrinth — Cloudflare, 19.03.2025. [blog.cloudflare.com](https://blog.cloudflare.com/ai-labyrinth/) [^31]: Parallel Web Systems hits $2B valuation five months after its last big raise — TechCrunch, 29.04.2026. [techcrunch.com](https://techcrunch.com/2026/04/29/parallel-web-systems-hits-2b-valuation-five-months-after-its-last-big-raise/) [^32]: Ahrefs Bots — Ahrefs. [ahrefs.com](https://ahrefs.com/robot) [^33]: MJ12Bot — Majestic. [mj12bot.com](https://mj12bot.com/) [^34]: wlwmanifest_link() — WordPress Developer Resources. [developer.wordpress.org](https://developer.wordpress.org/reference/functions/wlwmanifest_link/) [^35]: Easy PHP Blackhole Trap with WHOIS Lookup for Bad Bots — Perishable Press. [perishablepress.com](https://perishablepress.com/blackhole-bad-bots/) — future.log, «Bots on websites», https://ftrlg.com/en/wiki/web-bots/ · CC BY 4.0