Bots on websites

Bots on websites are programs that request pages on their own: search engine robots, data crawlers for AI, assistants that open a page at a person’s request, analytics services and vulnerability scanners. By Imperva’s estimate, in 2024 automated traffic exceeded human traffic for the first time in a decade, at 51% of all web traffic, and in 2025 it rose to 53%.¹²

Bots that introduce themselves honestly are easy to recognize by their user agent and to verify by their address. The rest are harder: crawlers pass themselves off as browsers, scanners hunt for other people’s passwords, and fakes call themselves Googlebot. There is one rule of conduct for all of them, the robots.txt file, but it is a request, not a lock.³⁴

History

• 17.12.2024Vercel and MERJ report that over one month GPTBot made 569 million requests across their network and ClaudeBot 370 million; none of the major AI crawlers other than Googlebot and Applebot executes JavaScript.⁵ ¶ cgrvjzz57
• 15.04.2025Imperva: automated traffic in 2024 made up 51% of all web traffic, and malicious bots 37%.¹ ¶ cy4y8dfw4
• 01.07.2025Cloudflare, by its own account the first infrastructure company to do so, begins to offer new sites, by default, the option of blocking AI crawlers, and launches a beta of payment for crawling (pay per crawl).⁶ ¶ crm7a7wpw
• 04.08.2025Cloudflare accuses Perplexity of stealth crawling: a blocked bot kept coming back disguised as an ordinary Chrome browser, switching addresses and networks, on tens of thousands of domains, with millions of requests a day.⁷ ¶ cgggkbbr8
• 29.08.2025Cloudflare: almost 80% of AI bot activity is data collection for training; in July, Anthropic’s crawlers loaded about 38,000 pages for every visitor they sent to a site.⁸ ¶ czcnfd3sw
• 15.12.2025Cloudflare’s review of 2025: AI bots account for an average of 4.2% of HTML requests, and Googlebot, which collects data for both search and AI, for 4.5%; crawling at users’ request grew more than 15-fold.⁹ ¶ ck9ra3ui5
• 26.03.2026The Wikimedia Foundation reports (in figures updated in April) that it blocks or throttles about 30% of automated requests from crawlers that do not follow its rules, and those crawlers have moved to “residential proxies”: other people’s home and mobile connections.¹⁰ ¶ cfjpxjuum
• 29.04.2026Imperva: bots account for more than 53% of all web traffic.² ¶ ci4shwu2u

Types of bots

Search. Googlebot, Applebot and Bingbot build indexes for search. Their authenticity can be checked: Google advises running a reverse DNS lookup on the address, making sure the name ends in googlebot.com, google.com or googleusercontent.com, and confirming with a forward lookup that the name points back to the same address. Apple’s Applebot powers Spotlight, Siri and Safari, but its data may also go into training Apple’s models.¹¹¹² ¶ cfadun3eq

AI training. GPTBot from OpenAI, ClaudeBot from Anthropic, Meta-ExternalAgent from Meta, Amazonbot from Amazon and CCBot from Common Crawl collect the texts that models are trained on. They bring the site almost no visitors: hence the gap between pages loaded and click-throughs that Cloudflare measures.^3413148 ¶ cc9eyyjum

AI search. OAI-SearchBot, Claude-SearchBot, PerplexityBot and newer services (ShapBot from Parallel Web Systems, the company of former Twitter CEO Parag Agrawal, the Exa crawler and KeenableBot) build the indexes that AI assistants and agents answer from. Keenable was founded by Andrey Styskin, former head of search, AI and cloud at Yandex, and Matthias Petri; the company says its index holds more than 100 billion documents.^315161718 ¶ cf2ksku4i

User-initiated. ChatGPT-User, Claude-User and Perplexity-User open a page when a user has asked about it. OpenAI writes that because such an action is initiated by a person, “robots.txt rules may not apply” to it; Perplexity, that its fetcher “generally ignores robots.txt rules”; Anthropic, by contrast, promises that all its bots respect the restrictions.³¹⁹⁴ ¶ cqb44j4pz

SEO bots. SemrushBot, AhrefsBot and MJ12bot (Majestic) collect the links between sites for search engine optimization services. According to Cloudflare, SEO bots generated more than 13% of requests from verified bots in 2025.^2021229 ¶ cvj7tbync

Vulnerability scanners. The most frequent “visitors” are looking not for texts but for weak spots: .env files with passwords, the .git folder, WordPress login pages and its old files, such as wlwmanifest.xml, which WordPress has not shipped since 2023.²³ In the experiment of a SANS researcher who disallowed popular paths in robots.txt, the violators’ most requested page over a year was /wp-login.php, at 1,140 times; in another researcher’s log, /.env alone drew more than 22,000 requests. In April 2025 GreyNoise saw a spike in scanning for .git/config: on 20–21 April, nearly 4,800 addresses a day, far more than usual, and 95% of the addresses engaged in it over 90 days were malicious.²⁴²⁵²⁶ ¶ cc88m7vvt

Fakes. Someone else’s name costs one line of text: Common Crawl warns outright that “we are aware of crawlers falsely identifying themselves as CCBot”. So a bot’s user agent proves nothing without checking its address.¹⁴¹¹ ¶ c62f53siq

Data

How many bots: it depends who is counting

[EDITORIAL CONCLUSION] When the same thing is measured, the figures agree: at Imperva, bots are more than half of all traffic; at Cloudflare, humans generated less than half of HTML requests. Gaps of several times appear when all bots are compared with AI bots alone, or different networks with each other. One thing is common to all of them: the share of programs is growing, and AI crawlers take a lot while sending few visitors back to the site. ¶ cguddn7wj

llms.txt. A Markdown file for language models was proposed in 2024 by Jeremy Howard. Ahrefs found that almost nobody reads it, and Google’s John Mueller, as reported by Search Engine Journal, compared it to the keywords meta tag, which search engines have long ignored.³⁰²⁸³¹ ¶ cxcdqt2hj

Criticism

robots.txt is a request. The big operators promise to follow it, but the rule is voluntary. Perplexity’s stealth crawling, the “residential proxies” at Wikimedia and the complaints from SourceHut show that some crawlers get around it: SourceHut founder Drew DeVault wrote that AI crawlers come from tens of thousands of mostly residential addresses, with one request from each.⁷¹⁰³² ¶ cg3jrxe4w

Load with nothing in return. At Wikimedia, bandwidth for downloading media files has grown by 50% since January 2024, mostly because of bots, and bots generate at least 65% of the most expensive traffic while accounting for about 35% of pageviews.³³ ¶ ciz668dck

Traps. An old trick is to disallow in robots.txt a path that a hidden link leads to: an honest bot will not go there, and a violator gives itself away.³⁴ A SANS researcher who disallowed popular paths in robots.txt concluded that “one might be able to catch a significant portion of interesting ‘bad’ bot behavior with just a single-page … honeypot”.²⁴ ¶ cmj98ebqu

Development

Signatures instead of names. ShapBot and the Exa crawler sign their requests under the Web Bot Auth scheme, based on HTTP Message Signatures (RFC 9421): a site can verify that a request came from whoever the bot claims to be, without lists of addresses.¹⁵¹⁷ ¶ czz5k6n9g

Payment and blocking. Cloudflare by default offers new sites the option of blocking AI crawlers, and lets them charge for crawling and feed violators a labyrinth of generated pages.⁶³⁵ ¶ cpkhegf79

[EDITORIAL CONCLUSION] The web is splitting its guests into two classes: those who introduce themselves and follow the rules, and those who hide behind someone else’s name or browser. For the first, signatures, payment and rules are emerging; for the second, there remain traps, address checks and blocking, and an arms race in which crawlers are already moving to home addresses. ¶ ch6tc2nsu