Read the Docs: in May 2024 a single AI crawler downloaded 73 TB of zipped HTML, almost 10 TB in a single day, downloading the same files over and over
translated by AI, reviewed by the editors
· Internet & infrastructure · Software development
- technology
- Bots on websites
- source
- about.readthedocs.com ↗
¶ cvdnx9hpy