3 months ago
1
The rise and fall of robots.txt
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
Continue Reading
https://www.theverge.com
Join the Discussion
Okay, little update from the front.
I had already adjusted my robots.txt, but also added an ai.txt to the root of my site. And I have adjusted my .htaccess-file, since that has near-firewall-power (according to this article).
Another, not really well measured, observation made in the server-stats and logs is that, indeed, the type of traffic is changing. Most visitors are staying longer on the pages (reading, probably) and less bot-activity is seen, but not measured. That last bit will be done in a few weeks time, since bots are no daily regulars.