Interesting piece from Stephen Follows, a data analyst for the film industry, he has a useful newsletter I subscribe to. In it he tracks the changes that AI crawlers have brought to the web, their impact and issues which emerged.
6 Likes
As someone who has owned a small, mostly hands-on web hosting company for almost 30 years, what this article describes matches up exactly with my own experience, albeit on a much smaller scale.
While I was aware of an overall increase in traffic throughout 2025 on the servers I manage, nothing really needed intervention until earlier this year. Since then, and especially in the last 6 weeks, the amount of time needed to deal with all the various sources of bot traffic has increased substantially, forcing other things to be put off.
Many typical small business websites that normally did 1-2 GB of traffic per month are now doing that much per day. One good example I can give is a website for the alumni of a decent sized high school. There are many pages devoted to each year going back to the 1950’s, full of stories, photos, event descriptions, obituaries, and so on. Last July that site did a total of 10 GB of traffic. This year, just in the first 3 weeks of July, it’s already at 450 GB, and that’s AFTER taking extra steps to block many bot sources.
Everyone that uses the internet should read that article and become familiar with the topic to have a better understanding of what’s happening and why. It’s only going to get worse before some kind of change becomes necessary, and whatever that change is, it’s unlikely to be beneficial to the average user.
4 Likes
Indeed. I’m on the board of directors for a small non-profit organization for survivors of abuse. We’re trying to decide what we should do with our twenty-year online presence, including planning for the possibility of shutting down entirely. We have no paid staff; everything we do is driven by volunteers.
We already had shut down our public support forum a couple of years ago, as the effort involved in screening bots and malicious users outstripped our moderating capacity. As artificial traffic increased and direct attacks multiplied, we replaced our old website with one that has significantly fewer features and information. Now it’s basically a minimalist, self-hosted WordPress installation with only a couple of very carefully selected plugins and significantly less information posted than in the past. We’re gradually adding back the most important content, but it takes time to do it well. We have a WordPress firewall installed, and it logs thousands of attempted attacks daily. (As an aside, another non-profit I know of was offline for more than a day due to a zero-day flaw in a WordPress plugin that allowed an external attacker to gain admin access without requiring a username or password.)
We have a private support forum with around 3,000 members. We had to move that to a new technology platform a few years ago, too, because of security concerns with the previous platform. While our new forum is hosted and managed by a reputable third-party, security remains a huge concern. We strive for confidentiality so that members can share stories candidly, but as attackers’ tools get better, liability is a serious concern. A data breach would cause serious damage to users and to the organization, so shutting that down is under discussion.
On top of all of that, we expect costs to rise significantly as artificial volume continues to increase. We take advantage of free or reduced-cost services offered by providers to non-profits, but the providers definitely are feeling the pressure.
To make a long story short, an important recovery resource that has helped thousands of people over two decades and that has been largely self-sustaining through that time is having serious conversations about whether it can continue much further into the future. Thanks to the negative effects of A.I., I suspect this is happening in many corners of the Internet.
2 Likes