Cloudflare to let customers block AI web crawlers

Of particular interest (I assume) to many site owners is the idea of permitting crawlers if they pay for the access.

Here’s the Cloudflare blog post, which explains more technical detail and has a link to sign up for the private beta, which I have.

https://blog.cloudflare.com/introducing-pay-per-crawl/

1 Like

This video on the point is interesting.

His other videos on YouTube are well worth watching as well. He’s very entertaining. I’d particularly recommend Art of Code and Plain Text (yes, he talks for an hour on “plain text”).

Thanks for that link—he does a good job explaining the proposal, but even better, comes to much the same conclusion as I was about how this could point to direct financial transactions for consuming created content.