Millions of websites now protect content from data-scraping AI bots

Cloudflare, which routes an estimated 16 percent of global internet traffic, is introducing a default protection layer against AI bots that scrape web content to train language models. The move comes as publishers around the world file lawsuits against OpenAI, Meta, and others—but with little legal success so far.

The new AI blocker targets so-called crawlers—automated bots that gather online content. These bots will now be blocked by default on millions of websites using Cloudflare’s infrastructure. Publishers can choose which bots to allow and even set rates for access—a model Cloudflare calls “Pay Per Crawl.”

Publishers like Condé Nast and The Atlantic call the measure a “gamechanger” and see it as the first meaningful protection against unauthorized data use. Cloudflare CEO Matthew Prince warns that AI-generated answers may divert traffic away from original sources, threatening both the visibility and sustainability of independent content.

OpenAI has refused to cooperate with the system, though smaller AI firms like search engine Gist have embraced it, citing a commitment to protect human creativity in the age of AI.

Read the full article on the Volkskrant website.