CloudFlare has announced a major policy shift that can reshape the dynamics between content creators and AI companies.
Starting Tuesday, the company will automatically block AI crawlers from cutting down on websites that provide power unless the site owner explicitly opts in. This will make CloudFlare the first major Internet infrastructure provider to implement a default permission-based model of AI content access.
The move comes amid growing concerns from content creators and publishers that the AI Giants are exploiting their jobs without consent or compensation.
Historically, search engines index content in a way that reverts traffic and ad revenue back to their original sources. However, AI bot crawlers used to train large language models, harvest huge amounts of data and send back much less traffic to the original creator of this content. These bots are used by industry giants such as Google, Meta, Openai and Anthropic.
CloudFlare, which manages and protects approximately 20% of the web's traffic, first deployed its optional one-click AI crawler blocking system in 2024. Over 1 million customers have enabled it. This has become the default. AI companies must obtain explicit permission from their website before inverting scripts with passive data harvesting.
“Original content makes the Internet one of the biggest inventions of the last century and needs to be brought together to protect it,” said Matthew Prince, CEO of CloudFlare.
“AI crawlers are rubbing content without restrictions,” he added. “Our goal is to bring the power back to the hands of creators while helping AI companies innovate. This is to protect the future of a free, vibrant internet with new models that work for everyone.”
Pay for each crawl
As part of a broader push to the permission-based internet, CloudFlare rolled out wages per crawl on Tuesday. This new feature allows some publishers and creators to bill AI companies to access content. Participants have complete control over how and how their work is being used in AI model training.
CloudFlare wants to create a transparent, consent-driven market that will help you decide whether to allow all AI crawlers, allow certain ones, or set your own access fees, turning the use of previously unmetalized content into a new revenue stream.
For AI companies, Pay Per Crawl offers a streamlined interface for browsing access terms, viewing pricing, and choosing to pay or leave without data.
Pay Per Crawl is currently available to partner groups of your selected group and has more access to this sign-up page.
Major publishers have signed or expressed the latest moves from CloudFlare, which blocks AI bot crawlers by default, including Gannett, Time, Buzzfeed, The Atlantic, and The Associated Press. Other companies, such as Stack Overflow, have approved the initiative.
CloudFlare's movement is at its heart, an attempt to reset the economic model of the Internet in a new era of generating AI. The initiative does not halt AI innovation, but encourages people to grow responsibly by paying their intellectual property and rewarding the creators behind their data.
Sign up for BI's Tech Memo Newsletter here. Please contact me by email abarr@businessinsider.com.
