According to a Tuesday announcement, leading Internet architecture provider CloudFlare has now blocked known AI web crawlers by default to block known AI web crawlers to prevent “accessing content without permission or compensation.” With this change, CloudFlare will begin asking owners of new domains if they want to allow AI scrapers, allowing some publishers to implement “per-crawl payment” fees.
The per-salary crawl program allows publishers to price AI scrapers and access content. AI companies can view pricing and choose to register for the “Pay per Crawl” fee or leave. This is only available to “a group of major publishers and content creators,” but CloudFlare says it “ensures that AI companies can obtain permission and compensation to use quality content in the right way.”
CloudFlare has been helping domain owners fight AI crawlers for a while. The company allowed AI Crawlers to be blocked on its website in 2023, but only applied to those that adhere to the site's robots.txt file. CloudFlare last year allowed websites to block “all” AI bots (whether or not you respect the site's robots.txt file or not), and this setting is enabled by default for new CloudFlare customers. (The company identifies the scraper by comparing it to a list of known AI bots.) CloudFlare launched a feature that sends web crawling bots to AI Labyrinth in March, blocking scraping sites without permission.
Some major publishers and online platforms including Associated Press, Atlantic Ocean, luckStack Overflow, and Quora are now in CloudFlare's new AI Crawler limits as websites are fighting against the future where they find information through AI chatbots rather than search engines. “It means that people have more trusted AI for the past six months, which means they haven't read the original content,” CloudFlare CEO Matthew Prince said at a live Axios event last week.
Additionally, CloudFlare says it can work with AI companies to help validate crawlers and “explicitly state their purpose” such as whether they are using content for training, inference, or search. The website owner can then review this information and decide which crawlers to put in.
“Original content is what makes the Internet one of the biggest inventions of the last century and must be brought together to protect it,” Prince said in a press release. “AI crawlers are rubbing content without restrictions. Our goal is to bring the power back to the creators' hands while helping AI companies innovate.”
