I check in with Cloudflare at the beginning of every quarter to see how the biggest AI company is handling the web. The latest data tells the familiar story that strip mining continues at a rapid pace.
For the week of July 1-7, the latest “crawl to browse” data shows that Anthropic remains the biggest outlier. The AI company’s bots crawled a web page approximately 2,800 times for every referral returned.
OpenAI was far behind, with Perplexity in third place. Next, Microsoft has fallen significantly again, followed by Google.
DuckDuckGo is one of the few players that offers a near-fair balance between giving and giving back information from the web, with three scrapes per referral, according to Cloudflare data.
Graph showing crawl to reference ratio cloudflare
Anthropic has seen an improvement from about 8,800 to 1 in early April. But a quick check of other recent 7-day periods shows that Anthropic is still raking in like crazy.
For example, in the first seven days of May, their bots scraped sites 24,700 times for every referral. (Perhaps some of the bots went on summer vacation in early July?)
Cloudflare’s metric measures how often the AI company’s bots request a crawl of web pages compared to how often its service sends users back to those sites. The numbers are a key indicator of whether the AI giants are preserving or undermining the economic deals that have historically underpinned the web, in which sites allow free crawling in exchange for traffic.
In the new world of generative AI, the deal is: break. Now that AI answer engines and chatbots provide answers directly to users, people are less likely to visit the website that created the content in the first place. This reduces the economic incentive to publish high-quality content on the web.
Anthropic touts itself as the most ethical AI company, but this data is a reminder that what is and isn’t ethical can be in the eye of the beholder. I’ll leave it to you to decide.
Anthropic has criticized competitors in recent months for using the output of AI models to develop and improve their own models. The company calls this technique a “distillation attack” and says it violates its terms of service.
If you step back a little bit, this starts to look a lot like what Anthropic has been doing with their website. They collect content, often against the wishes of content owners, and use it to develop and improve their products.
Welcome to Anthropic, the modern web. You may have to get used to having your output used in ways you don’t like.
Note: Anthropic previously disputed Cloudflare’s methodology, saying it could not verify Cloudflare’s calculations and claiming that new search features were driving referrals.
Sign up for BI’s Tech Memo newsletter here. Please contact us by email. abarr@businessinsider.com.