CloudFlare, a content delivery network, is a move that could have a major impact on the ability of AI developers to train models, and by default, artificial intelligence crawlers start blocking access to content without the permission or reward of website owners.
Starting today, every new web domain you sign up for CloudFlare will be asked if you want AI crawlers to be allowed, preventing bots from cutting data from their websites.
For decades, the internet has been operating with simple exchanges. Search engine index content and users are directly reverted to their original websites, generating traffic and advertising revenue for websites of all sizes. This cycle rewards creators who generate high-quality content with money and follow, while helping users discover new relevant information. That model is broken. AI crawlers collect content such as text, articles, images and other content to generate answers without sending visitors to their original sources. The satisfaction of knowing that content creators are taking away revenue and someone is showing the content. If the incentive to create original quality content disappears, society will be lost and the future of the Internet will be at risk.
CloudFlare helps to strengthen one of the world's largest networks and manage and secure 20% of the web's traffic. As the company handles trillions of requests every day, it has the world's most advanced bot management solution, accurately distinguishing between human users and AI crawlers. In September 2024, CloudFlare introduced the option to block AI crawlers with just one click. Over a million customers have since chosen this option. This is intended to be an aggressive but simple solution that will stop scraping while deciding on your AI strategy.
“If the Internet is about to survive in the age of AI, we need to provide publishers with the control they deserve and build a new economic model that will serve everyone: creators, consumers, the founders of tomorrow, the future of the web itself.” “Original content is what makes the Internet one of the biggest inventions of the last century, and it is essential that creators continue to make it. AI Crawls is rubbing content without restrictions. Our goal is to put the power in the hands of creators while helping innovate AI companies.
Mixed response to the solution
Dr. Kolochenko, CEO of ImmuniWeb and Fellow of the British Computer Society (BCS), has issued a statement on news that he does not grasp the pros and cons of the development.
He states: “This long-awaited feature by CloudFlare has been a real disaster for many Genai vendors and may be fatal to Genai's current business model. Given that CloudFlare protects most of the world's most popular websites, it protects millions of small websites publishing academic and scientific content.
Ironically, the fierce legal battles currently taking place in court on both sides of the Atlantic have challenged the alleged copyright infringement by numerous AI vendors, but most have re-studied arguments that have already been lost. Additionally, content providers have added certain contractual provisions that expressly prohibit the use of data for the purposes of LLM training.
“If we violate such terms of service, the content provider may be subject to liquidated damages for each violation, which could be in great advantage for the plaintiff. Furthermore, in some jurisdictions, the protection of the bot is engraved, so that it can compensate for the violation. Ultimately, Genai vendors who argue that for the purposes of LLM training constitute an exception to fair use under copyright law, could face even greater liability under an avalanche of contract claims.
Game-changing technology
Many industry leaders praised the move as a step towards protecting intellectual property on the Internet. “CloudFlare's innovative approach to blocking AI Crawlers is a game changer for publishers, setting new standards for how content is respected online. When AI companies no longer receive what they want for free, it opens the door to sustainable innovation built on permissions and partnerships. “This is a critical step in creating fair value exchanges on the Internet that protect creators, support quality journalism, and hold AI companies accountable.”
Anastasia Nyrkovskaya, CEO of Fortune, said: “The rise of AI presents an incredible opportunity, but publishers lock in appropriate belongings and compensation for our valuable intellectual property and present carefully crafted content. It contributes to the sustainability of businesses and publishers.”
Darragh Lucey, CEO of Half Baked Newsletter, expressed a positive response to the news as a small business owner, saying the move would have a beneficial impact. “As a small publisher, we rely on the trust and engagement of our readers. CloudFlare's movement provides the controls we need to protect our content and continue to build reality in the world of AI noise.
“Creators and publishers around the world use Pinterest to expand their businesses, reach new audiences and measure success directly. As AI continues to rebuild its digital environment, they are committed to building a healthy internet infrastructure that content aims for.
Dr. Kolochenko added: “In summary, most Genai vendors face a harsh reality right away. Paying the right price for high-quality training data remains beneficial. Given the horrifying competition emanates from China, many Western Genai companies may quit their economically impossible business.”
