• Generating AI bots could account for more than half of the world’s web traffic. Much of their activity involves directly scraping HTML content, which is not compatible with traditional analysis tools.
• By ignoring certain protocols, AI agents and crawlers overload processors and bandwidth, forcing publishers to fund expensive infrastructure that doesn’t generate revenue.
• Web players seek to convert these bots into paying customers through paid feeds, risking creating a two-tier internet between large and small businesses.
Popular AI tools like Claude, ChatGPT, and Gemini access websites to summarize content, aggregate news, and more. Publishers place great importance on search engine optimization (SEO) of their sites, including these tools, to improve the visibility of their sites. However, these activities put a strain on hosting infrastructure, especially since AI bots often ignore standard caching protocols to ensure they get the latest version of a page. In fact, bots increase both bandwidth usage and CPU load (rendering, database, API) on your infrastructure. According to the 2026 AI Bot Impact Report, bots account for 52% of global web traffic (30% to 50% according to Akamai). A resource crisis is occurring. According to the report, AI and LLM indexing bots quadrupled their share of traffic from 2.6% to 10.1% in just eight months. OpenAI’s GPTBot alone has grown by 305%.
Publishers face a paradox. Blocking some bots prevents your content from being indexed by AI tools and next-generation search engines.
Alex Taylor, founder of Blankspace, points out that a new wave of bots is consuming web content in ways that are undetectable. “These LLM agents crawl and retrieve content directly from HTML/DOM without running any JavaScript, thereby bypassing common analytics platforms (such as Google Analytics) that rely on in-browser scripts.” In other words, the traffic generated by AI bots may be underestimated.
Additional costs are impacting publishers
Philippe Ansalgue, vice president of software engineering at Orange, explained in an editorial that rapid requests from bots “put a huge strain on databases and content delivery networks. To make matters worse, bots often bypass or override caching mechanisms, forcing requests all the way back to the origin server, a much more costly route.” In other words, publishers will have to absorb the spike in infrastructure costs caused by AI bots.
“For companies operating on tight profit margins or fixed IT budgets, these unexpected costs quickly turn into painful trade-offs. Which services need to keep running? How should resources be allocated between paying customers and indexing bots that don’t buy anything? And therein lies the real question: These additional costs don’t generate revenue.” According to Akamai, AI bot triggers recorded 63% target the publishing industry, impacting publishers’ business models that rely on advertising.
Toward mandatory monetization
Publishers face a dilemma. Blocking some bots prevents their content from being indexed by AI tools and next-generation search engines like Perplexity. This is why new tools are emerging that allow publishers to monetize traffic from AI bots. The idea is to turn AI bots into paid feeds or licensed customers through the implementation of dedicated APIs or “AI feeds” via contracts, subscriptions, or content license agreements for model training in exchange for access to better structured data. This requires the ability to identify AI agents and distinguish between legitimate bots and aggressive scrapers. In any case, cloud providers will need to adapt and offer specific solutions through filtering options and partnerships with AI companies.
Existing solutions designed to help publishers take back control require a consistent technical, economic, and legal strategy to avoid compromising their business models. Companies that treat AI bots as a new category of customer will turn a cost center into a strategic asset if they can measure this traffic, have protocols in place, and negotiate with AI providers. This requires resources that are owned only by the specific publisher. “The risk is that a two-tier internet will emerge. Large enterprises have the means to adapt, but small and medium-sized enterprises are seeing their cloud costs rise without knowing why,” Philippe Ansalgue points out. “Edge computing is often cited as a solution. The idea is appealing: serving from a node close to the bot to reduce transport costs. This lowers the cost per request, but it doesn’t address the volume of traffic at the source. Bots that make 10 million requests a day are here to stay.”
