Cloudflare announced a new feature that automatically serves Markdown websites to AI agents. The company uses Markdown for Agents to address the growing role of AI systems in processing web content for search engines, summarization, and analysis.
This solution makes it easier and more efficient for AI agents to read information from websites without the need for website owners to modify the content.
Importantly, Cloudflare can convert web pages from HTML to Markdown at the edge when requested by a client. This is done via an HTTP Accept header with the value text/markdown. When the AI agent sends this header, Cloudflare converts the HTML page to Markdown and returns it in real time. For regular visitors, the HTML display remains unchanged.
According to Cloudflare, Markdown is better suited for AI processing than HTML. HTML often includes a lot of additional code for styling, scripting, and layout, whereas Markdown represents content in a cleaner, more semantically clear way. This means less noise in language models and other text analysis systems, which can potentially improve processing efficiency and reduce computing costs.
Huge token savings
You could potentially save a lot of tokens. Based on Cloudflare’s example, The Register calculated that a simple Markdown heading costs about 3 tokens, while the HTML variant quickly consumes 12 to 15 tokens, even without additional markup such as div elements, navigation bars, or scripts. For full blog posts from Cloudflare itself, the difference is even greater. According to The Register, the number of tokens decreased from 16,180 for HTML to 3,150 for Markdown, a decrease of about 80%. This concerns AI services that operate with limited context windows and token-based cost models.
When a website owner enables Markdown for the agent, Cloudflare adds an additional header x-markdown-tokens to Markdown responses. This header contains the calculated number of tokens in the returned content. This helps the AI agent determine whether the document fits within the available context window or needs to be split.
No need for separate Markdown versions
This feature applies to HTML pages and not other document formats such as PDF files. Conversion happens within the Cloudflare network, eliminating the need for publishers to maintain separate Markdown versions. In doing so, the company is positioning itself as an Internet infrastructure provider, with human visitors as well as AI agents being important target groups.
This move is notable since Cloudflare previously introduced mechanisms to charge AI crawlers for content access. The company previously positioned itself as an organization that protects publishers from scraping, but now it offers a way to make automated access more efficient. According to The Register, coding agents like Claude Code and OpenCode have already indicated demand by explicitly requesting Markdown in their Accept headers.
Markdown for Agents works in conjunction with the previously introduced Content Signaling Policy. This allows publishers to use robots.txt to indicate whether their content can be used for AI training, AI search applications, or post-training AI input (such as search enhancement generation). Like robots.txt, these signals are voluntary and non-enforceable, but they provide a standardized way to communicate your usage preferences.
Cloudflare is taking the next step in adapting its internet infrastructure to growing AI traffic with Markdown for Agents. HTML has been the browser standard for decades, but parallel layers are now emerging that are optimized for the economic realities of machine readers and token-based AI models.
