All Customers' AI Indexes

Today we are presenting it Private Beta of AI index For CloudFlare's domain, for a new type of web index that provides content creators with tools to enable AI to discover data, and allows AI builders to access better data for fair compensation.

When you enable AI indexing in your domain, it automatically creates an A-Optimized Search Index for your website, exposing a set of ready-to-use standard APIs and tools, such as the MCP server, LLMS.TXT, and the search API. You have the ability to own and control its index and how it is used and monetize it through access Pay for each crawl And new X402 Integration. You can use it to build the latest search experience on your site, and more importantly, interact with external AI and agent providers to help you discover more content while being significantly compensated.

For AI Builders – Whether you're a developer who writes agent applications or an AI platform company that offers a basic LLM model, Cloudflare offers a new way to discover and retrieve web content. Pub/Subconnection To individual websites with AI indexes. Instead of indiscriminate crawling, builders can subscribe to certain sites they opted into for discovery, receive structured updates as soon as content changes, and pay fairly for each access. Access is at the site owner's discretion at all times.

From individual indexes, CloudFlare also builds aggregated layers. Open indexit bundles participating sites together. Builders get a single location for searching across collections or broader webs, but all sites still retain control and can be obtained from participation.

AI platforms are becoming one of the main ways people can discover information online. Whether you ask your chatbot to summarise news articles or find product recommendations, the path to that answer most often starts with crawling original content and using that data for training. Today, however, the process is primarily controlled by the platform. raw is how often the site owner enters into the question.

CloudFlare offers you the ability to monitor and control how AI services respect access policies and how they access content, but viewing new content remains difficult. Content creators don't have an efficient way to send signals to AI Builders when pages are published or updated. On the other hand, AI builders waste resources, especially if they don't know the quality and cost in advance.

A more equitable and healthy ecosystem for content discovery and use that bridges the gap between content creators and AI builders.

If your domain is installed in CloudFlare, or if you have an existing domain in CloudFlare, you can enable AI indexing. When enabled, it automatically creates an A-Optimized Search Index for the domains you own and control.

As the site updates and grows, the index evolves with it. New or updated pages are processed in real time using the same technology that runs CloudFlare AI Search (formerly Autorag) And that Website As a data source. Above all, we control everything. There's no need to worry about computing, storage resources, databases, embedding, chunking, or individual components of an AI model. Everything happens automatically behind the scenes.

The important thing is which content you can control. Include or exclude From your website index, and Who is You can access content through ai Crawl controlensuring that only the data you publish is searchable and accessible. You can also opt out of AI indexes completely. It all depends on you.

Once the AI ​​index is set, you get a set of ready-to-use APIs.

• MCP Server: The agent application is Model Context Protocol (MCP)enabling agents to discover content in a standardized way. This includes support nlweb Tools, an open project developed by Microsoft, defines the standard protocol for natural language queries on websites.

• Flexible search API: This endpoint will Returns the relevant results in structured JSON.

• llms.txt and llms-full.txt: Standard file that provides machine-readable maps of the site to LLMS New open standards. These help you understand how models use site content when inference. Examples of llms.txt It is present in the CloudFlare developer documentation.

• Bulk Data API: Endpoints It can be used with the rules you set to efficiently transfer large amounts of content. Instead of queriing all the documents, AI providers can ingest it in one shot.

• Pub-Sub Subscription: The AI ​​platform can subscribe to site indexes and receive events and content updates directly from CloudFlare in real time, allowing you to stay up to date without recrawling.

• Discoverability Directive: robots.txt and famous URIs allow AI agents and crawlers to visit the site and automatically discover and use available APIs.

Indexes are directly integrated AI Crawl Controlso you can see who is accessing the content, set rules, and manage permissions. and Pay for each crawl and X402 Integrationyou can choose to monetize access to content directly.

As an AI builder, you can discover and subscribe to high quality, permitted web data via the AI ​​index of individual sites. Instead of blindly sending crawlers to the open internet, they connect via pub/submodels. Participating websites publish structured updates whenever content changes and can receive those updates in real time. In this model, the new workflow might look like this:

1. Discover the website you opted in. Browse and filter the directory of your website that will allow indexes to be used through CloudFlare.

2. Evaluate content using metadata and metrics. Get information about content metadata information (for example, uniqueness, depth, context-related, popularity) before accessing it.

3. Pay fairly for access: If the content is valuable, the platform can directly compensate creators through pay per crawl. These payments not only allow access, but also support the ongoing creation of original content and help maintain a healthier ecosystem for discovery.

4. Subscribe to updates: You can use your Pub-Sub subscription to receive events regarding changes made by your website, so you can see when you want to retrieve or crawl new content without wasting resources with constant re-crawling.

By shifting from blind crawling to permitted Pub/Sub systems for the web, AI builders save time, reduce costs, and access cleaner, higher quality data.

Individual indexes are provided to AI platforms that allow direct access to data from specific sites, allowing you to subscribe to updates, evaluate values, and pay full content access per site. However, if a builder needs to work on a large scale, managing dozens or hundreds of individual subscriptions can be complicated. Open index Additional options are provided. A bundled opt-in collection of these indices. It features sophisticated features such as quality, uniqueness, originality, and content depth filters, all accessible in one place.

Open Indexes are designed to make discovering content at scale easier.

• Get unified access: Many participating sites simultaneously query and retrieve data. This reduces integration overhead and allows builders to connect to a Curly data collection or use it as a ready-made web search layer that can be accessed during queries.

• Discover a wider scope: Use topic-specific bundles (e.g. news, documents, scientific research) or general discovery indexes that cover the broader web. This makes it easy to explore new content sources that could not be individually identified.

• Bottom-up monetization: The results still come from the AI ​​index of individual sites, and monetization flows back to that site through pay per crawl, helping to maintain fairness and sustainability at scale.

Together, per-site AI and open indexes provide flexibility and precise control when you need full content from individual sites (i.e. training, AI agents, or search experiences) and provide wide range of search coverage when you need unified search across the web.

AI Index and CloudFlare Open Index create models where websites determine how to access content and AI builders receive structured, reliable data at scale, in order to help them build a more equitable and healthy ecosystem for content discovery and use on the Internet.

Let's start with Private Beta. If you want to register your website with AI indexes or access Pub/Sub web feeds as an AI builder, Sign up today.