NEW YORK—As media companies face increasing challenges with AI bots and crawlers, the IAB Tech Lab has issued guidance to help publishers and content owners better assess and manage how non-human agents, including AI systems, access their content as they work to implement the Tech Lab’s CoMP API framework for managing content crawls.
The new guidance, currently open for public comment until June 26, 2026, outlines different approaches that content owners and publishers can take to signal access from non-human user agents, with the goal of helping the ecosystem make more informed decisions in parallel with the adoption of CoMP API V1.
Anthony Catur, CEO of IAB Tech Lab, said: “This initiative helps simplify a complex area, allowing companies to make decisions that are right for their business while supporting a more sustainable market.”
The IAB Tech Lab noted that after the release of CoMP API V1, it became clear that many content owners and publishers had not yet developed a formal strategy for approaching bots and crawlers outside the scope of the API itself.
To address this gap, the CoMP Working Group has produced this guidance outlining the range of approaches available and the benefits and trade-offs of each strategy. This gives stakeholders a clearer understanding of how to manage access from all non-human traffic, including AI systems, in a way that aligns with business objectives.
More specifically, this guidance is designed to support CoMP API adoption by helping content owners navigate what can be operationally complex and resource-intensive areas, and is intended to help non-technical leaders provide the questions they need to ask.
These questions will ultimately help you understand the value and hard costs associated with each lever you pull. With clear direction around these strategies, organizations can make more confident decisions, reduce risk, reduce operational costs, and operate more efficiently when working with AI systems. Establishing these strategies is also critical to enabling the partnerships between AI systems and content owners needed to build sustainable and scalable content markets.
Many industry participants involved in developing the guidance also emphasized the importance of clarity and practical direction as the ecosystem evolves.
said Shailley Singh, EVP and COO of IAB Tech Lab. “This guidance will help you narrow down your options and help you choose an approach that aligns with your goals.”
“With the AI revolution, many publishers are opening the door wide open to bots and crawlers to steal intellectual property.” “The IAB Tech Lab ‘s guidance is the essential roadmap content owners need. It’s a practical framework for navigating the complex maze of decisions, technologies, and tools available to protect their businesses. By helping publishers identify who’s really out there, this guide helps differentiate between the “allies” that bring value and the “extractors” that drain resources. Managing AI bots is no longer just a technical background task. This is an important business strategy to protect what you’ve built and restore fair value exchange for your content. ”
Similarly, Dave Bellous, Vice President of Strategy at Metal Toad, said, “Non-human traffic now dominates much of the open web, and AI agents and crawlers are steepening the curve. Blanket bot blocking is no longer a useful strategy. Differentiating useful automation from invalid traffic through progressive, standards-based controls is a fundamental requirement for publishers, advertisers, and ad tech alike. The IAB’s Bot and Crawler Guidance is the foundation for modern Web It’s a much-needed tool for building.” strategy. ”
Tom Koch, head of sales, advertising and marketing ecosystem at TwelveLabs, also emphasized the importance of guidance when dealing with bot and crawler issues. “As AI agents increasingly act as surrogates for consumers, publishers and platforms need a clearer framework for transparency and control. A crawler management framework is an important step in giving content owners greater visibility into how their content is being accessed and used. TwelveLabs is proud to contribute, as this work is fundamental to ensuring video is the richest form of AI.” Data is no longer ignored in how we interact with the web. ”
The guidance was developed by the CoMP Working Group and provided for public comment with an opportunity for industry feedback prior to final decision. The IAB Tech Lab will continue to work with members to support implementation and accelerate adoption of both the guidance and CoMP API.
IAB Tech Lab invites industry to participate in a public comment period until June 25, 2026. To check the guidance, https:///iabtechlab.com/botmanagement/.
