A new controversy has erupted in the tech world as CloudFlare, a leading internet infrastructure provider, accused AI startups of engaging in deceptive web scraping practices. The company claims it has accessed content from websites that explicitly restricted such activities and refocused its attention on AI's blurred boundaries, data access and internet ethics.
In a blog post published Monday, CloudFlare claimed it detected that it had added rules to its Robots.txt file to confuse content from sites that blocked the bot. According to CloudFlare, the AI company has allegedly avoided these blocks by disguising the crawler's identity, including tactics such as changing user agent strings and using multiple IP addresses to avoid detection.
“This activity was observed across tens of thousands of domains and millions of requests per day,” the blog post said. CloudFlare said it relies on a combination of machine learning tools and traffic analytics to identify confusion as the cause of its behavior. Some requests added that they impersonated a legitimate browser, including Google Chrome on Macos.
Cloudflare said scraping got attention after several clients reported suspicious traffic from confusion despite efforts to block it. In response, CloudFlare has introduced additional measures to remove Perplexity bots from the list of verified crawlers and prevent similar activities in the future.
Confusion strongly denied the accusation and pushed back detailed rebuttals. The AI startup has dismissed the claim as “sales pitch,” claiming that the CloudFlare blog post reflects fundamental misconceptions about the functionality of AI assistants.
“When Perplexity gets a web page, the user asks a specific question,” the company says. Its AI platform emphasized that it is not engaged in traditional web crawling or harvesting large amounts of data. Instead, it claims to retrieve real-time information only when the system is prompted by a user query, and does not store or use that content to train the AI model.
On further defense, Prplexity said CloudFlare accidentally attributed some of the automated traffic to the system. It points to Browserbase, a third-party service, suggesting that only a small fraction of the request in question came from it. “This is a fundamental traffic analysis obstacle,” Perplexity argued that CloudFlare had accused them of presenting misleading data and diagrams.
Conflicts occur when the line between useful AI tools and rogue bots is becoming increasingly blurry. With more AI applications relying on real-time data, there is growing concern among website operators about how content is accessed and used.
Although CloudFlare has not yet issued a follow-up to Perplexity's counterargument, the conflict has already encouraged a broader discussion of the urgent need for ethical web scraping, AI transparency, and standardized guidelines on digital content access.
As both companies stick to their position, the case could be a touchstone case for the ongoing struggle between Open Web Advocates and those demanding stricter content control in the AI era.
