Amazon Web Services has reportedly launched an investigation into whether Perplexity AI is violating its rules. WiredTo be precise, the company's cloud division is reportedly investigating allegations that crawlers hosted on the company's servers are ignoring the Robots Exclusion Protocol, a web standard where developers place robots.txt files on their domains with instructions on whether bots can access certain pages. Following these instructions is optional, but crawlers from reputable companies have generally respected them since web developers began implementing the standard in the '90s.
In a previous article, Wired The company reported finding a virtual machine that was circumventing robots.txt directives on its websites. The machine was hosted on an Amazon Web Services server using the IP address 44.221.181.252 and was “surely operated by Perplexity.” The machine reportedly visited other Condé Nast sites hundreds of times over the past three months and scraped their content as well. of Guardian, Forbes and The New York Times We also detected multiple visits to their publications. Wired To verify whether Perplexity was really scraping content, Wired Users entered the headline and a short description of an article into the company's chatbot, and the tool returned a mostly paraphrased version of the article, “with minimal citations.”
Recent Reuters Perplexity is not the only AI company that is circumventing robots.txt files to gather content used to train large-scale language models, according to the report. Wired It only provided Amazon with information about Perplexity AI's crawlers. “AWS' terms of service prohibit abusive or illegal activity, and our customers are responsible for complying with these terms,” Amazon Web Services said in a statement. “We regularly receive reports of suspected abusive activity from a variety of sources and expect our customers to understand those reports.” The spokesperson added that the company's cloud division told Amazon: Wired The company was investigating the information provided by the magazine, as it does with any report of potential violations.
“Perplexity is a game changer,” said Perplexity spokeswoman Sarah Platnick. Wired The company has already responded to Amazon's inquiries, denying that its crawlers are circumventing the robots exclusion protocol: “Our PerplexityBot, running on AWS, respects robots.txt, and we have confirmed that Perplexity-managed services do not crawl in a manner that violates AWS's terms of service,” she said. Platonic said Amazon has investigated. wired The media investigation was conducted solely as part of standard protocol for investigating reports of resource misuse, and the company appears not to have heard of any kind of investigation from Amazon to date. Wired I contacted the company. Platonic said: WiredHowever, if a user includes a specific URL in their chatbot inquiry, PerplexityBot will ignore the robots.text.
Perplexity CEO Aravind Srinivas also previously denied that his company was “ignoring robot exclusion protocols and lying about them.” Fast Company Perplexity uses its own web crawlers as well as third-party web crawlers. The bots Wired One of them was identified.
Updated June 28, 2024 at 2:20 PM: We've updated this post with Perplexity's statement to Engadget.
Updated June 28, 2024 at 8:27pm: We have updated this post with a statement from Amazon Web Services.
