The developer of the artificial intelligence (AI) chatbot Claude claims to have captured Chinese government-backed hackers who used the tool to carry out automated cyberattacks against around 30 global organizations.
Anthropic said the hackers tricked the chatbot into performing automated tasks under the guise of conducting cybersecurity research.
The company claimed in a blog post that this was “the first reported cyber espionage operation orchestrated using AI.”
But skeptics question the accuracy of that claim and the motives behind it.
Anthropic said it discovered the hacking attempt in mid-September.
The hackers pretended to be legitimate cybersecurity workers and gave the chatbot small automated tasks, which they combined to form a “highly sophisticated espionage operation.”
Anthropic researchers said they had “high confidence” that the people behind the attack were “Chinese state-sponsored groups.”
They said humans selected the targets, including large tech companies, financial institutions, chemical manufacturers and government agencies, but the company did not provide further specifics.
The hackers then used Claude’s coding assistance to build an unspecified program that “autonomously compromised selected targets with little human interaction.”
Anthropic claims that its chatbots have successfully infiltrated various unnamed organizations, extracted sensitive data, and cataloged valuable information.
The company said it has since banned the hacker’s use of the chatbot and notified affected companies and law enforcement.
However, Martin Zugec of cyber firm Bitdefender said the cybersecurity industry had mixed feelings about the news.
“Anthropic’s report makes bold and speculative claims, but provides no verifiable threat intelligence evidence,” he said.
“While the report highlights areas of growing concern, it is important that as much information as possible is provided about how these attacks occur so that the true risk of AI attacks can be assessed and defined.”
Anthropic’s announcement is perhaps the most high-profile example of a company claiming that malicious actors are using AI tools to perform automated hacking.
This is the kind of risk many have feared, but other AI companies have also claimed that state hackers have used their products.
In February 2024, OpenAI published a blog post in collaboration with Microsoft cyber experts stating that it had thwarted five nation-state-linked attackers, including participants from China.
“These attackers commonly sought to use OpenAI services to query open source information, translate it, find coding errors, and perform basic coding tasks,” the company said at the time.
Anthropic did not say how it concluded that the hackers in this campaign were affiliated with the Chinese government.
The Chinese Embassy in the United States told reporters that China was not involved.
This comes as some cybersecurity companies have been criticized for overhyping cases where AI has been used by hackers.
Critics argue that the technology is still too cumbersome to use in automated cyberattacks.
In November, Google cyber experts published a research paper highlighting growing concerns that AI could be used by hackers to create entirely new forms of malicious software.
However, the paper concludes that these tools have not been very successful and are only in the testing phase.
Like the AI industry, the cybersecurity industry contends that hackers are targeting companies using this technology to increase interest in their products.
Anthropic argued in a blog post that the answer to thwarting AI attackers is to leverage AI defenders.
“The very ability to use Claude for these attacks is also important for cyber defense,” the company claimed.
And Anthropic admitted that its chatbot made a mistake. For example, they fabricated fake login usernames and passwords and claimed to have extracted sensitive information that was actually publicly available.
“This remains a barrier to fully autonomous cyberattacks,” Anthropic said.
