Anthropic said Chinese state hackers took over its AI model, Claude, and carried out cyberattacks without “substantial” human involvement.
The company said in a blog post on Thursday that Claude handled “80 to 90 percent” of cyberattacks on about 30 targets around the world, and that it had “high confidence” that Chinese state-backed groups were behind them.
Targets include large tech companies, financial institutions, chemical manufacturers and government agencies, Anthropic said. Efforts to infiltrate these companies and agencies were successful in “a small number of cases,” the company added.
AI agents (programs that can perform tasks autonomously) are increasingly being employed by businesses to handle repetitive tasks such as customer support tickets. These can increase the productivity of white-collar workers, but they can also be recruited for illegal jobs. Anthropic announced in August that it had detected and thwarted cybercriminals using Claude to carry out hacking operations with small teams.
While AI has been used to some degree in hacking operations over the years, Anthropic said it believes this new operation is the first documented case of a “large-scale” cyberattack carried out primarily by AI.
The Amazon-backed startup said Claude has safeguards in place to prevent abuse. But by breaking Claude’s requests into smaller chunks, the hackers were able to successfully jailbreak him without raising any alarms, Anthropic said. It added that the hackers were pretending to be conducting defense tests for a legitimate cybersecurity company.
The attackers then used Claude code to spy on the target company’s digital infrastructure, writing code that penetrated defenses and extracted data such as usernames and passwords.
Anthropic said it will share its findings with the public to help the cybersecurity industry better defend against AI-powered hacking attacks.
“The enormous amount of work performed by AI would take a human team an enormous amount of time,” Anthropic said in a blog post. “The AI sent thousands of requests per second, an attack rate that no human hacker could match.”
OpenAI and Microsoft have also shared reports of nation-states using AI during cyberattacks, but in those cases the technology was primarily used to generate content or debug code rather than autonomously performing tasks.
Jake Moore, global cybersecurity advisor at internet security company ESET, told Business Insider that the incident was not surprising.
“Automated cyberattacks can scale much faster than human-led operations and have the potential to overwhelm traditional defenses,” he said. “Not only is this something that many have been concerned about, but these attacks have broader implications by allowing very low-skilled attackers to launch complex intrusions at relatively low cost.”
While AI facilitates attacks by cybercriminals and nation states, it is also seen as part of a defensive solution.
“As AI is being used to attack as well as defend, security now relies just as much on automation and speed as it does on human expertise across the organization,” Moore said.
