AI is on track to revolutionize cybersecurity

[SAN FRANCISCO] Late last year, Anthropic announced that state-backed Chinese hackers had used its artificial intelligence technology to break into the computer systems of about 30 companies and government agencies around the world.

Anthropic said in a blog post that this is the first reported case of a cyberattack in which AI technology collected sensitive information with limited help from a human operator. The company said human hackers handled about 10 to 20 percent of the work required to carry out the attack.

Five months later, this remains the only known example of a cyberattack carried out primarily by an “AI agent” – a technology that can write computer code and use software on its own. But as Anthropic and its main rival OpenAI prepare to release new, more powerful AI systems, cybersecurity experts are increasingly warning that AI is fundamentally changing cybersecurity.

Technology from companies like Anthropic, OpenAI, and Google could allow hackers to identify security holes in computer systems much faster than before, greatly increasing the stakes in a decades-long battle between hackers and the security experts who protect computer networks.

But like other tools in the long history of cybersecurity, modern AI can be used both offensively and defensively. As hackers deploy AI to break in and steal, security professionals are also relying on it to discover flaws in systems, even those that have gone unnoticed for decades. The question is, who will find the flaw first?

“This is the biggest change in the cyber landscape ever,” said Francis D’Souza, chief operating officer and president of security products at Google Cloud. “AI must fight with AI.”

Since last year, major open source software projects that provide the underlying infrastructure for sites and services on the Internet have been flooded with messages from people using AI to identify security holes.

Many of these so-called bug reports were false due to mistakes made by the AI ​​system. But in recent months, as AI advances, legitimate bugs have begun to be identified at an alarming rate, and programmers are scrambling to fix them.

“These AI models are extending what humans can do,” said Daniel Stenberg, who runs an important and popular open source project called Curl. “When used correctly, these tools greatly improve your ability to find problems in your software.”

Anthropic announced in February that it had used its AI technology to discover more than 500 so-called zero-day vulnerabilities (security holes unknown to software makers) in a variety of commonly used open source software. The following month, Anthropic researchers revealed that they had used AI to discover a critical security vulnerability at the core of the Linux operating system. The Linux operating system is the software that powers much of the Internet and is used in computer servers, cloud computing services, Android phones, and Teslas.

This bug has been around since 2003, but apparently went undiscovered.

Experts are divided on whether one side of this struggle has gained much from AI. And they are also uncertain about how the battle will unfold in the coming years. However, most agree that businesses and governments that do not deploy modern AI for defense purposes are leaving themselves highly vulnerable.

Chatbots like Anthropic’s Claude and OpenAI’s GPT have become very good at writing computer code. These systems help engineers create new software. You can use Internet tools such as email programs and online calendars. You can also investigate weaknesses in software and online services to look for security vulnerabilities.

Over the past few months, new AI tools made specifically for coding, such as Anthropic’s Claude Code and OpenAI’s Codex, have helped developers create AI agents that can handle a variety of tasks primarily on their own. This involves identifying and exploiting security holes in software.

“Four or five months ago, there was a big shift in what you can do with these systems,” said OpenAI board member Zico Colter, a Carnegie Mellon University computer science professor who specializes in security and AI.

AI is helping attackers in other ways. Cybersecurity experts say some people are using chatbots to create phishing emails and ransom notes. Some are using AI to parse large amounts of stolen data to determine what information is valuable. Without the help of AI, an attacker might be able to break into a computer network within minutes, but with the help of AI, D’Souza said, it can be done in just seconds.

Some hackers specialize in breaking into systems and selling access to other attackers. DeSouza said these handoffs could take up to eight hours as the hackers negotiated sales and handed over compromised entry points. Now, that process has been sped up to about 20 seconds, and hackers sometimes use AI agents to speed up the process, he said.

Anthropic, OpenAI, and other AI companies are trying to add guardrails to their tools to prevent them from being turned into cyberweapons. However, attackers were able to bypass these barriers by telling the AI ​​system that they were not actually attacking.

For example, they’ll say they’re just playing a “capture the flag” game. This is a cybersecurity exercise that simulates real attacks and allows engineers to practice finding and exploiting vulnerabilities.

Some experts argue that the guardrails added by companies like Anthropic and OpenAI could actually be an advantage for malicious attackers. They argue that while guardrails could allow chatbots to deny assistance to users trying to protect their systems from attack, persistent hackers could be more diligent in finding vulnerabilities and keep their tricks secret.

“Claude is built with strong safeguards to prevent misuse of our model,” Anthropic spokesperson Palul Maheshwary said in a statement. “As the barriers to conducting sophisticated cyberattacks continue to fall, we believe these protections are essential to prevent AI from being used as a tool by attackers.”

AI technology has given offensive hackers new powers, but experts are divided on whether these tools give attackers an overall advantage over defenders.

Even after months of steady improvement, AI technology still has flaws, meaning it requires the expertise of experienced cybersecurity professionals. Tools are often still limited by the skill of the person using them.

“You still need a software architect for these systems,” Colter says.

He and others argue that defenders have an advantage because their jobs are easier. They just have to find the hole. An aggressive hacker must find that hole and exploit it.

“It’s easier to find vulnerabilities than it is to meaningfully exploit them,” Colter said.

Anthropic sparked a new debate across the cybersecurity community last month when Fortune reported that the company had mistakenly published the contents of a blog post describing an AI system it had not yet released. The blog post said the technology represents a new “step change” in AI performance.

In response to the leaked blog post, Nikesh Arora, CEO of cybersecurity company Palo Alto Networks, published his own blog post warning businesses and governments about the need to adopt modern tools.

“Our ability to identify vulnerabilities will be better than ever before,” he said in an interview. “We must be prepared to solve these problems.” New York Times