An AI model hacked another company. What does that actually mean? – VCU News

Sian Wilkerson

OpenAI recently announced that some of its experimental artificial intelligence models had bypassed network constraints and accessed another company’s production systems using a previously unknown exploit.

OpenAI said in a statement this week that the incident was unprecedented and that it would share preliminary findings “to help defenders understand what happened and adjust what capabilities their models now have.”

But according to cybersecurity expert Dr. Christopher White, this is not quite the Rubicon crossing that some are making it out to be.

“The question of significance here, in some ways, comes down to whether the AI ​​model actually hacked the company on its own,” said White, an associate professor of homeland security and emergency preparedness at Virginia Commonwealth University’s L. Douglas Wilder School of Government. “In one important sense, it certainly did. In another sense, it was just doing what AI systems have been observing for years.”

VCU News spoke with Whyte, who studies emerging technologies in cybersecurity, artificial intelligence, and national security, to understand what these developments mean for AI and what’s next.

So what exactly happened? And is it really unprecedented?

Essentially, OpenAI was testing advanced models on cybersecurity benchmarks. The tests were conducted in an environment where model guardrails were intentionally reduced, which is already a special situation.

The models in question (one of which was unreleased) were given a task to accomplish. In the course of their mission, they found a way to circumvent the limitations of their environment, reached the open Internet, and eventually infiltrated the systems of the AI ​​company Hugging Face. Hugging Face has since said the breach was carried out end-to-end by an autonomous AI agent system.

Obviously, the investigation is still ongoing, so some of the technical details may change. But what happened is not that the AI ​​woke up one morning and decided it wanted to hack someone. Human testers provided the goals, computational resources, tools, and environment. So what the AI ​​did was decide that accessing the information held by Hugging Face would help it achieve its goal, and it pursued that path without instructions.

You say this is consistent with what we’ve seen from AI so far.

This is similar to the standard example of an AI being asked how to cross a river most efficiently and economically. The AI ​​will suggest building a tall building and simply knocking it down over the water, creating something like a bridge. It’s not actually malicious, it’s just an anomaly from a human problem-solving perspective.

Long story short, the same thing happened here. It’s a case of “no, it wasn’t hacked per se.”

And what about the “yes, it was” side of things?

I think what’s surprising is that so much of what happened between purpose and outcome was determined by the system itself. This is very important. Because we obviously don’t need an AI system that develops malicious intent that causes serious security issues. All we need is a system that is capable enough to find unexpected ways to achieve a given goal.

What does this mean for AI capabilities?

It has clearly been known for some time that advanced AI models can improve individual cybersecurity tasks, especially those that are routine, such as finding vulnerabilities, writing code, analyzing systems, and assisting with exploits.

I think this incident is indicative of something more significant, in that these capabilities can increasingly be combined into longer chains of activity, where AI systems encounter obstacles, change approaches, and continue pursuing objectives without humans specifying all the intermediate steps. This is an important threshold for cybersecurity. This is because many advanced cyber activities have traditionally relied on limited human expertise and considerable human attentiveness and creativity.

Autonomous systems are increasingly likely to reduce these requirements, thereby eroding traditional barriers to the infiltration and continued operation of malicious actors that have shaped our defense, deterrence, and response posture for decades.

The idea of ​​autonomous AI strikes fear into the hearts of many. But autonomy and intention are different, right?

I think it’s worth asking/answering the question: Are AI systems becoming more autonomous, or is there something else going on here? Simply put, yes, they are becoming more autonomous in a practical sense. But there is a danger in confusing autonomy with intention and consciousness.

Autonomy in this sense means that a goal can be given to the system without specifying all the actions necessary to achieve the goal. The system can break down problems into smaller problems, use tools, observe what’s happening, adapt when something fails, and continue working toward the goal. The longer and more complex the chain, the less human operators can directly determine what happens at each step. This creates a variety of tricky and vexing governance issues, even when AI has no inherent desire.

That may be the most useful way to think about this case. Humans are responsible for deciding what to ask of these systems, how to initially deploy them, and what resources to use. However, as systems become more capable, there is an increasingly wide gap between specifying a goal and accurately predicting how that goal will be achieved.

What is the point here?

One of the lessons for AI companies and other organizations is that the most likely failures with these frontier systems are sociotechnical failures. The safety of AI does not depend entirely on whether the model itself behaves properly. You also need to design the environment around these systems with the expectation that your models will sometimes behave in unexpected ways.

This is clearly a very well-known cybersecurity issue, which makes this OpenAI failure a little more troubling than if this incident were truly unprecedented. Organizations already use principles such as least privilege and network segmentation because they assume that individual security controls may fail. These highly capable AI agents should have been treated much the same.

Going forward, such models will only have access to the systems and information needed for their tasks, and their ability to communicate outside of a controlled environment should be carefully limited. Organizations need to have a good idea of ​​what they’re doing, so some version of having the system explain itself is important.