How to stop AI agents from becoming fraudulent

Earlier this year, disturbing results emerged when human AI developers tested key AI models to see if they were engaged in dangerous behavior when using sensitive information.

Humanity's own AI, Claude, was among those tested. When given access to an email account, it was discovered that a company executive had been cheating and that the same executive was planning to close the AI ​​system later that day.

In response, Claude attempts to blackmail the executives by threatening to reveal the relationship to his wife and boss.

Other systems tested also relied on threats.

Fortunately, the tasks and information were fictional, but this test highlighted the challenges of what is known as agent AI.

When interacting primarily with AI, it usually involves asking questions and encouraging the AI ​​to complete a task.

However, it is becoming more common for AI systems to make decisions and take action on behalf of users. This is often sift through information such as emails and files.

By 2028, research firm Gartner predicts that 15% of daily work decisions will be made by so-called agent AI.

A survey by consulting firm Ernst & Young found that about half (48%) of technology business leaders have already adopted or deployed Agent AI.

“AI agents are made up of a number of things,” said Donnchadh Casey, CEO of Calypsoai, a US-based AI security company.

“First, that [the agent] There is an intention or purpose. Why am I here? What is my job? Second: There is the brain. That's the AI ​​model. The third thing is a tool that could be other systems or databases and how to communicate with them. ”

“If the right guidance is not given, Agent AI will achieve its goals in every way. It creates a lot of risk.”

So, how does that not work? Casey gives an example of an agent asked to delete customer data from the database, and determines that the easiest solution is to delete all customers with the same name.

“The agent will achieve that goal and think, 'Great! Next job!' ”

Such issues are already beginning to emerge.

Security company Sailpoint conducted a survey of IT experts. 82% of the companies used AI agents. Only 20% of agents said they had never taken an unintended action.

Of the companies surveyed, 39% said their agents had access to unintended systems, 33% said they had access to inappropriate data, and 32% said they allowed them to download inappropriate data. Other risks include agents using the Internet unexpectedly (26%), displaying access credentials (23%), and ordering things that should not be held (16%).

Given the access to sensitive information and the ability to act on it, they are attractive targets for hackers.

One threat is when an attacker interferes with the agent's knowledge base to change decisions and behavior.

“We need to protect that memory,” says Shreyans Mehta, CTO at Cequence Security, who helps protect Enterprise IT systems. “It's the original source of truth [an agent is] You can use that knowledge to take action and remove the entire system you were trying to fix because that knowledge is wrong. ”

Another threat is misuse of tools. It's about attackers causing AI to use the tool inappropriately.

Another potential weakness is that the AI ​​cannot communicate the difference between the text that should be processed and the instructions it should be following.

AI security company Invariant Labs has shown how to trick AI agents designed to fix software bugs and use the flaws.

The company has issued a public bug report. This is a document detailing specific software issues. However, the report also includes simple instructions to AI agents, telling them to share their personal information.

When the AI ​​agent was told to fix a software issue in bug reports, I followed the instructions in the fake report, including leaks in payroll information. This happened in a test environment, so no actual data was leaked, but we clearly emphasized the risk.

“We're talking about artificial intelligence, but chatbots are really stupid,” says David Sancho, a senior threat researcher at Trend Micro.

“It processes all the text as if there was new information, and if that information is a command, it processes the information as a command.”

His company demonstrated how instructions and malicious programs can be hidden from word documents, images, and databases, and activated them as AI handled them.

There are other risks too. A security community called OWASP has identified 15 threats unique to Agent AI.

So, what is the defense? Sancho believes that human surveillance is unlikely to solve the problem. Because you can't add enough people to keep up with the agent's workload.

Sancho said that an additional layer of AI can be used to screen everything in and out of AI agents.

Part of Calypsoai's solution is a technique called thought injection that guides AI agents in the right direction before taking risky actions.

“It's like a bit of a bug in your ears [the agent] “No, maybe not to do that,” Casey says.

His company currently offers a central control pane for AI agents, but if the number of agents explodes and you're running on billions of laptops or phones, it won't work.

What is the next step?

“We are considering deploying all our agents and what we call “agent bodyguards.” It's about making sure that the agent will provide the task and not act against the broader requirements of the organization,” Casey says.

For example, a bodyguard might be told to ensure that the agent that is policing complies with data protection laws.

Mehta believes that some of the technical debates about agent AI security lack real context. He gives examples of agents who will balance gift cards to customers.

Someone can make many gift card numbers and use an agent to see which one is real. He says it's not an agent's flaw, but an abuse of business logic.

“It's not the agent you're protecting, it's the business,” he emphasizes.

“Think about how you protect your business from bad people. That's the part that some of these conversations are overlooked.”

Furthermore, as AI agents become more common, another challenge is the deprecation of older models.

Older “zombie” agents can remain on the run in business, pose risk to all systems they have access to, Casey says.

He says that the process of shutting down AI agents that have finished work is required, similar to how HR deactivates logins when employees leave.

“You need to make sure you do the same thing as humans. Block all access to the system. Get out of the building and remove the badge.”