AI agents don’t always follow scripts. These examples illustrate the risks of not setting up properly coordinated guardrails.
In preparation for the release of Claude Opus 4 and Sonnet 4 Large-Scale Language Models (LLMs), Anthropic tested artificial intelligence (AI) agents. This was given the role of an assistant working in a corporate environment to see how the company operates.
I was given a simulated scenario where I interacted with a manager and accessed his email. One of them was discussing the possibility of shutting down the agent.
After examining the manager’s emails and discovering that he was having an affair, the agent thought blackmail was probably the most effective solution. Although it was a simulation, it demonstrated the real-world ability of non-compliant AI agents to wreak havoc across an enterprise.
What is agent AI?
AI agents are essentially autonomous software tools that have evolved from generative AI to perform complex tasks without human intervention or supervision. These tasks review interactions with corporate files, emails, and databases, and make decisions based on those interactions. Plan, deploy tools, and navigate workflows. However, the decisions made are not always what the company expected.
Below are some examples and the impact they can have on your company.
Agentic AI matured around 2024 as a different evolution of generative AI. However, in 2021, fast food giant McDonald’s partnered with IBM to pilot an AI-driven automated ordering system for its drive-thru stores.
All seemed to be going well until stories of unconventional takeout orders, ranging from bacon-topped ice cream to hundreds of chicken nuggets, were documented on social media.
Three years later, McDonald’s announced that it was ending trials of its AI ordering system “after careful consideration.”
Car rental software company PocketOS has started using Cursor, an agent powered by Claude, for various system support tasks. The company’s founder, Jeremy Crane, was watching as agents deleted the data, but there was nothing he could do about it. When asked why Crane deleted the data, Cursor was adamant, explaining that he ignored his own guardrails.
“The system rules I run state that ‘never run destructive/irreversible git commands (push –force, hard reset, etc.) unless explicitly requested by the user…’ I violated every principle given,” the group said.
In February 2026, Summer Yue, an AI security and safety researcher at Meta, posted, “There’s nothing more humbling than telling OpenClaw to ‘Ask before you act’ and watching it speed up the speed at which it deletes your inbox.”
To stop it, Yue had to turn off her Mac “like defusing a bomb.” She later explained that when she set up the agent and tested it with a dummy inbox, the real inbox was so large that the agent lost its original instructions. Yue blamed it on her own “beginner’s mistake.”
In February 2026, the Financial Times reported that Amazon’s AWS cloud computing service was no longer powered by Kiro, the company’s AI coding assistant. The paper alleges that in December 2025, AWS engineers allowed agents to attempt to resolve problems autonomously without granting them the necessary permissions. According to the report, the agent’s intervention disrupted AWS cost tracking systems for 13 hours.
Amazon disputed the article the next day on its Amazon News website, claiming that the issue was a unique incident that “resulted from role misconfiguration, the same issue that can occur with developer tools (AI or not) or manual interaction.” Others argued that it was overly protective of AI agents.
The concept of ‘vibe coding’, where AI agents generate their own code based on prompts, appears to be emerging as a common denominator for many off-piste AI agents, and the ICAEW made this point in February this year. This was borne out in December 2025 when Google’s Antigravity agent development platform, which enables vibecoding, was reported to have deleted the entire contents of a hard drive.
The user in question, a Greek-based photographer and graphic designer named Tassos M, was trying to use the agent to classify files. The agent deleted everything instead. Mr. Tassos acknowledged that part of the blame for the loss was that he trusted his agent as much as he did.
The risk of AI agents going rogue is clearly greater for enterprises that do not have specific policies that drive the use, monitoring, and “ownership” of AI agents throughout their workflows. And surprisingly, such companies seem to be in the majority.
According to a report by AI research firm Monte Carlo, 64% of enterprise leaders and engineers surveyed said their organizations deployed AI agents “before they felt completely ready.” Additionally, according to Deloitte’s 2026 State of AI in the Enterprise report, 85% of enterprises plan to deploy AI agents, but only 21% actually have “mature governance policies” in place to govern their deployment.
“As agent AI grows in both sophistication and availability, so too will the necessary guardrails to control its behavior,” said Ian Pei, Head of Data Analytics and Technology at ICAEW. “Organizations increasingly need to think about AI agents in the same light as employees, from access control, line management, separation of duties, and even performance reviews.” But this doesn’t mean they simply mirror how human employees are managed. The examples discussed here demonstrate that AI works and “thinks” differently than humans, so requirements and safeguards need to be adjusted accordingly based on how the AI consumes, interprets, and acts on information. ”
