Sci-fi prophecies of AI agents and robot takeover have never felt more real.
No longer just chatbots that can be companions, AI agents that can act, reason, and complete complex tasks will be deployed at scale in 2026.
But as these autonomous systems become more integrated into workflows, so do the concerns about their unpredictability and the risks they pose to the business.
Sam Gloede, KPMG’s Trusted AI lead, told Business Insider that while organizations are preparing to expand agent systems across the enterprise, clients remain wary of agents.
“Perhaps one of the biggest concerns is how do we prevent them from running amok and taking over, while still allowing them to autonomously do the valuable things we need them to do?”
KPMG has built a multifaceted framework to protect both its clients and its own employees from worst-case scenarios, Grode said.
“A robust set of controls is very important,” she said. Companies must clearly define what agents are allowed to do and ensure that monitoring systems can detect when agents cross those boundaries. Gloede said agents should only interact with systems and data that are strictly necessary to limit the potential impact of errors.
Sam Gloede, Trusted AI Leader at KPMG. KPMG
Every KPMG agent has their own unique identifier and system card, which allows the company to record and monitor actions, track decisions, and track interactions with other agents, Gloede told Business Insider. Monitoring will be done through an AI operations center staffed by both agents and human observers, she added.
Another important step in stress testing systems before problems occur is red teaming, which runs simulated risk scenarios, Gloede added.
Taken together, she said, these measures ensure that agents operate within defined boundaries without manual intervention at all times.
“It’s not about scrutinizing people’s behavior for performance or adjustment,” Grode said. “It’s the ability to constantly monitor the technology ecosystem.”
Incorporate a kill switch – but don’t expect to use it
Beyond technological safeguards, human oversight remains “very important,” Grode said. If agents start to stray from their original roles, there needs to be “a kill switch or fallback option to turn them off.”
That may sound at odds with the promise of autonomy that agents provide, which is one of the key selling points for business leaders. But the level of oversight depends on the risk, Glode said.
Low-risk tasks, such as booking a meeting room or drafting an email, can be automated if proven reliable. High-risk scenarios that could impact financial results or require access to sensitive data require “human beings on the ground,” he said.
Gloede added that if a company has multiple other controls in place, there will be little need to activate a kill switch.
Agent fraud is a major concern for companies
Glode’s comments come at a time when concerns about Terminator-like scenarios are very real.
Earlier this year, we got a glimpse of how weird things can get with the launch of Moltbook, a Reddit-like social network where AI agents can post and interact with each other.
Within hours of the site going live, one agent announced the new cryptocurrency, saying, “Humans can monitor it, they can participate in it, but they no longer have the right to decide.” In other posts, agents were seen questioning their consciousness and founding religions.
Moltbook feels like an internet fever dream, but the stakes are even higher in the corporate world.
Earlier this month, Amazon’s AI coding tools contributed to an error that caused shipping giant Amazon to have about 120,000 lost orders and 1.6 million website errors.
McKinsey, a global consulting firm that helps companies securely implement AI, suffered an embarrassing PR blow last week when a cybersecurity firm announced it had used an AI agent to hack into Lilli, McKinsey’s internal AI platform. The company has positioned itself as an AI expert, with CEO Bob Sternfels saying in January that 25,000 of its 60,000 employees are AI agents.
“McKinsey was recently alerted by security researchers to a vulnerability related to Lilli, our internal AI tool. We immediately identified the vulnerability and fixed the issue within hours,” a McKinsey spokesperson told Business Insider.
The company’s investigation, which was assisted by a third-party forensics firm, found no evidence that customer data or sensitive customer information was accessed, the spokesperson added.
The best way to protect against agent fraud is a multi-pronged approach: technical controls, human oversight, and technology to monitor and manage, KPMG’s Gloede told Business Insider.
“I really believe that if you intentionally build on that and establish an ecosystem of agents, you won’t end up in a situation where they get out of control,” she said.