Features “You can’t blame the box,” says the head of Britain’s financial regulator. What about the people who sold you the box? Good luck, says a global technology analyst.
When AI agents are considered to act on behalf of an organization, the risk of decision-making becomes ambiguous and unpredictable. It also informs the reallocation of AI risk due to unknown parameters
At a time when AI agents promise to “actively run your business,” those looking for an explanation of who is responsible for the output of the statistical machines that supposedly conquer the world may, not unreasonably, land on the above paragraph.
The stakes are high. The largest enterprise application providers are now talking about using AI agents to automate decision-making in human resources, finance, and supply chain management. LLM illusions in performance summaries, inaccurate regulatory filings, and shortages of critical supplies are among the risks weighing on companies that rely on AI for decision-making.
Technology suppliers are eyeing a $1 trillion opportunity in AI, but who will be responsible if things go wrong?
“There’s a historical assumption that if something goes wrong, the vendor is responsible. That’s more or less the starting point for all discussions,” said Malcolm Dowden, senior technology lawyer at Pinsent Masons.
Given vendor claims, it’s no wonder users have high expectations for AI. Oracle announced enhancements to its AI Agent Studio for Fusion Applications, saying the technology “is capable of reasoning, taking actions, and continuously executing processes across business systems,” and that its software “enables businesses to actively run their businesses with the governance, reliability, and security they need.”
However, from a legal perspective, vendors may take a different view.
“If you think about a typical tool or system, its behavior is predictable, so the person giving the assurance has a pretty clear idea of how much responsibility they’re taking on,” Dowden said. “The case with AI is different. “The further down the chain you go, the more you go down the chain (something that falls into the category of AI), the more likely there is to be unexpected behavior. That’s a big concern from a vendor’s perspective if you’re providing guarantees about how something will work.” It does behave, but it is inherently unpredictable, which makes it very unpleasant as a contractual promise. ”
It may also be alarming for the companies using these systems, given what is at stake and the responsibility required of them.
In the UK, for example, this week the Financial Reporting Council (FRC) made its guidance on AI adoption clearer than ever.
“While technology changes, the fundamental principles of our regulatory framework remain the same: people – businesses and responsible individuals – are responsible for the quality of audits.”
Or, as Mark Babington, executive director of the FRC, told the Financial Times: “You can’t blame the box. Even if you use this technology, the onus is on you.”
Nevertheless, technology purchasers can at least try to hold suppliers accountable in terms of their contracts.
For example, users deploying AI to review job applications should be aware that decision-making is automated and may be challenged under data protection law. The Information Commissioner’s Office, the UK’s enforcement agency, recently announced that it supports automation as long as users monitor for bias, are transparent with job seekers and explain their recourse rights.
Mr Dowden said user organizations would be liable for issues such as bias in training models, as they were data controllers under UK law. “They will be looking to reduce vendor liability through contractual clauses that explain how AI works and contractual obligations to ensure there is no inherent bias.”
But vendors are very likely to push back on outright assertions that the bias must be in the model itself, he said. They will want to focus on the interactions between models, algorithms, and user prompts.
“When it comes to negotiated assurances, you’re looking at things like a promise that the system is tested for bias, that the tests are updated regularly, that the model is adjusted, but there’s no liability if the bias can be traced back to how the prompts were created or formulated. Both sides are essentially trying to establish the other party as the responsible party, and that’s where negotiations tend to focus,” Dowden said.
Gartner predicts that by mid-2026, new categories of illegal AI-based decisions will result in more than $10 billion in remediation costs across global AI vendors and enterprises that leverage AI. Gartner Vice President Analyst Lydia Clougherty Jones said decision-making by AI agents could take AI responsibility to a new level.
“When AI agents are assumed to…act on behalf of an organization, the risk of decision-making becomes ambiguous and unpredictable. It also signals a redistribution of AI risk with unknown parameters,” she said.
“Organizations that fail to immediately deploy defensible AI, make AI-enabled data “AI decision-ready,” and overhaul the explainability of ML models are at risk of significant loss of investment, government investigations, civil penalties, and even criminal charges. ”
Clougherty-Jones recommended that users understand the idea of ”defensible AI.” That means focusing on technologies that “reliably and repeatedly stand up to scrutiny, interrogation, and investigation,” including AI-driven decision-making.
Organizations may want to implement content and decision guardrails for language model-based solutions throughout the AI lifecycle, from data to model to output, she said.
Balaji Abbabatulla, vice president at Gartner and principal analyst for Oracle, said last week that there is a lot of legal language that protects vendors from a technology perspective. Instead of legal liability, they talk about monitoring, observability, and auditing.
“The difference between an AI agent’s decisions and a human’s decisions is the scale and pace of those decisions, and those decisions can cascade rapidly,” he said. “If something goes wrong and it’s not identified and prevented, it can quickly cascade before anyone notices the problem. They’re talking about continuous monitoring to identify exceptions, or what we call guardian agents. But issues around liability are a key challenge for all vendors.”
Georgina Kon, digital, data and commercial law partner at Linklaters, says it’s precisely the risk of unnoticed cascading erroneous output that worries vendors about liability.
“The risk of multiples is very high, but it’s difficult to pinpoint blame,” Kong said. “A lot of current law doesn’t really help much, because it always assumes that people or companies are doing something, which isn’t true. But we also can’t have a world where people create agents and don’t have responsibilities to them. It comes down to whether the market is commercially viable.”
For this reason, vendors soft-launched their products and tested them with users first.
As with social media in the early part of this century, it remains to be seen how people will deploy and respond to AI agents, Kong said.
“When you have something like AI, it’s like the top of a hill where you never know what’s going to happen, because these agents can be unexpected and they can successfully learn what they did wrong.The vendor is not responsible for everything. No wonder, but what they can be held accountable for is the process they followed and the safeguards they put in place. There will come a point where it becomes unattractive for a vendor to develop an agent that can take on typical contractual responsibilities.”
However, some users were willing to accept the risk and deploy agents in order to stay on the leading edge of the market or increase the efficiency of their processes. Conn said it varies by sector, but the approach in financial services and healthcare, for example, is more conservative.
AI investment is expected to reach $2.52 trillion this year, with the majority coming from hyperscalers, model building companies, and software companies. They will expect a good return on their spending.
Senior IT managers and directors will attest to the bold marketing claims of vendors that promise to automate internal decision-making at unprecedented speed and scale. But until the law becomes clearer and cases reach the courts, it will remain difficult to hold them accountable for their output.
Major application vendors were given the opportunity to explain how much responsibility they would accept in their customers’ implementation of AI agents. Microsoft and SAP declined to comment. Workday, Salesforce, ServiceNow, and Oracle have not responded. Despite industry hype, reconciling market claims and legal liability remains a challenge. ®
