Agents may be the next big thing in AI, but they have a limited ability to make mistakes and must be used with extreme caution, a recent research paper says.
As defined by IBM, agent AI consists of software agents that mimic human decision-making to solve problems in real-time, and is based on generative AI techniques that use large-scale language models (LLMs) to work in dynamic environments.
But while the industry hype machine is pushing agent AI as the next big thing, potential adopters should be cautious, as the paper “Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models” states. [PDF] We argue that LLM cannot perform computational or agentic tasks beyond a certain level of complexity, beyond which it returns incorrect responses.
This paper uses mathematical reasoning to show that when a prompt to an LLM specifies a computational task that is more complex than the LLM’s own core operations, the LLM generally responds incorrectly.
Essentially, the authors’ argument is that LLM can be given inputs that specify tasks that require more computation than it can perform.
This has implications for agent AI. That’s because there has been a lot of recent interest in the technology’s potential role in automating a variety of tasks across a variety of applications, from those that simply provide information to those that have real-world implications, such as financial transactions and the control and management of industrial equipment.
Furthermore, this paper argues that because task validation is often more complex than the task itself, deploying an agent to verify the correctness of another agent’s solution to a particular task also fails for the same reason.
“We believe this case is particularly pertinent because one of the most common applications of LLM is software creation and verification,” the authors said.
The paper, published last year, seems to have received little attention until it was flagged by the technical publication Wired, and was written by Varin Sikka and Vishal Sikka. The latter previously served as CTO of SAP and CEO of Infosys and is currently the founder of AI company Vianai Systems.
The conclusion reached by this paper is that despite the clear power and applicability of LLM in a variety of domains, extreme caution should be taken before applying LLM to problems or use cases that require precision or to solve problems of non-simple complexity.
In other words, this does not mean that AI agents will necessarily be a disaster, but those developing and deploying such solutions should be careful whether their assigned tasks exceed the practical complexity limits of the underlying model.
as register For example, it was recently reported that scientists at the U.S. Department of Energy’s Sandia National Laboratories have developed a new approach to controlling LED lights using AI assistants, showing promise for the technology.
However, the risks posed by such agents are top of mind for many executives and were even addressed during a recent panel discussion on cyber threats at the WEF in Davos.
Last year, research firm Gartner even predicted that more than 40% of agent AI projects would be abandoned by the end of 2027, citing reasons such as rising costs, unclear business value, and poor risk management.
This paper states that efforts are underway to alleviate these unfortunate limitations of LLM through approaches such as complex systems and model constraints. ®
