Business risk of almost correct code

Anthropic CEO Dario Amodei’s prediction in March 2025 that eventually all coding will be generated by AI is already coming true. Amodei said human software developers will need to train LLMs with design features and conditions, but eventually every task will be automated.

In fact, in a January 27 post on X, Boris Cherny, creator and head of Anthropic’s Claude code, admitted that “almost 100%” of the company’s code is now generated by AI. While timelines for AI adoption often lean toward overly optimistic trends, the transition from human code to AI-generated code is not.

AI coding agents are generating an increasing amount of code. Some analysts predict that 40-60% of today’s code is already generated by AI. And while the popular narrative focuses on improving efficiency, AI code generation fails in practice. It is not yet clear how often and what the consequences will be.

Roman Zednik, field CTO at Tricentis, says code validation and quality control are becoming increasingly important as the amount of code generated by AI increases. As systems become more complex, tools can check syntax and basic security patterns, but it becomes much more difficult to verify that code works correctly when integrated into complex enterprise ecosystems with multiple backend systems, interfaces and supply chains, he says.

Small changes generated by AI can have unpredictable impacts on the entire system, especially in high-risk environments such as banks, insurance companies, and telecommunications companies. Developers need to ask the question, “If I change this small piece of code, what impact will it have on the entire ecosystem?” Zednik says.

“Furthermore, AI often generates extra code that is not in the specification and can be functionally unnecessary and semantically nonsense in a business context,” Zednik points out. The question then becomes: should this extra code be tested or should it be removed?

Zednik says that while AI can speed up source code generation, it doesn’t automatically increase the quality of the code commensurately. While developers can use AI for simple tasks, more complex tasks, especially when it comes to system integration, still require human coding.

Big bottleneck in AI code testing

Although AI speeds up code generation, that code still needs to be reviewed and tested, increasing the total testing effort and potentially creating quality assurance bottlenecks.

In 2021, OpenAI developed HumanEval, a widely used benchmark for evaluating LLM code generation capabilities. It was designed to assess whether AI can write functional Python code based on natural language instructions.

At the time of publication, Stanford University’s 2025 Artificial Intelligence Quotient report found that Claude 3.5 Sonnet (HPT), Anthropic’s advanced coding and development tool, was the leader in HumanEval performance, achieving a score of 100%.

However, the Stanford study, like any performance benchmarking exercise, does not necessarily reflect real-world coding accuracy. Because it only tests benchmark performance on specific test parameters, it may not take into account the rapid developments in AI modeling.

Professional Reasoning Bench (PRBench), an independent benchmarking organization, examines complex real-world financial and legal questions written by experienced industry experts. According to PRBench, the best available AI models only score 39% for hard finance tasks and 37% for hard legal tasks, indicating that industry expectations often overestimate AI capabilities in specific professional applications.

AI coding agents can generate their own test systems, but can these test environments be trusted? They still require human oversight. And the challenge is not necessarily verifying that the code works, but whether enterprise engineers can control and secure the code at scale in high-risk environments.

For many organizations, especially those that still do manual testing, their testing capabilities cannot, and likely never will, scale to the large amount of new code generated by AI.

“A surprising number of really large companies still rely on manual testing,” Zednik says, advising companies to move to automated processes as soon as possible. “If we don’t, we won’t be able to deliver new features on time and lose out to our competitors over the long term. We’re already seeing this happen.”

Does AI code save labor?

If a company relies on manual testing and requires human involvement in the code, AI and human job arbitrage may not generate the savings that the company expects.

Many companies have cited AI as a reason for layoffs in the last year, but the data on this is still inconclusive and the evidence is more anecdotal. “The advent of intelligence tools has changed what it means to start and run a company,” Jack Dorsey, CEO of Square’s parent company Block, said in a letter to shareholders on February 26. Dorsey fired 40% to 50% of Block’s employees in February.

“We’re already seeing that internally. The tools we’re building allow significantly smaller teams to do more and do better. And the capabilities of our intelligence tools are rapidly evolving every week.”

Still, as of March 2025, U.S. Bureau of Labor Statistics estimates that employment of software developers in the U.S. is expected to grow by 17.9% from 2023 to 2033, from 1.69 million to 1.99 million. That growth rate is more than four times the average growth rate for occupations.

AI-generated code opens a can of security worms

Kevin Curran, professor of cybersecurity at Ulster University and co-founder of Vaulttree, is amazed at the speed at which AI code generation is improving, even as “huge errors and vulnerabilities” are beginning to surface.

Curran describes agent AI as opening a can of worms. “Unfortunately, we’ve opened up an attack surface that relies on documents that come back.” [from queries]And this leads to an immediate injection attack. “Because we had our agents do a detailed investigation that would have taken several weeks,” he explains.

“But they’re at the mercy of prompt injection instructions that steal data, wreak havoc on local systems, and open up endpoints. And we’re just clicking permit, permit without even knowing what we’re being granted permission for,” he explains.

Additionally, AI-generated code can be very bloated. “Because there are so many dependencies, potentially tens of thousands of lines, the attack vector can be large,” Curran said. And it’s becoming increasingly difficult for human code auditors to identify insecure code. Even with established security processes, the amount and complexity of AI-generated code makes it very easy for vulnerabilities to slip through.

Amazon’s “Almost Correct” Code Prank

In early March, the Financial Times reported that flaws in AI-generated code led to numerous outages at Amazon.com. The company fired back in a blog post on March 11, claiming, “In fact, only one of our recent incidents was related in any way to our AI tools, and in that case, the cause was unrelated to AI, and instead our systems allowed our engineering team’s user error to have a broader impact than it should have.”

Despite Amazon’s defensive stance, Niranjan Vijayaraghavan, chief product and technology officer at Nintex, said the incident proves that even large companies can be affected by AI-assisted code changes.

“A small, plausible-looking change can slip through review, behave differently in production than it did in the sandbox, and ripple through the environment faster than the team can detect and roll back. But this is more of a governance issue than a code failure,” he says.

Organizations are layering AI over undocumented and inconsistent processes and expecting consistent results, says Vijayaraghavan, who advises treating AI tools as power multipliers within a managed delivery process, including clear ownership and accountability, automation-driven controls, strong testing in a controlled environment, and human oversight of high-impact changes.

“If you can’t track and manage how AI-assisted changes move from development to deployment and where you can intervene when something goes wrong, you’re not ready to scale,” he says.