In the age of AI, Silicon Valley’s “move fast and break things” ethos is proving to be literal.
Earlier this week, Business Insider reported that Amazon had put up new guardrails after a series of failures. This included failures primarily caused by the company’s AI coding tools, which led to approximately 120,000 lost orders.
Similar failures have occurred at other companies deploying AI. In January, the founder of an events company said his AI agent made four mistakes in one week, including handing out free tickets. And last summer, the CEO of a browser-based coding platform apologized after an AI agent wiped a client’s codebase and lied about it.
These incidents highlight the delicate balancing act required for employers keen to leverage AI. Too tight a grip on the worker will make experimentation difficult. If you loosen the reins too much, the risks from errant AI agents and poorly reviewed code can quickly increase.
“You need to know your own risk tolerance,” says Matt Rosenbaum, principal researcher at The Conference Board, a nonprofit that provides data and insights for business leaders. “You also need to know what to do if things go wrong and what needs to be changed to prevent it from happening again.”
Speed and power, unchecked
Part of the challenge is that software developers aren’t expected to write as much code as they used to, said Todd Olson, CEO and co-founder of Pendo, an AI startup that helps companies improve user experiences. A large part of a developer’s job now shifts to reviewing code written by AI, he said.
“These are completely different skill sets and different habits,” Olson told Business Insider.
Another problem: Because AI can generate code in seconds, employees racing to meet deadlines may be tempted to accept the output at face value, increasing the risk that mistakes will go unnoticed.
A global study by KPMG and the University of Melbourne found that around two-thirds of employees accept AI-generated work without carefully checking it, and 72% say AI has reduced their effort at work. The findings are based on a survey of more than 30,000 workers between November 2024 and January 2025.
“The lesson that companies are learning is that speed without large-scale analytical discipline can put them at systemic risk,” said Lauren Buitta, founder and CEO of Girl Security, a nonprofit that prepares young women for careers in national security.
Uncertainty surrounding the rapidly expanding capabilities of AI adds further complexity. As tools become more powerful and accessible, employees may push their limits without fully understanding the downstream consequences.
“Just because you can do something doesn’t mean you should,” says Kevin Serwatka, founder of recruiting platform Benchmarket and previously held recruiting leadership roles at companies like Google, Meta, and Robinhood.
The lesson from these mistakes, he said, is not to discourage experimentation, but to “put guardrails around what that looks like at your company.”
light of hope
Olson said Amazon’s outage, while painful, likely served as a lesson for the company.
“They probably just found a bunch of test cases that they could train the AI on, so the AI could review these things in the future,” he says.
Other companies that use AI to write code are also likely to make mistakes, which is a natural part of experimentation, said Andrew Filev, founder and CEO of coding agency Zencoder.
“Small issues are actually good,” he said, but ideally they would be identified and addressed internally rather than exposed to customers. “People will learn and improve their guardrails and systems.”
Mr Filev said it was important to remind employees of the importance of speaking up about errors discovered by AI, noting that if problems were ignored it could lead to “an accident with a much larger explosive radius”.
Filev said that to achieve AI autonomy, we need to start by combining AI with human auditing.
“Both processes will need to operate in parallel for a period of time until AI reviews are at least on par with human reviews,” he said.