Amazon is tightening internal guardrails after recent outages, including one related to its AI coding assistant Q, have hit its e-commerce business.
Dave Treadwell, Amazon’s SVP of e-commerce services, told employees on Tuesday that a “trend in incidents” has been emerging since the third quarter of 2025, including “several significant” incidents that have occurred in the past few weeks, according to an internal document obtained by Business Insider. At least one of these disruptions is related to Amazon’s AI coding assistant Q, while others reveal deeper problems, another internal document explained.
The issues included what he described as a “major change in the blast radius,” and the lack of adequate protection in the control plane caused the software update to propagate far and wide. (The control plane guides how data flows on a computer network).
Additionally, data corruption could take several hours to recover. Some failures were due to basic mechanisms being missing or bypassed, such as requiring two people to approve code changes.
In response, Amazon is introducing stricter controls that require engineers to more thoroughly document code changes and secure additional approvals. At the same time, the company is developing other safeguards aimed at introducing what executives call “controlled friction” into the code change review process.
“We are implementing temporary safety measures to introduce controlled friction into changes to the most important parts of the retail experience,” Treadwell said in a statement Tuesday. “In parallel, we will invest in more durable solutions that include both deterministic and agentic safeguards.”
Amazon’s snafus is an example of how generative AI is changing the way software is created, checked, and shipped. AI coding services like Claude Code and Amazon’s Q and Kiro services help engineers write far more code than before. However, this code should be checked for bugs and other potential problems before it is released to the world. Problems can arise when this avalanche of new code impacts traditional software review processes.
“Agential” vs. “Deterministic”
Amazon’s Treadwell wrote that the company’s new code guardrails combine AI-driven “agent” tools with more predictable, rules-based “deterministic” systems.
This solves one of the central problems with AI models. These powerful new services are not definitive. This means that if you ask the same question twice, the AI model may give slightly different answers. This may make this technology inappropriate for enterprise workflows that need to be 100% accurate at all times. This includes core enterprise resource planning software systems that can be applied to important things like product, price, order, and transaction data in large e-commerce marketplaces, such as those operated by Amazon.
Earlier Tuesday, Mr. Treadwell held a meeting with some Amazon employees to discuss how to address recent issues. An Amazon spokesperson told Business Insider that the meeting was part of a regular weekly review and that Amazon Web Services’ cloud business was not involved in these incidents.
“As part of our normal business operations and as we focus on continuous improvement, the meeting will include a review of the availability of our website and apps,” the spokesperson said.
GenAI “accelerates exposure”
Some of the most serious disruptions occurred last week, according to internal documents.
On March 2nd, Amazon Marketplace customers were shown incorrect delivery times when adding items to their carts. The incident resulted in approximately 120,000 lost orders and approximately 1.6 million website errors. According to internal reviews, Amazon’s AI tool Q was one of the main contributors to this event.
“The use of GenAI in control plane operations will accelerate the exposure of sharp edges and areas where guardrails do not exist,” one internal document said regarding the March 2 incident. “We need to invest in control plane security.”
On March 5, another outage caused a 99% drop in orders across Amazon’s North American market, resulting in 6.3 million lost orders, one of its internal documents said. One key factor was operational changes introduced without a formal documentation and approval process known as modeled change management.
“There will be no automated pre-deployment validation,” the document states. “A single authorized operator may be able to perform high blast radius configuration changes without guardrails.”
An Amazon spokesperson told Business Insider that only one incident investigated on Tuesday was AI-related, and none of it involved code created by AI.
90 day safety reset
According to one of its internal documents, Amazon is currently rolling out temporary safety guidelines for 90 days that will serve as an addition to existing policies.
The new policy targets approximately 335 “Tier-1 systems” owned by VP-level organizations, or services that can directly impact consumers, that have had multiple order-impacting incidents since last year.
The new policy requires Amazon engineers to have their work reviewed by two people before making any changes to their coding. You must also use internal documentation and approval tools and automated coding systems that strictly adhere to Amazon’s central reliability engineering rules.
Amazon is also notifying all Tier-1 system owners and directors and vice president-level leaders to audit all production code change activity within their organizations.
An Amazon spokesperson told Business Insider that it’s not accurate that junior and mid-level engineers need to get approval from senior engineers for AI-assisted changes.
The Financial Times previously reported that Amazon held a “detailed” meeting on Tuesday about the outage, saying its Kiro AI coding tool was partially responsible for a 13-hour AWS service outage in December.
Any tips? To contact this reporter via email, please specify the following address: ekim@businessinsider.com or on Signal, Telegram or WhatsApp 650-942-3061. Use a personal email address, non-work WiFi network, and non-work device. Here’s a guide to sharing your information securely.
