Millions of lost orders, website errors, and 'sharp edges': Amazon cracks down on code changes

In response to a series of significant disruptions impacting its e-commerce operations, Amazon is instituting stricter internal protocols regarding code changes. Over the past few months, the company has faced a troubling trend of incidents, including one particularly damaging outage linked to its AI coding assistant, Q. Dave Treadwell, Amazon's Senior Vice President of e-commerce services, informed staff about these challenges during a recent meeting. Internal documents reveal that the company experienced several major incidents since the third quarter of 2025. Some issues stemmed from what Treadwell referred to as "high blast radius changes," which occurred due to insufficient safeguards in the control planes that dictate data flow within the network. The repercussions of these outages have been severe, including significant delays and incorrect order information for customers. Notably, a March 2 incident resulted in nearly 120,000 lost orders and approximately 1.6 million website errors. An internal assessment indicated that Amazon's AI tool, Q, played a critical role in triggering this chaos. Another outage on March 5 led to a staggering 99% drop in orders across North America, culminating in losses of 6.3 million orders due to unauthorized production changes. To combat these issues, Amazon is rolling out a 90-day safety initiative aimed at enhancing oversight of code changes. This initiative will require engineers to document their code alterations meticulously and secure additional approvals before implementation. The goal is to introduce what executives term "controlled friction" into the review process, ensuring that critical parts of the retail experience are adequately safeguarded. Treadwell emphasized that the new guidelines will incorporate both AI-driven tools and more traditional, rules-based systems to create a more robust framework for code management. This approach addresses the inherent unpredictability of generative AI, which can produce varied outputs even when posed with identical queries, making it less suitable for tasks demanding absolute accuracy. Amazon's recent challenges highlight the transformative impact of generative AI on software development processes. While tools like Q allow engineers to generate code at unprecedented rates, they also necessitate rigorous checks to mitigate potential bugs and system failures. The company is now focusing on investments in control plane safety to avert future incidents and ensure a reliable e-commerce experience for its customers. As part of this initiative, Amazon is notifying its Tier-1 system owners and management to conduct thorough audits of all production code changes. These new measures are part of an ongoing commitment to improve reliability and maintain customer trust in the face of technological advancements.