How a tiny bug spiraled into a massive outage that took down the internet

A significant outage on AWS this past Monday disrupted access to numerous popular applications and services worldwide, all stemming from a seemingly minor glitch. According to Amazon’s post-incident report released Thursday, the issue arose when two automated systems attempted to update the same data at the same time, leading to a cascading failure that engineers raced to resolve. The impact of this outage was felt by countless users who found themselves unable to order food, connect with hospital networks, access mobile banking, or manage their smart home devices. Major brands, including Netflix, Starbucks, and United Airlines, temporarily lost the ability to provide online services to their customers. In an official statement, Amazon expressed its regret, saying, "We apologize for the impact this event caused our customers. We understand this incident affected many in significant ways. We are committed to learning from this and enhancing our service reliability." At the core of the problem was a conflict between two programs vying to modify the same DNS entry—essentially a critical component of the internet's directory system. This conflict resulted in a vacant entry, which caused chaos among various AWS services. Angelique Medina, head of Cisco's ThousandEyes Internet Intelligence network monitoring service, likened the situation to losing access to a phonebook: "If you don’t know how to reach the people on the other line, you’re in trouble. That phonebook effectively vanished." Indranil Gupta, a professor at the University of Illinois, provided a classroom analogy to illustrate the situation. He compared it to two students working together on a shared notebook, where the faster student continuously attempts to correct the slower student's contributions, leading to an empty or crossed-out page when the instructor checks it. This empty page caused the failure of AWS's DynamoDB database, which subsequently affected other services like EC2 and the Network Load Balancer. When DynamoDB finally resumed operations, EC2 struggled to restore all servers simultaneously, resulting in further complications. In response to the outage, Amazon plans to implement various changes, including addressing the race condition that led to the conflicting updates and introducing an additional test suite for the EC2 service. While outages of this magnitude are infrequent, Gupta emphasized the importance of how companies react to such events. "Large-scale outages like this can occur, and while they can’t always be prevented, a company’s response and communication with customers during these times are crucial," he remarked.