On October 19th and 20th, AWS experienced an extended outage triggered by a failure in Amazon DynamoDB that affected most services in its most popular region, Northern Virginia. The cloud provider released an analysis of the incident, sparking discussions in the community about redundancy on AWS, moving out of public cloud, and multi-region approaches.
According to the post-mortem, which provides details on the DynamoDB DNS management architecture, the incident was triggered by a latent defect in the service’s automated DNS management system, leading to endpoint resolution failures for DynamoDB. Other popular services that rely on DynamoDB, including new EC2 instance launches, Lambda invocations, and Fargate task launches, were also impacted during the outage. In the summary of the Amazon DynamoDB service disruption in the US-EAST-1 region, the team acknowledges that the reliability issue significantly affected many customers. They write:
The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair.
While the incident report focuses on specific 2- or 3-hour periods and services, the repercussions for many customers lasted significantly longer, with some reporting issues for up to 15 hours. While in many online threads developers joked about “it’s always DNS”, Yan Cui, AWS Hero and serverless expert, highlights in his newsletter:
The DNS failure was the first symptom, not the root cause of the recent AWS outage. The root cause was a race condition in an internal DynamoDB microservice that automates DNS record management for the regional cells of DynamoDB.
During the outage, new EC2 instances were created at the hypervisor level, but their networking configuration never completed. Furthermore, delays in network state propagation for newly launched EC2 instances also caused a cascading impact on the Network Load Balancer service and all other services that rely on it. Jeremy Daly, director of research at CloudZero, writes:
The good news? It’s all back to normal. The bad news? People are losing their minds over this and thinking that a T1 line to their garage would be more resilient.
AWS has not yet addressed the race condition scenario, but is planning several short- and long-term changes as a result of this outage. The team writes:
We have already disabled the DynamoDB DNS Planner and the DNS Enactor automation worldwide. In advance of re-enabling this automation, we will fix the race condition scenario and add additional protections to prevent the application of incorrect DNS plans. For NLB, we are adding a velocity control mechanism to limit the capacity a single NLB can remove when health check failures cause AZ failover.
Furthermore, the cloud provider plans to improve the throttling mechanism in EC2 data propagation systems to rate-limit incoming work based on the size of the waiting queue, protecting the service during periods of high load.
In a popular LinkedIn post, Roman Siewko, a DevOps consultant at ERGO Technology, argues that while the outage impacted many large providers and received significant attention, practitioners should also understand the history and the actual numbers before making changes. Siewko comments:
The ~15-hour AWS outage on October 20, 2025, left a strong impression. In contrast, it’s hard to compare that with the nearly silent reliability of the previous years. (…) Most of the time, it’s closer to four nines. As of October 25, 2025, the rolling figures are 99.84% (1-year) and 99.95% (5-year).
Mudassir Mustafa, co-founder and CEO of Rebase, adds:
In reliability engineering, memory bias often outweighs data. Teams overreact to rare events but underinvest in the slow, invisible work that keeps uptime steady year after year.
The AWS event history tracks all impacted services and their timelines.
