On Tuesday 28 February 2017, Amazon Web Services’ Simple Storage Service (S3) failed in its US-EAST-1 region. ThousandEyes recorded a complete outage from 9:40am to 12:36pm Pacific time. Catchpoint, another monitoring firm, put the disruption at three hours and 39 minutes, with knock-on effects across other Amazon cloud services and many websites that depend on the platform.
Forrester analyst Dave Bartoletti described S3 as “air in the cloud”. He argued that the incident was not a trend, because S3 had been reliable and secure enough to be the crown jewel of Amazon’s cloud. In his view it should work as a prompt for customers to check that their own applications could ride out the next disruption.
What went wrong
Amazon’s own account said the S3 team was debugging a slow billing system. At 9:37am Pacific time, an authorised team member ran a command from an established playbook to remove a small number of servers. One input was entered incorrectly, so a larger set of servers was removed than intended. Those servers supported two other S3 subsystems.
One was the index subsystem, which manages the metadata and location information of all S3 objects in the region. The other was the placement subsystem, which manages allocation of new storage and needs the index subsystem to work properly. The failure also disrupted services that depend on S3, including EC2, EFS, API Gateway, Athena and others.
Quora, Trello and IFTTT were among the affected sites. Amazon’s own status dashboard, whose icons were hosted on S3, could not be updated for a time. Amazon said it would address the fact that the tool allowed too much capacity to be removed too quickly. Gremlin’s retrospective noted that Amazon added safeguards to the tool, audited similar tools and changed the dashboard to run across multiple regions.
Lesson one: do not keep all your eggs in one basket
An application or data set deployed to a single point in the cloud has little fault tolerance. How many “baskets” to use depends on how available the application must be. There were three tiers of protection.
- The minimum AWS recommended was to spread workloads across multiple Availability Zones. At the time AWS had 16 regions, each split into at least two and sometimes as many as five zones. Each zone was meant to be isolated from the others in its region, and AWS provided low-latency links between them.
- For more protection, applications could span several regions. This matters in a case like this one, where the failure hit a whole region.
- The most thorough option was to use several providers, such as Microsoft Azure, Google Cloud Platform or internal or hosted infrastructure as a backup.
Bartoletti said urgency would differ between customers. A business that makes money from the cloud, or relies on it for productivity, needed to make sure the setup was fault tolerant and highly available. A company that only stores rarely accessed backup files might accept the occasional disruption.
Lesson two: identify failures quickly
Responding to a failure starts with knowing it has happened. In 2017 AWS offered Health Checks, which gave each account a view of the status of the AWS resources it used. Amazon CloudWatch could be set up to track service availability, monitor log files, raise alarms and react to failures.
These tools only work if the team knows what normal behaviour looks like, so that abnormal behaviour stands out. Once an error is detected, the responses need to be configured in advance. That means the multi-zone, multi-region or multi-cloud layout described above, load balancers that redirect traffic, and backup systems that start automatically.
The incident showed a related weakness in monitoring. When the status page itself depends on the failing service, customers lose their main source of information. Amazon could not reach its own dashboard to warn users and used Twitter for updates instead.
Lesson three: build redundancy from the start
Trying to respond to an outage in real time is rarely effective. Preparation beforehand is what counts. There were two basic approaches to building redundancy into cloud systems.
Standby
The application detects a failure and fails over automatically to a backup system. The backup can be switched off but ready to start when an error is detected. Alternatively it can run idle in the background all the time, which costs more but shortens failover. Either way, there may be a lag between detection and the backup taking over.
Active redundancy
To avoid downtime in theory, the application is spread across several redundant resources. When one fails, the others absorb a larger share of the work. Sharding is one way to do this. In the example given, an application running on eight virtual machine instances is divided into four groups of two, with traffic load balanced between them. If one group fails, the other three pick up its traffic.
Why the incident was a test of dependency
ThousandEyes said the outage exposed many services’ dependencies on S3, in many cases over-dependencies, and a lack of redundancy in their cloud storage. Gremlin noted that many S3 users in the affected region had no multi-region failover in place. The three lessons above address that gap directly: spread the workload, detect failure early, and design backup paths before they are needed.