Amazon Web Services has offered a brief explanation for the outage that impacted its EC2 IaaS and other services in Sydney yesterday.
A status update posted overnight says explains that the problem manifested at 11:07 AM yesterday, Sydney time.
“We immediately began investigating the root cause and identified that the data store used by the subsystem responsible for the Virtual Private Cloud (VPC) regional state was impaired,” the update says.
“While the investigation into the issue was started immediately, it took us longer to understand the full extent of the issue and determine a path to recovery. We determined that the data store needed to be restored to a point before the issue began. We began the data store restoration process, which took a few hours and by [5:50 PM Sydney time], we had fully restored the primary node in the affected data store.”
Which was nice.
But not a complete fix.
“At this stage, we began to see recovery in instance launches within the AP-SOUTHEAST-2 Region, restoring many customer applications and services to a healthy state,” the update continues.
“We continued to bring the data store back to a fully operational state and by [6:50 PM Sydney time], all API error rates and latencies had fully recovered.”
And then comes the standard apology: “We apologize for any inconvenience this event may have caused as we know how critical our services are to our customers. We are never satisfied with operational performance of our services that is anything less than perfect, and will do everything we can to learn from this event and drive improvement across our services.”
AWS usually publishes decent reports on root causes of such incidents. CRN will watch for it with interest.
While the incident was of course unwelcome, one channel player found a way to use it as a way to drum up business – and move a customer from AWS to Azure!
Looks like there is no HA built in to @ABC_COMEDY 's infrastructure. Only one AWS region is impacted (Sydney). Maybe think about going multi region?!
— David O'Brien (@david_obrien) January 23, 2020
Or come have a chat to us, we'll help you get onto Azure and go multi-region in Australia. https://t.co/CmtTKGR1nA
-1.jpg&w=100&c=1&s=0)
.png&w=100&c=1&s=0)
