Amazon Web Services' cloud services have been affected by “increased API error rates and latencies” in its Asia-Pacific (Sydney) region.
Users began reporting issues at 11:15am on Thursday, and remains ongoing as of 4:00pm.
AWS’s status page confirmed issues affecting AppStream 2.0, Elastic Compute Cloud, Elastic Load Balancing, ElastiCache, Relational Database Service and Lambda.
We are investigating increased API error rates and latencies in the Asia Pacific (Sydney) Region. Connectivity to existing instances is not impacted. More updates will be available in our Service Health Dashboard: https://t.co/QLlsejEe3z.
— AWS Support (@AWSSupport) January 23, 2020
AWS said the root cause was identified by 12:18pm and is continuing to resolve it, but notes that connectivity to existing instances aren’t affected.
As of 3:49pm, AWS revealed more details on the issue:
"A data store used by a subsystem responsible for the configuration of Virtual Private Cloud (VPC) networks is currently offline and the engineering team are working to restore it," AWS's status page read.
"While the investigation into the issue was started immediately, it took us longer to understand the full extent of the issue and determine a path to recovery."
AWS said the data store needed to be restored to a point before the issue started, so writes needed to be disabled and the error rates will continue until the restoration is complete.
"We are working through the recovery process now. With issues like this, it is always difficult to provide an accurate ETA, but we expect to complete the restore process within the next 2 hours and begin to allow API requests to proceed once again," the status page read.
"We apologize for the impact and continue to work towards full resolution."
AWS customers also reported issues on Twitter:
Heads up, ap-southeast-2 (Sydney) experiencing a partial outage for VPC management operations. #aws
— Ian Mckay (@iann0036) January 23, 2020
#aws outage Sydney - Its been 3 hours already ...Anyone knows what's happening and recovery timeframe. Impacted ones include glue services as well, as they depend on the compute.
— Krishna Saranathan (@KrishnaSaranath) January 23, 2020
@awscloud AWS EC2 instance are not visible in the console for Sydney region. Error occurred while fetching instance data.
— Gaurav Singh (@katoch21_gaurav) January 23, 2020
Please confirm if ther is an ongoing issue.service dashboard shows alarm state.@amazon
-1.jpg&w=100&c=1&s=0)
.png&w=100&c=1&s=0)
