ATO website outage blamed on HPE 3PAR storage problem

A HPE storage network that crashed on Monday caused the loss of 1 petabyte of data and the downing of the Australian Tax Office’s online services, as first reported by CRN's sister title, iTnews. 

Early on Monday morning the ATO advised that its online portals, digital services and website were offline following an unspecified “hardware issue”.

All its online services remain down at the time of writing, almost two days after the problems first arose.

The ATO confirmed to iTnews that the issue was triggered by the catastrophic collapse of a hardware storage solution provided by HPE. It is understood the ATO acquired two new HPE 3PAR SANs late last year.

The collapse of the SAN caused the loss of 1PB of data.

The issue was compounded by the failure of back-up systems, which did not kick in immediately following the initial outage: iTnews understands the data corruption issues from the primary SAN were replicated onto the second SAN.

The ATO is currently in the process of rebuilding both and restoring from back-up, it is understood.

As CRN reported in November, the ATO switched from previous supplier EMC for its midrange storage and began replacing it with 3PAR from Hewlett Packard Enterprise.

The tax office emphasised that no sensitive taxpayer data had been compromised.

Acting ATO CIO Steve Hamilton told iTnews it was the first time this problem had been encountered anywhere in the world.

"We are working with HPE to determine the underlying cause. While these investigations are ongoing, we have had to implement alternative recovery procedures that are taking longer to complete," he said.

iTnews understands that the ATO's Siebel CRM and Sharepoint systems are also currently unavailable, leaving many of the agency's staff unable to perform their work.

Hamilton said the agency had made alternative work arrangements for staff impacted by the outage.

The tax office said it was hoping to have its critical systems – such as its tax agent portal – back online by lunchtime on Tuesday, and the main ATO website later in the day.

It said the outage was unrelated to scheduled system maintenance over the past weekend: the ATO took its online services for individuals, its mobile app, and its practitioner lodgment service offline from 10pm on 10 December to 7am the following morning.

Update 4:51pm: The ATO is continuing to work on bringing its systems back online and said it hoped to resume the tax agent portal and some services on the main website "later today".

"After full restoration, investigations will continue on the cause of the outage to ensure we treat the underlying issues and determine why the switch-over to back-up systems failed on this occasion," Hamilton said.