ATO flags deep investigation into "worst unplanned outage"

The Australian Taxation Office will enlist an independent specialist to investigate the causes behind last week's major outage.

On 12 December, the ATO suffered its "worst unplanned system outage" due to the crashing of its HPE storage network, which had been recently replaced from ageing midrange EMC storage to HPE 3PAR.

The issue was compounded by the failure of backup systems, which did not kick in immediately following the initial outage.

Chris Jordan, commissioner of taxation, said on Friday that with the operations returning to normal, there will be a full investigation of the events that occurred last week.

"There will be a comprehensive and independent review to help us answer the questions – exactly what happened, how and why, and what measures need to be taken to avoid such a situation in the future.

The investigation will be conducted by an independent expert who will issue a description of the failure and determine its causes, what contributed to the outage and its duration, scale and scope.

The independent investigation is also expected to determine the adequacy of back-up and contingency arrangements, explanation as to why failover to its secondary site did not work, and adequacy of restoration and resumption procedures of technology, infrastructure and applications. It will also investigate the likelihood of recurrence.

The review will also consider the ATO's immediate response to the failure, how business resumption processes was managed, how the ATO communicated with the public, the ability to detect the outage and to ascertain the scale and scope of the impact.

In a statement sent to CRN, an HPE spokesperson said: "HPE has taken immediate action to help resolve the hardware issues which have impacted the ATO’s online services, portals and website.

"This is a top priority for HPE and we continue to manage this closely with our client to ensure that all the systems are restored to functionality as soon as possible."

ATO goes offline

On 13 December 1.11pm, the ATO issued a statement addressing the outages that affected its online services, portals and website. Acting chief information officer Steve Hamilton said that no taxpayer information had been compromised.

At 6.38pm on 13 December, the ATO announced that ato.gov.au was back online for tax and super information access, but some tools were still unavailable.

On 14 December, the ATO confirmed that not data had been lost. "The petabyte of data referred to in media reports relates to storage capacity, which includes not only data but applications and systems as well. This figure does not relate to data impacted by the outages. 

"While we experienced some corruption of data, we are in the process of fully restoring this information from backup. No taxpayer information has been compromised," read a statement.

This was the first time the problem had occurred anywhere in the world, according to Hamilton.

The ATO said that the restoration process was quite complex and was "requiring significant manual processing which unfortunately is taking longer than we expected to complete".

On 15 December, the tax agent portal, BAS portal and business portal were back online.