The Australian Taxation Office has reached a settlement with storage vendor HPE that will see it recoup its costs from its damaging storage network failure and provided a first glimpse into what went wrong, as first reported by CRN's sister title, iTnews.
The ATO's 3PAR storage area network (SAN) problems started last December when an unexpected three-day outage took all its online services down.
A second five-day outage in February followed as a result of the ATO's efforts to stabilise the faulty SAN that fell over two months prior.
It was later forced to take its online tax services down for several weekends across April and May as it swapped out the old SAN and bedded down new equipment.
HPE flew its faulty SAN back to the US to forensically test the equipment, which had only been installed at the ATO just over a year before the collapse.
ATO commissioner Chris Jordan later said HPE had never seen a failure of this type across any of the 67,000 organisations worldwide using the same equipment.
The tax agency commissioned PwC to review the incident, and HPE also said it would conduct its own root cause analysis. The ATO has similarly undertaken an internal review.
It today revealed those reports would be made public next week.
Ahead of the publication Jordan offered a glimpse into the detail contained in the reports at a budget estimates hearing today.
The failure of the 3PAR SAN was the result of a confluence of events: the fibre optic cables feeding the SAN were not optimally fitted, software bugs on the SAN disk drives meant stored data was inaccessible or unreadable, back-to-base HPE monitoring tools weren't activated, and the SAN configuration was more focused on performance than stability or resilience, Jordan said.
While only 12 of 800 disk drives failed, they impacted most ATO systems.
Additionally, some of the recovery tools that were required to restore the equipment were located on the SAN that failed.
"A relatively small disk drive failure had a large impact," Jordan said.
"[However] I want to assure the community that our systems are restored, no taxpayer data has been lost, and we are tracking well for tax time 2017".
The ATO commissioner also revealed the agency had reached a commercial settlement with storage vendor HPE over the failure.
The details are confidential, he said, but the deal recoups key costs incurred by the ATO, and gives the agency new and "higher-grade" equipment to equip it with a "world-class storage network".
-1.jpg&w=100&c=1&s=0)
.png&w=100&c=1&s=0)
