ISPs sweat on Primus data centre power failure report

Power to the data centre, which is a Pipe Networks' peering exchange, was cut at about  2pm.

It was restored about four hours later according to ISP service disruption reports but Internet forum users continued to report issues into the night.

It affected ISPs including Internode, iiNet and Westnet.

A spokesperson for Primus could not be reached for comment.

Inundated with complaints, ISPs and their customers vented their frustration on forums and to iTnews.

"We haven't had a detailed report yet on the circumstances [surrounding the failure]," Greg Bader, CTO of iiNet, told iTnews.

"Once we receive the report this afternoon we'll sit down as a group and look at what steps we can take to mitigate the risk of it happening again.

"No one is interested in pointing fingers - we just want to get services back up, but the situation isn't ideal. We expect our data centres to be able to withstand power failures," Bader said.

Internode re-routed interstate and some local traffic through a back-up data centre in Melbourne, said its managing director, Simon Hackett.

"Had we not had that dual data centre strategy, there would have been major service issues in other states as a result," Hackett said in a statement posted to Whirlpool.net.au.

"That did indeed happen for some other providers who had only that site in Melbourne as their Melbourne access point."

He told iTnews he would wait on the post-incident report before deciding next steps.

"We do of course expect (and we normally obtain) better results during loss of external power than that from all of our data centre providers nationally," Hackett said.

"We're awaiting a (report) from Primus before deciding what additional steps to take in response to this outage, based on understanding what Primus themselves intend to do in order to mitigate against the potential for a recurrence.

"The Primus data centre concerned is a major communications site for much of the industry, so you can rest assured that we'll be ensuring, with Primus, that this doesn't become a habit. We don't expect that it will."

Read on to page two for the latest on the cause of the failure.

Bevan Slattery, managing director of Pipe Networks, said yesterday that while "details are still being determined, there appears to have been a loss of mains power, combined with a failure of backup diesel generation, and therefore the depletion of the UPS batteries.

"This will have resulted in an outage to the Pipe Melbourne peering exchange, and to customers with Ethernet circuits that transit through or terminate at this site," Slattery said.

A service disruption notice posted by Westnet appears to confirm catastrophic failure of multiple redundancy layers.

"Fault identified as a failure of primary, secondary and redundant power systems," it said.

Hackett speculated that the Primus centre did not have the same plant-redundancy levels as the ISP's data centres in South Australia.

"[Our] main one actually uses two generator sets to back up mains power with automatic 'failover' should one of them not start up during a power outage," Hackett said.

"It appears Primus have a single generator set in their Melbourne facility which (obviously) didn't start up properly on this occasion."

Cheyne Jonstone, national sales manager at Cove, indicated in the Cove support forums that the centre's batteries had kicked in after the initial power failure.

"Battery power picked up the load but for some reason the generators were not adding charge to the batteries (from what I overheard)," Jonstone wrote.

"The only reason they would not be adding charge is if they had failed to start or there were some other issues when trying to bring that power into the mix."

While Jonstone said he had "no complaints about the Primus ground crew" that worked to restore power to the facility, he was disappointed that Cove wasn't warned that data loads were running off batteries.

"If I had known that, we could have shut down servers as a precautionary measure to ensure there was no data loss (not that there was any in the end)," he said.

A report detailing the cause of the failure is expected to be issued this afternoon.