Hotkey in turmoil as Primus data centre blackout bites

Primus Telecom has gone to ground over the cause of the outage, which took down PIPE Networks' Melbourne peering exchange and affected ISPs such as Internode, iiNet and Westnet.

Numerous calls placed by iTnews to the data centre and Primus corporate headquarters this week weren't returned.

Hotkey is emerging as one of the hardest hit by the data centre failure, with customers furious after enduring a week without email and web hosting.

Hotkey had earlier given an ETA of late Tuesday for the resumption of services, and separately claimed that 70 per cent of its web servers had been restored.

But the latest update on the ISP's website points to "intermittent" email access almost a week after the failure, and many customers are losing patience.

"I've been with Hotkey for over 10 years and had no problems before, but 10 years of hard work has come undone because of this," Martin Giles, managing director of O.U Norman Consulting Services, told iTnews.

"If we don't have something by 12pm today I'm going to call a couple of ISPs about churning. I have to do something because there's no guarantee they're going to come back [online]."

Giles said his four staff had been without email almost a week. He said he had experienced a "trickle of less than 10 emails" for a short period on Wednesday afternoon, but nothing before or since.

"Hotkey has not been at all apologetic or admitted that the situation is unacceptable," Giles said.

"The biggest problem is we don't know what the truth is. If they don't know a date when it will be fixed, [they should] tell us that. Don't say there's no ETA.

"They must have some indication of what the problem is, whether they can fix it and whether or not they've lost all my old emails. The biggest concern all us customers have is [that] they've gone silent."

Giles is not alone. A thread on the Whirlpool forums has grown to more than 19 pages, with many customers now exploring options to churn.

Hotkey's general manager, Robert Siddle, told iTnews in a brief interview that response to the extensive outage had been "mixed".

"We haven't had a mail outage in three-and-a-half years but we're making up for that now," he said.

Siddle said Hotkey was "trying to fix the issues". He also said that the blackout had forced Hotkey to move its email and web servers to another area of the Primus data centre, which thwarted its back-up strategy.

"We already had a backup situation but moving the equipment meant we couldn't implement it," Siddle said.

"We had to shift because of the limited amount of power Citipower could supply to us."

iTnews has contacted Citipower for comment. Giles, however, did not buy the explanation.

"It might be correct but it's irrelevant," he said. "It shouldn't take this long to do these things.

"Let's give them the benefit of the doubt and say it takes them two days to relocate the equipment. Why isn't it live again by Wednesday?

"No one is getting any emails still. As a customer, you start to think the whole thing is just one big spin cycle."

Primus' woes have also been compounded by the news that some of the other ISPs affected by the blackout still haven't received the post-incident report.

"We've asked them to resend it twice now and still haven't actually seen it," said Cheyne Jonstone, national sales manager at Cove.

"We've been constantly tailing our mail logs to see when their server hits ours."

Jonstone said Cove had only had one customer churn because of the blackout "but we did expect a few more". He also said that Cove would stick with Primus - for now.

"I'd much rather sit tight, have Primus say they stuffed up and hopefully see some more transparency around what they do," said Jonstone.

Other ISPs, including Internode, iiNet and Westnet, were contacted but were unable to comment at the time of writing.