Datacom NOC staff win support

Network operations centre staff at Datacom won the praise of at least one customer for their response to an apparent flood crisis caused by Melbourne storms on Saturday.

iTnews revealed yesterday that some customers had lost equipment when water leaked through the ceiling and into a co-location area used by a number of major carriers.

Datacom was expected to make a formal statement shortly.

Speaking on condition of anonymity, an online service provider told iTnews today that it had "learned of flooding at about 15.30" on Saturday.

"We called the network operations centre (NOC) due to a different issue and were told about the issue at hand and how busy all the staff were. At that point, one of our staff started making their way to the facility," the provider said.

"At 16:10, we observed a complete drop in internet connectivity. As this was happening, Datacom provided my staff and I with updates, giving us details of what was happening and ETAs as to resolution, both to myself over the phone and to my colleague who was in attendance.

"Transit connectivity came back up at 17:05, but some of our comms had continuing issues whilst various carriers sorted out whatever problems they had on their hands. My contacts at quite a few carriers report having spent quite a while sorting out their issues on the data floor.

"Above all, I would like to emphasise how well Datacom responded to the situation and the excellent communication they had with us throughout the night. We were given regular updates thoughout the evening by staff on the floor and they were very honest about progress and the nature of problems.

"As an example of how helpful Datacom were, they were able to confirm which carriers were on the data floor during each update as to give us an expectation of when certain third-party provided services would become available."

Optus confirmed yesterday that it was one of the carriers to be affected by the power outage at Datacom.

Data centre operator Micron21 told iTnews it had limited exposure to the third-party carriage issues that stemmed from the Datacom outage.

"I was first notified of issues via SMS on Saturday the 6th of March at 4:09pm EST that one of our Optus circuits managed by Uecomm was experiencing one directional packet loss," infrastructure manager James Braunegg said.

"Our BGP network configuration automatically routed around the affected circuit, so our Micron21 clients on the affected circuit were mostly unaffected. Our network administrator shut down any other circuits which ran though Globalcenter manually, seamlessly routing around any other potential issues which might have occurred.

"Upon further investigation the Uecomm NOC confirmed the affected Optus circuits were located in Globalcenter and were not responding due to the facility being severely affected by bad weather."