Melbourne IT hammered for a week by firewall problems

Australia's largest web registry Melbourne IT has suffered ongoing network issues over the past week related to an issue with the company's firewall that resulted in it dropping up to half its data packets.

The service degradation issues started last Wednesday at the same time as the company struggled with a separate email delivery issue as well as difficulties migrating users off its legacy platform and onto that of its subsidiary NetRegistry, which it purchased for $50 million in February last year.

The confluence of separate issues resulted in its call centre becoming entirely overwhelmed with calls, and a resulting customer backlash against long wait times.

"The problem we have is that the issues all happened at the same time, and together they drove a lot of calls into the call centre," Melbourne IT CEO Martin Mercer told iTnews.

"And that meant customers couldn't get an agent to help them. And that's really regrettable, and frustrating and in some cases upsetting for these people."

Melbourne IT struggled to bring network services back online throughout the rest of the week and until yesterday, when it advised that its engineers had potentially isolated the root cause of the packet loss issue, which it attributed to a firewall in need of replacement.

Melbourne IT is understood to use firewall products from both CheckPoint and Cisco. Melbourne IT CEO Martin Mercer said the company had purchased a new Cisco firewall as a replacement.

Packet loss reached levels as high as 50 percent of all packets during peak loads, the company revealed, which slowed and killed access to websites and email.

The company shuttered services for 15 minutes yesterday to reconfigure the network and was forced to order a new Cisco firewall from the US, which it expects to arrive on January 26th. It will then kill services for another 15 minutes as it adds the new hardware to the network.

"After the outage today our current modelling indicates the packet loss issue will be resolved. After we bring the network back up, we will monitor the situation through the afternoon, and provide any further updates if the situation either changes or there is any unexpected outcomes."

Later yesterday afternoon Melbourne IT advised customers that the changes appeared to have addressed the issue, with packet loss down to 0 percent. 

"We are cautiously optimistic that we are at full service capability and will be able to remain so until new infrastructure is in place."