Outage downs OVH, world's third-largest hosting firm

A major outage struck OVH, the French company that claims to be the world's third-largest hosting provider.

OVH hosts 260,000 servers across 22 data centres in 19 countries, and claims to support 155 out of the 1000 largest European companies.

The major outage even temporarily took down the company's corporate website, though most client websites were back up at time of writing after an outage lasting several hours throughout Thursday morning in France.

OVH wrote on Twitter at 6.33pm AEST on Thursday 9 November: "An incident is ongoing impacting our network. We are all on the problem. Sorry for the inconvenienace (sic)."

According to the company's status page: "We are experiencing an electrical outage on Strasbourg site. We are investigating." 

OVH chief executive Octave Klaba also turned to Twitter to actively update users on the status of the outage.

SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. #Murphy

— Octave Klaba (@olesovhcom) November 9, 2017

OVH rebooted its servers at its central European data centre in Strasbourg, France, which boasts 140,000 servers. 

The outage was largely centred on Europe, where OVH has data centres across several French cities, including Strasbourg, Paris, Gravelines and Roubaix.

Hello, incidents were detected in Roubaix and Strasbourg this morning. The normal functioning of certain services is affected. Octave Klaba has communicated from incident detection, via tweets and continues to do so. All our teams are on the bridge.

— Support OVH EN (@ovh_support_en) November 9, 2017

OVH launched in Australia in 2016 with its first footprint in one of Equinix's Sydney data centres, and in 2017 established a Melbourne office. 

In April, OVH acquired VMware's struggling vCloud Air business, though the deal did not initially include Australia.

Users took to the company's Facebook page to complain about service disruptions.

Update: 10 November, 6.45am AEST

OVH revealed that "two separate incidents" were to blame for the outage, according to its status page.

"The first incident impacted our Strasbourg site (SBG) and the 2nd Roubaix (RBX). In SBG we have 3 datacentres in operation and 1 under construction. In RBX, we have 7 datacentres in operation."

Power had been restored at the Strasbourg facility, and customers were told their services could take up to four hours to come back online.

The second problem was blamed on a software fault on third-party optical equipment from an unnamd supplier.

"We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be cut from our site in RBX.

"We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident. "

"We are in the process of retrieving the details to provide you with information on the SBG recovery time for all services/customers. Also, we will give all the technical details on the origin of these 2 incidents. 

"We are sincerely sorry. We have just experienced 2 simultaneous and independent events that impacted all RBX customers between 8:15 am abd 10:37 am and all SBG customers between 7:15 am and 11:15 am. We are still working on customers who are not UP yet in SBG."