Power Failure Leaves Customers Fuming

“It is clearly unacceptable to experience a power outage of this gravity”

Data centre giant Equinix has admitted suffering a critical power outage at its IBX LD8 data centre in London’s docklands — the issue has knocked services offline for numerous customers including several ISPs since approximately 4am this morning, BST — leaving customers fuming at the apparent lack of resilience.

“Equinix engineers have diagnosed the root cause of the issue as a faulty UPS (uninterrupted [sic] power supply) system and we are working with our customers to minimise the impact. We regret any inconvenience this has caused” the company said: it was not immediately clear why backup mechanisms failed. 

Various customers have identified this as a failing output static switch in the Galaxy UPS system (sold by Schneider). This is a component that connects the critical load to either conditioned power from the UPS or raw mains from the bypass supply; the latter in some instances provided by on-site diesel or other generators.

One customer affected was ISP Giganet. It told customers: “We’re still waiting for our network rack to regain power following Equinix and their contractors migrating power supplies onto the new infrastructure following the earlier fault.

“There is sadly still no estimated fix time which is most frustrating. They have assured us that they will provide this information when they can. Equinix are being continually chased for updates. As you can appreciate this is a P1 issue affecting many 100s of other carriers/ISPs – so it’s been given the maximum priority.”

BT is understood to be among the others impacted.

Major DC outage ongoing down London way. Equinix LD8. Looks a bit like power systems that were due to be replaced have failed early and are NOW being replaced. A decidedly non-live migrate. Best info seems to be coming from one of their customers here: https://t.co/rZrg2KkXda

— John Leach (@johnleach) August 18, 2020

The co-location is described by Equinix as offering access to “dense concentrations of financial services, Internet service providers, cloud and IT services, enterprises and content and digital media companies.” BT is among those affected.

Computer Business Review was not immediately able to confirm rumours that the location had suffered a fire owing to the UPS failure.

Today Equinix IBX LD8, in the Docklands, London, UK, experienced a power outage. This has impacted customers who are based there. The outage may have also affected customers’ network services.

— Equinix UK (@EquinixUK) August 18, 2020

Equinix said it is allowing customers “more flexible access to LD8” as it scrambles to fix the issue, “working within our COVID-19 restrictions including mandatory temperature checks and wearing face coverings.” 

Giganet added: “We have lost both A+B feeds to 1 of our 2 Equinix LD8 racks at approximately 4.23am. This follows a UPS failure, which then triggered the fire alarm in the data centre according to reports from Equinix. The rack that we have lost power to houses our core Juniper MX router and Cisco LNS.

“The Juniper MX router is our core device which is needed for everything in LD8 to function, including terminating a number of leased line connections as well as providing connectivity to our vDC platform. All our equipment power suppliers are dual fed with ‘diverse’ A+B power feeds provided by the data centre – however after this incident we suspect that there is a lack of resiliency and will be sure to raise this after the incident is resolved as this is clearly unacceptable to experience a power outage of this gravity.”

Are you affected by the outage? Want to vent/share thoughts? Get in touch