Facebook's Massive Blackout: Here's What Data Centers Have Learned So Far

Facebook’s massive outage put data centers on front pages across the globe this week — and may pose some uncomfortable questions for operators moving forward.

When Facebook, Instagram, Facebook Messenger and WhatsApp went down for more than six hours on Monday, the outage created a cascading series of consequences around the world as the loss of messaging services interrupted communications, businesses relying on Facebook’s platforms saw sales grind to a halt, and companies and individuals lost access to devices and services logged in through Facebook.

The tech giant itself also lost control over many of its internal systems and infrastructure during the blackout, which reportedly cost the company more than $100M in lost revenue.

The cause of the outage — the largest in history, according to downdetector.com — was a faulty update of a system called the Border Gateway Protocol that routs traffic between the company’s private networks and the outside internet, linking the company’s data centers and digital infrastructure together.

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” wrote Facebook's vice president for infrastructure, Santosh Janardhan, in a blog post.

“This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”

The outage was caused by a network problem, rather than an issue with a data center itself like a power outage or damaged fiber line. But the length of the outage — reportedly due to the need to get engineers physically into a data center — may make this incident an important case study for an industry facing a labor shortage and increasingly looking toward a future with few on-site personnel.

Fixing the outage Monday required Facebook engineers to physically access a data center in Santa Clara, California, as the systems normally used for remote access failed. That proved difficult. First, the outage impacted the company’s keycard access systems. Perhaps more significantly, the engineers with the skills needed to perform the necessary reset were not anywhere near the Santa Clara data center, according to reports.

“The people with physical access [are] separate from the people with knowledge of how to actually authenticate the systems and people who know what to actually do,” Interesting Engineering reports a Facebook insider as saying. “So there is now a logistical challenge with getting all that knowledge unified."

Some sources places blamed pandemic-driven work-from-home policies on the lack of staff on-site to perform the reset but data centers are facing a dire shortage of workers overall, particularly in markets far from data center hubs. Increasingly, facilities are monitored from central operations centers, which could be hundreds of miles away. So-called dark data centers, which have no on-site staff, are increasingly common.

Although industry leaders are increasingly investing in using artificial intelligence, remote monitoring and automation to try to reduce the human role in data center operations, Facebook’s outage may serve as evidence of the importance of expertise on-site or at least nearby, even in the most sophisticated data centers.

And with more than $100M in lost revenues and the company’s mistakes a top news story around the world, it also showed what's at stake.

Continue reading this story with a free account

Log in or register
Sign up for more articles like this
Subscribe to Bisnow's National Newsletters
Related Stories

CRE's Clearance Rack Is Empty, Leaving Landlords To Create Their Own Value

Ares And Scion's Student Housing JV Acquires 4 Communities For $435M

Anthropic Inks $35B Deal With Nvidia-Backed AI Cloud Provider

Deutsche Finance Alleges BVK Is Responsible For CRE Portfolio Losing $1B In New Lawsuit

Chobani To Invest $1.2B In Manufacturing, Warehouse Campus In Pennsylvania

The Transfer Wave: How Companies' Operations Can Survive Founder Dependence

Why Factoring Tenant Satisfaction Into Office Programming And Operations Can Bring Strong Returns

Lab Landlords In 'Hand-To-Hand Combat' To Last To Recovery

4 Asset Classes Post $4.6B In New Distress

National Park Service Considers Transferring Yosemite Land To Developer

JCPenney Creditors Sue For Damages After $934M Deal For 117 Stores Collapses

Texas Governor's Data Center Power Pause Gets Split Verdict From Developers