How Facebook Vanished from the Internet

Originally published at: How Facebook Vanished from the Internet - TidBITS

On 4 October 2021, Facebook disappeared from the Internet for most of the day. Internet infrastructure company Cloudflare explains how it happened, even if we don’t know exactly why.

Too bad it wasn’t permanently! :laughing:

4 Likes

This explains why for a time when trying to access replies sent to me in Instagram I just kept getting “Can’t refresh feed…”

Some really good memes are here:

At least there haven’t been any reports of hysterical people flooding 911 call centers, or user data being compromised. I’d like to believe that some Facebook users went to actual news sites instead, but I haven’t read a word about traffic shifting in this direction.

And it wasn’t like the response of many to Orson Wells’ 1938 “War Of The Worlds” radio broadcast:

I had reason to call T-Mobile for support yesterday, and I was greeted not with their usual “We’re having high call volumes and will call you back” message (which works really well) but with “We’re experiencing technical difficulties. Please try again later.”

Then today, when I called again, I asked the support rep about it. He didn’t know about the different system message but said that they were absolutely flooded with calls yesterday, many about the problems with Facebook. That jived with the Downdetector screenshot we published in the ExtraBIT showing the cellular carriers as receiving a lot of down reports as well. Lots of people must see the carrier support departments as handling anything that happens on their phones.

Last night (Monday, 4 Oct 21) on the news reports about the outage, it was said that the timing was interesting as it followed Sunday’s “60 Minutes” piece on the Facebook whistleblower.

Much more likely that a carrier would be having problems rather than Facebook.com

Yes, but as you can see in the graphic I referenced, Spectrum, T-Mobile, Verizon, and AT&T all had significant down reports spiking at the same time as Facebook. It’s clearly connected.

2 Likes

I was speaking as the typical consumer would have seen it, addressing why the support departments were overwhelmed. Carriers outages, usually regional, are much more common from the consumer viewpoint. It’s doubtful Joe User could determine there was a different cause until the news hit the national media.

All reports indicate that this was caused by someone injecting bad data into the global BGP routing network.

For those unfamiliar, the Internet is a “network of networks”. Various sites (large and small companies) run their own networks that are, more or less, self-contained. The links that connect these networks are typically leased from the telcos (although some big companies own their own as well).

The BGP protocol is used to distribute information globally about the connections between these networks. It is critical to the Internet, since the servers you access are almost always located on somebody else’s network and BGP is how the Internet knows how to move your packets from your location to the destination.

The problem here is that BGP has only minimal security. Someone injecting bad data into a BGP router (whether by accident or maliciously) can create all kinds of problems, usually resulting in network outages for various segments of the Internet until the correct data can be re-distributed to replace the bad data.

In this case, bad data was injected, making Facebook’s DNS server unreachable, effectively knocking them off line. Whether this was an accident or a deliberate attack, and the source of the bad data has yet to be determined.

2 Likes

According to Facebook, it appears that the BGP misconfiguration originated from Facebook’s own network.

I don’t believe the conspiracy theory that Facebook did it deliberately for some reason. If they wanted to silence a story, they can just censor it, the way they’ve done for countless other stories, without taking the rest of their system down.

Outages like this do happen from time to time, so it is definitely possible that this was an accident. Someone accidentally clicks “delete” instead of “update” (yes, an oversimplification) on a network management console can do this, and even if the problem is discovered immediately it can take a few hours for the corrected data to propagate to the rest of the Internet.

On the other hand, any time a major site goes down, you should never rule out sabotage. I’m sure FB’s IT people are investigating. Hopefully they’ll let the rest of us know what they discover, but I’m not counting on that.

1 Like

The supposed panic response to War of the Worlds is an urban legend, whipped up at the time by newspapers jealous of the popularity of radio.

From what I always herd from my parents, grandparents and those of other friends, family and work colleagues was that although there weren’t stampedes of people in NYC and suburbs, as well as relatives in Chicago, there were some people some screaming in the streets. It was not hordes of people, just a few. My grandparents said that they saw a few people up on the roof of a neighboring apartment building to check for space aliens.

The disturbing thing is that not just Facebook went down - the problem also took down the associated WhatsApp and Instagram systems. That is simply poor structure. Those systems should be quite separate and have internal redundancy. The only reason to skimp on this is to save money. My suspicion is that the actual problem was due to some configuration that tried to more closely link these services so they could share hardware resources - what could possibly go wrong! It’s hard to come up with a good analogy - since I have a medical background perhaps: it would be as if 3 operating rooms tried to share one big set of instruments. We always have known that this group is ethically corrupt, but they are also technically bankrupt.

2 Likes

Am I the only one here who had never heard of BGP before yesterday?

1 Like

Nope. I never did.

Selling cross platform and cross device advertising packages is what keeps Facebook’s profits growing. Here are a few examples:

https://www.facebook.com/business/help/1011620975524097?id=1794272243992044

More details have emerged:

Not sabotage. Just a colossal screw-up combined with bugs in the audit software designed to detect such mistakes.

The more I learn about Facebook, the less I like them. That’s all I’m going to say, for now.