Facebook-owned sites were down

[redacted]

When I noticed HN was loading slowly, I already knew FB was down.

One real potential cost to FB here is breaking people's addictions to FB and IG. This might just be the little finger-snap to wake up a sizable chunk of the user base that they life is just a little better during the outage.

nslookup www.facebook.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53

* server can't find www.facebook.com: SERVFAIL

I assume all of the tertiary sites that use "Login with Facebook" are broken now too? So glad I never adopted that.

This is all total left brain looping and complexity coming home to roost.

it is always DNS!

Suspecting it might be related to the recent letsencrypt cert authority expiring? Was just debugging an issue earlier today and just couldn't help wondering how much of the internet is secured by letsencrypt.

All of the static hosts providing free SSL: vercel, netlify, render, firebase hosting, github pages, heroku etc. ...

It does work on modern browsers and devices but goes terribly broken on a lot of old devices.

No like for you

Their stock is down 5% too. Everything is down for them today ;)

If it is an DNS error, why is the .onion site also offline?

- https://en.wikipedia.org/wiki/Facebook_onion_address

- facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion

add comment

The down page shows copyright from 2020 smh

From the archived ramenporn reddit comment thread at [0]:

> This must be incredibly stressful so for your sake I hope you sort it out quickly... but for the world's sake, I hope you fail and make the problem worse before jumping ship followed by every other engineer, leaving it to Zuckerberg to fix himself. But I still hope it's not too stressful for you!

https://archive.is/Idsdl

> Facebook-owned sites are down

And the world rejoiced.

Thoughts and prayers...

Looks like HN is being hit pretty hard right now?

Thought experiment: what if they were down for a week and the world completely healed itself?

And right now, the world is a slightly better place

Posting this comment will be like farting into a hurricane, but here goes.

Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity.

What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct.

Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.

Whatsapp's down too. Tough month for FB, especially with the leak.

Anyone wanna estimate the cost of total downtime for facebook and instagram, as far as lost ad revenue goes - per minute?

Interesting, even some open source sites like: https://fbinfer.com are down

but https://glean.software and https://reactjs.org aren't

Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.

Facebook down, WhatsApp down, but Signal still works. Time for a change?

EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.

Seems to be DNS related.

None of the listed facebook nameservers are resolvable or reachable:

a.ns.facebook.com b.ns.facebook.com c.ns.facebook.com d.ns.facebook.com

Glad to report that facebook's dns in china is not affected. You can dig facebook.com and the depth of the internet happily reply with a random ip address as usual.

One thing that triggers my OCD is leaving the Facebook session open, though it's my own computer.

Maybe it's DNS. It's always DNS.

I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)?

How could they all go down at the same time, if they have different teams of engineers running each product separately?

Could anyone with some background (or person familiar with the matter) explain how their system's set up?

It's like a nuclear bomb exploded on the internet.

add comment

Why is this not the first post on hn? It’s double the points and has over 590 comments?

Whatsapp down as well

Somebody published vogon poetry and pictures and the internet routed around the damage...

value of fb directly proportional to traffic flow curious about why fb is down ;)

We can only hope that they will be gone forever... and HN is having major issues at the same time!

Is it just me or HN also feels kinda laggy?

That explain the new contacts in Telegram.

Is this related to the outages from lets encrypt root cert expiring? Probably not since this looks like a DNS issue, but still it's a crazy coincidence that two major internet breaking events happen in the same week

Many local governments use FB to get info out.

Events like this show they should use multiple outlets instead of the big monopoly.

Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example:

[root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached

[root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53

Non-authoritative answer: Name: downforeveryoneorjustme.com Address: 172.67.166.187 Name: downforeveryoneorjustme.com Address: 104.21.91.48

[root@app ~]#

Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.

I love how understated companies always are about things like this.

> Facebook said: "We are aware some people are having trouble accessing our apps and products. We are working to get things back to normal as quickly as possible and apologise for any inconvenience."

https://www.bbc.co.uk/news/technology-58793174

Some Oculus Quest owners can't use their device https://www.reddit.com/r/OculusQuest/comments/q18xwy/faceboo...

The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?

Telegram seems down too, is it down for you?

ok mom i commented

Not an indepth technical comment here but: seeing a tech megacorp go offline for a day makes me very very happy.

Rumormill is suggesting that facebook badge readers are also down causing issues with trying to get to the servers to manually fix them.

https://twitter.com/sheeraf/status/1445099150316503057?s=21

rich people serve their revenge for pandora papers

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook.

I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

Reddit wasn’t working a few min ago. Broader issue?

Does anyone have a reasonable guess on how much money they have already lost?

now if only tiktok would fail

Confirming that we're seeing a major outage with all of our integrations with FB products.

Rejoice! The revolution has started!

Yes, I know humor is not welcome in HN

This outage is huge. I'm waiting for the write-up, assuming they release one

Is facebook being down causing hacker news to get the hug of death??

This is all left brain implimentation with looping and classic complexity coming home to roost. As we move through time, we build off of solutions of the past which are solving a problem, but complexity keeps adding on and this is a classic programming/computer science delemma.

Ironic enough; status.fb.com also down.

It's not hyperbole to say that this is going to literally save lives.

Cutting off Facebook's firehouse of hate and misinformation for just a couple hours is going to have a obvious positive effect on millions of people. At this scale, at least one person will get vaccinated today because they didn't see the wall of ignorance that is FB's news feed.

Maybe we should introduce "digital blue laws", where one day a week, social media is shut down for the overall good of society.

https://blog.cloudflare.com/october-2021-facebook-outage/

Duplicates:

https://news.ycombinator.com/item?id=28748233

https://news.ycombinator.com/item?id=28748199

https://news.ycombinator.com/item?id=28748159

https://news.ycombinator.com/item?id=28748246

https://news.ycombinator.com/item?id=28748131

Imagine this happening to AWS

FB seems to be finally loading for me, after nearly 6 hours.

This will be a highly discussed topic for a bit.

As are Instagram and WhatsApp

This is taking a longer time than expected for a company like Facebook - must be serious where a rollback isn't possible or trivial.

How much revenue does facebook loose per hour down?

still down, people going to signal to have a chat

Signal is welcoming everyone:

https://nitter.mailstation.de/signalapp/status/1445062426739...

do we know what caused the DNS outage at FB?

Should say "Facebook owned. Sites are down."

Funny enough, I went to https://www.isitdownrightnow.com/ to check if Facebook is down, and isitdownrightnow is down itself... probably from the massive number requests coming to check if Facebook is down

I like it, it feels like it's 1999.

Once again, it's DNS.

https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...

Twitter seems to be a big buggy now too maybe just a coincidence. User comments under posts are not appearing.

  % ping whatsapp.com
  ping: whatsapp.com: Name or service not known
  % ping web.whatsapp.com
  ping: web.whatsapp.com: Name or service not known
  % ping facebook.com
  ping: facebook.com: Name or service not known
  % ping instagram.com
  PING instagram.com (31.13.65.174) 56(84) bytes of data.
  64 bytes from 31.13.65.174 (31.13.65.174): icmp_seq=1 ttl=53 time=110 ms

can productivity (or emotional stability) for the overall US economy be tracked on a daily basis? I wonder if a wholesale facebook outage would show up on that graph as a brief blip in the positive direction.

If it bleeds we can kill it!

good

LOL! They should have used the USA Fact-Check Algorithm from the Science Ministry.

Does everyone just buy in that this is just a network change gone wrong? OR could they be mitigating a breach/hack? OR could it be some other theory?

Good, now I can go for lunch.

for me even this site loads very very slowly. pinging google name server is fast as usual. it could be a more wide problem not just FB related.

This is honestly the best feature Facebook has ever developed. I hope it's permanent. It has the following effects: you feel better about yourself, you can spend more time with your family, you are more productive.

Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.

it is probably unrelated but HN is crawling

Looks like the routes to their hosted nameservers are down, e.g. A.NS.FACEBOOK.COM

Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.

I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook.

For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.

Fixed

Doesn't seem too clever that Facebook's NS servers are a.ns.facebook.com, b.ns.facebook.com etc. IIRC that kind of setup requires some glue records.

Seems like Telegram went down with a big whatsapp-is-down hug of death.

Here we can see why you should not have all your DNS servers in the same AS (in this case, AS32934).

between outage and whistleblower, this has got to be the worst day in facebook's life

Oh no...

Anyway...

Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.

There is a global outage this morning starting from 8:00 AM. This list has Google, Tiktok, Zoom, Slack and of course FB products and services.

For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL:

    $ dig facebook.com

    ; <<>> DiG 9.16.21 <<>> facebook.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23982
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;facebook.com.   IN A

    ;; Query time: 16 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Mon Oct 04 17:53:00 CEST 2021
    ;; MSG SIZE  rcvd: 41

In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.

I would like that say that after my "burn it down" comments on another Facebook related post that I had nothing to do with this.

frankly, who cares? Seriously.

Those services are toxic for years now and everybody knows that. Who still uses them occasionally let alone relies on them can't be helped, can they?

Is this in some way connected to the Facebook data leak of 1.5 billion users? The timing seems quite odd that both these things happen around the same time.

In other news, a bunch of people got a lot more work done today than normal I suspect...

After changing the screen resolution all operating systems will prompt the user if the applied settings where correct, otherwise it will time out and reset to last known good setting. Maybe time for the core internet infrastructure to implement something similar? :)

Ha-ha!

It is about time to speculate about sabotage, a disgruntled employee or something more exotic.

All this BGP talk is boring.

I feel for the sysadmins who are fighting ulcers and migraines at the moment, but I can't shake feeling that the world is just a little bit better for this small window of time.

At 21:44 UTC, facebook.com resolves for me.

So they managed to remove facebook.com from 1.1.1.1 and 8.8.8.8. That is impressive. Not something anyone can achieve in such short time by even trying.

My father (https://bit.ly/3acZAAI), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake.

Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.

good ridance

Is this caused by missing glue records? I can’t resolve any of FB’s nameservers. Anyone know how that could happen?

This is a great argument for the antitrust authorities to break up Facebook. Allowing the big social media companies to buy each other creates a single point of failure. If Instagram and WhatsApp were separate companies, a technical disaster at one would not take out the other two.

From this tweet: https://twitter.com/BlazejKrajnak/status/1445063232486531099

"Because of missing DNS records for http://Facebook.com, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."

Just wondering - would the engineer who made the mistake be fired?

tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  eehub.home [192.168.1.254]
  2     3 ms     3 ms     3 ms  172.16.14.63
  3     *        5 ms     3 ms  213.121.98.145
  4     5 ms     3 ms     4 ms  213.121.98.144
  5    17 ms     8 ms    18 ms  87.237.20.142
  6     8 ms     6 ms     7 ms  lag-107.ear3.London2.Level3.net [212.187.166.149]
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     7 ms     7 ms     6 ms  be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
 10    70 ms    69 ms    70 ms  be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
 11    73 ms    73 ms    74 ms  be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
 12    84 ms    85 ms    84 ms  be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
 13    90 ms    90 ms    90 ms  be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
 14   143 ms   142 ms   143 ms  po111.asw02.sjc1.tfbnw.net [173.252.64.102]
 15   114 ms   119 ms   114 ms  be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
 16   125 ms   126 ms   124 ms  be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
 17    91 ms    92 ms    91 ms  po734.psw03.ord2.tfbnw.net [129.134.35.143]
 18    91 ms    93 ms    90 ms  157.240.36.97
 19    74 ms    74 ms    73 ms  a.ns.facebook.com [129.134.30.12]

this is what i got now

DNS servers of a major internet provider in the Czech Republic are down now. Probably not a coincidence (other DNS server's stats show increased traffic so my guess is that Vodafone's DNS servers were unable to cope with the increased traffic and crashed https://twitter.com/BlazejKrajnak/status/1445063232486531099).

It's crazy that half the country doesn't have internet because Facebook stopped working.

Its alive!

drill @1.1.1.1 www.facebook.com ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2172 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; www.facebook.com. IN A

;; ANSWER SECTION: www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com. star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36

pings to a.ns.facebook.com are no longer timing out

Hope it's permanent

Just thinking about all the conspiracy theories you could make of this. Yesterday pandora papers, today the internet stops working.

well... this is unfortunate:

; <<>> DiG 9.10.6 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36072 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  eehub.home [192.168.1.254]
  2     3 ms     3 ms     3 ms  172.16.14.63
  3     \*        5 ms     3 ms  213.121.98.145
  4     5 ms     3 ms     4 ms  213.121.98.144
  5    17 ms     8 ms    18 ms  87.237.20.142
  6     8 ms     6 ms     7 ms  lag-107.ear3.London2.Level3.net [212.187.166.149]
  7     \*        \*        \*     Request timed out.
  8     \*        \*        \*     Request timed out.
  9     7 ms     7 ms     6 ms  be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
 10    70 ms    69 ms    70 ms  be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
 11    73 ms    73 ms    74 ms  be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
 12    84 ms    85 ms    84 ms  be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
 13    90 ms    90 ms    90 ms  be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
 14   143 ms   142 ms   143 ms  po111.asw02.sjc1.tfbnw.net [173.252.64.102]
 15   114 ms   119 ms   114 ms  be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
 16   125 ms   126 ms   124 ms  be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
 17    91 ms    92 ms    91 ms  po734.psw03.ord2.tfbnw.net [129.134.35.143]
 18    91 ms    93 ms    90 ms  157.240.36.97
 19    74 ms    74 ms    73 ms  a.ns.facebook.com [129.134.30.12]

...Including that one

Alle Störungen shows a massive spike in problems for every service it keeps track of: https://xn--allestrungen-9ib.de/

Interesting Timing!

Facebook hacked?

I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I can never remember...

What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days.

Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.

What would a full day of WhatsApp outage mean for the world?

It's about damn time. Hopefully they stay down. It will do the world some good (long term) to have some time away from this platform and platforms like it.

Facebook is proving that it's systemically important by taking the entire site down.....

Zuckerberg is taking his ball and going home unless you stop writing mean things about him /s

Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.

I find myself a little bit happy that it's down. I use Facebook quite often, but mostly because everyone else I know uses it. If everyone is forced to find an alternative, that'd be fine by me.

wonder how much of internet traffic as a whole is down now..

Does this mean Oculus is unusable now?

Finally! A small (or not so small) outage for FB, a large benefit for mankind :)

Good.

Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only.

It's so important to diversify, such as building a website.

When I click the HN link I am presented with facebook login page. I don't have an account so can not proceed.

Is the article link just https://facebook.com/ ?

This morning around 10:00 AM I've been posting on a forum a message in to which I have said that I'm hoping Facebook will get shut down.. Now, Facebook situation you see it might be my fault. Sorry folks, didn't meant that for real.. I still need FB's products! =))

https://natrmd.com

I just got the login page. It was fun while it lasted.

A lot of addict's will now feel how is it to live in the reality not scrolling dump images. I hope this will be tradition at least once a month.

somewhere an engineer is begging a mnesia instance to come back online.

Interestingly, their .onion site¹ is also down.

https://en.wikipedia.org/wiki/Facebook_onion_address

Imagine if they need access to fb.com email to re-enable the access for the on-site technician.

https://www.whatsmydns.net/#A/facebook.com

Anyone able to Connect w WhatsApp at the moment?

Great time to take a break from Facebook and Instagram. Use Telegram instead of WhatsApp

I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.

New York Times has coverage.[1]

"A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo."

[1] https://archive.is/iBzs3

Discord [1] is taking a toll from the increased traffic as well:

"We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service.

We have temporarily disabled typing notifications. We expect these to be re-enabled soon."

[1] https://discordstatus.com/

Someone save a risky release for 9am on a Monday morning? Decided Friday afternoon was too risky?

There's still no connectivity to Facebook's DNS servers:

    > traceroute a.ns.facebook.com
      traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
      1  dsldevice.attlocal.net (192.168.1.254)  0.484 ms  0.474 ms  0.422 ms
      2  107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1)  1.592 ms  1.657 ms  1.607 ms 
      3  71.148.149.196 (71.148.149.196)  1.676 ms  1.697 ms  1.705 ms
      4  12.242.105.110 (12.242.105.110)  11.446 ms  11.482 ms  11.328 ms
      5  12.122.163.34 (12.122.163.34)  7.641 ms  7.668 ms  11.438 ms
      6  cr83.sj2ca.ip.att.net (12.122.158.9)  4.025 ms  3.368 ms  3.394 ms
      7  * * *
      ...

Everything is a f*king Facebook problem

This solves the disinformation problem

They seem to be back now.

I unblocked Facebook right now from my hosts file so I could message someone and couldn't figure out why Facebook failed to load. I tested HN and viola I see that the entire world has sent Facebook requests to 0.0.0.0 lol

Gotta post this every time theres a big DNS issue, which seems daily now.

Check out Dug! Its a global DNS propagation/monitoring toolon the CLI: https://github.com/unfrl/dug/

Yet another reason to not over-rely on a few big tech companies for the majority of the planet's communication. Forget concerns about competition, monopolies and so on for now (as important as they are), what we want are many social networks, video conferencing apps, messenger apps. Every country should strive to build their own Google or FB, or certainly many more should. State-backed if needed. It's a question of resilience and security as much as anything.

Any way to read this without an account?

my home connection with ISP is down Vodafone Ireland, so I guess they have such a big churn in Vodafone from FB BGP routes that it blew Vodafone network. Is it DNS or routing issue?

It's back as of approx 14:47 PST.

Who else sees their deleted messages on WhatsApp that shouldn't be there?

https://twitter.com/Pytlicek/status/1445072626729242637

I'm pretty sure this has been building up since the morning (Germany). I've had odd connectivity problems to a number of sites including slack for a moment.

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow.

makes me think, why dont porn sites ever go down ?

Teen depression and suicide rates plummeting right now

Let's hope it's permanent.

I wish it would stay that way.

India runs on whatsapp. They'll have more backups now.

Is there any place to see how the overall internet bandwidth usage has changed during this outage?

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?

hahaha, good riddance

https://www.cyberciti.biz/humour/a-haiku-about-dns/

facebook.com resolves again!

> ping facebook.com

PING facebook.com (31.13.83.36) 56(84) bytes of data.

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=1 ttl=54 time=12.2 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=2 ttl=54 time=12.1 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=3 ttl=54 time=11.7 ms

Can't yet traceroute to a.ns.facebook.com tho

DNS configuration is becoming a single point of failure. A few weeks ago, many services running out of AWS West 2 failed because the within-the-datacenter DNS system broke down somehow.

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow and "having trouble serving requests".

Signal FTW

And a lot of people seem to be coming to HN to find out why, judging by how laggy HN is getting right now...

The joy that people are getting from this is quite shitty. I hate social media but there are people earning a living working for these companies. Like others have pointed out, businesses and neighborhood watches rely on tech like this. At some point we've all had sites/apps go down, in a situation like that the last thing you want is people enjoying it. The lack of empathy in this thread is telling.

I am sure it is just a symptom of the Facebook outage, but it seems like every website I am going on is slower than usual today.

This would be a golden opportunity to launch your 'Facebook Killer' app. Preferably a social network where people don't pay with their data, but with, you know, a thing called Money.

Here is a handy troubleshooting flowchart for megacorp outages:

> Is it a DNS issue? -> yes

It can be used in reverse as a postmortem too.

Outage is top story on CNN and Fox. Facebook is not returning their calls. Sheera Frenkel at the New York Times has been able to get a little more info, but not much.

Now Twitter is starting to have problems with overload.

So how much do we need to pitch in to keep it down?

This really is a fascinating case-study of what is truly resilient systems. More often than not, they are not centralized.

Where could the physical data centers be that they need to access? How far away could it be?

Looks like someone found the light switch and turned everything back on!

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue:

>As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.

User is providing live updates of the incident here:

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...

"When you are strong, appear weak."

It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it (https://twitter.com/BlazejKrajnak/status/1445063232486531099).

Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)

Was receiving an error page, now just a server not responding error.

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?

[I'm also getting server error trying to submit this comment]

So how much cash do we need to pitch in to keep it down?

The cynic in me wonders if this is related to the Pandora Papers leak

https://www.status.fb.com/ is back online now

Out of band management is an important feature for the reliability of your network.

Oh no

What are some of the possible scnarios beyond the DNS issue suggested? (and might it be an attack?)

Somebody just had their very own "onosecond".

https://www.youtube.com/watch?v=X6NJkWbM1xk

The video is one that Tom Scott published in June 2020 about the worst typo he ever made in one of his prior jobs, and while the Facebook mistake is almost certainly not going to be anything irrecoverable like this one, you can bet that Facebook pride themselves on being available all the time.

Perhaps allowing Facebook, WhatsApp and Instagram to merge was efficient after all - now that they have synchronized outages, people finally have a chance to get on with their lives, free of clickbait news and misinformation.

World productivity just grew by 10%

This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.

Maybe intentional?

Zuck trying to give an example of what a world without FB would look like, kinda saying to detractors what would happen if they had it their way.

Is HN hit by something as well? It's loading really slow for me.

Oculus is also down

The internet felt a little more fresh and clean today.

Even Facebook's Onion site isn't working: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t...

Fascinating simply that it's apparently not just a DNS issue.

I keep trying to submit to HN but I keep getting an error.

What's wrong with the internet?

FaceBook is down.

My friend from Slovenia is having trouble with discord. It eats his messages.

I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.

TrackMania players have talked about having input lag.

ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)

ycombinator turned out to be giving only errors.

Some sites I've found via google results seem to report that they are suffering from slow connections.

Do you have anything to add to this?

Delete Facebook / Instagram / WhatsApp when it comes back up. They are all trash.

Is there any site that tracks number of users for messaging apps? I'd be really curious to see if signal\telegram\etc are seeing a big bump.

I noticed that some websites are loading slowly due to the third party script https://connect.facebook.net/en_US/fbevents.js timing out.

When uBlock Origin is running, this script gets blocked and pages return to feeling snappy.

Reading the thread, I'm surprised at the number of nearly identical "How much do we have to pay to keep it down? xD" posts I'm seeing, often from throwaway accounts. Some accounts with multiple near-identical posts within the same minute.

Could this be a coordinated smear in HN comments?

Somebody moved fast.

Still amazes me their infra team is supposedly the best in the world, and compensated as such, yet things like this happen.

Personally I'm glad FB went down for a few hours, but it's hard to imagine how that would happen in the first place.

makes me wonder, why dont porn sites ever seem to go down ?

Have they tried turning it off and then turning it back on again?

Its always a DNS problem

Whataspp too. This seems pretty big

I had problems with my internet connection and loaded my ISPs site. Strangely, my bill was paid. Even stranger, some sites load while others do not.

Then it hit me: I am so dependent on Facebook owned properties (Whatsapp, Facebook, insta) that a Facebook failure looks to me like an internet failure.

Also instagram and messenger

Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don't you get that?

I keep trying to submit to HN but I keep getting an error.

What's wrong with the internet?

FaceBook is down.

My friend from Slovenia is having trouble with discord. It eats his messages.

I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.

TrackMania players have talked about having input lag.

ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)

ycombinator turned out to be giving only errors, but now seems to be working occasionally. I can not submit anything, though.

Some sites I've found via google results seem to report that they are suffering from slow connections.

Do you have anything to add to this?

Hopefully forever.

Who else sees their deleted messages on WhatsApp that shouldn't be there?

https://news.ycombinator.com/item?id=28749652

I am soooo happy I did not even notice! Don’t use any of these apps.

I want to to thank you for this good read!! I definitely enjoyed every little bit of it. I've got you book-marked to look at new stuff you post…

http://infocampus.co.in/ui-development-training-in-bangalore... https://www.firstenquiry.com/ui-development-training-in-bang... http://infocampus.co.in/web-designing-training-in-bangalore.... http://infocampus.co.in/web-development-training-in-bangalor... http://infocampus.co.in/react-js-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.htm...

When I do dig instagram.com I get an A response for this IP: 31.13.65.174 or similar addresses, which leads to an empty page.

The level of intelligence of the comments in this thread kinda confirm my suspicions that the armchair experts from Reddit et al (including myself) are discovering this site.

Facebook outage is now the top story on CNN and Fox. Facebook stock down 5%. Facebook is not returning calls from Fox, or CNN.

DNS?

Good riddance.

downdetector looks like a real mess for it.

I'm going to parrot the other comment here and say nothing of value was lost.

https://downdetector.com/status/facebook/

Expecting to get messages on WhatsApp alternatives tonight ...

Ignore.

Facebook and Messenger are working now.

Instagram and WhatsApp - not yet.

Nothing of value was lost

The timing of this is so rich in irony I can't help but wonder if there is an element of internal sabotage. How many FB employees hate FB right now? The latest expose of FB is both effective and truly awful. I can't imagine feeling good about a FB job. And it's gotten worse! Now they look like they can't even keep their websites up.

When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt.

I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.

hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.

Did I just read that the Facebook IRC fallback went down too?!? I was about to say what’s wrong with freenode ( but yeah on 2nd thoughts let’s not talk about freenode )

Aren't there places around poorer countries where Facebook is basically an ISP? What about them? They have literally 0 info.

https://tcrn.ch/3kOHco1

Wow. I can’t remember the last time whatsapp was done. I pretty much use messenger/instagram/whatsapp to talk to most of my friends and family. I’m happy that I do use other platforms otherwise I would be completely cut off from my parents right now.

Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.

https://twitter.com/sheeraf/status/1445099150316503057

I guess the "prophets" at Victory Channel / Flashpoint called down Holy Fire on the Facebook infrastructure in retribution ... https://youtu.be/FbSkFuvqFdA?t=1127 . (I'm an Evangelical Christian but those folks are nuts ... Mario Murillo, Lance Wallnau, Hank Kunneman, Gene Bailey, etc.)

Hopefully it never comes back up!

Terrible day for many people. Both working for Facebook and those depending on their services.

They made they own BGP tools and looks like it failed https://www.youtube.com/watch?v=wHfYUbKNEyc

What I think is interesting is the effects of this type of thing across peripheral news sites, like HN. I wonder how much spike HN gets with people rushing here to find out what's going on and to read the (articulate) related discussions.

Let this be permanent - not a huge loss for humanity.

The whistleblower said she wanted to fix Facebook.

Mission accomplished, I'd say. For now at least.

"It's always DNS."

Aren't there places around poorer countries where Facebook is basically an ISP? What about them?

https://tcrn.ch/3kOHco1 2Africa cable, as an example

Just a sidenote I think outlook is down as well.

From [0]

> ...there is no limit to the scandals, leaks, whistleblowers, lawsuits or penalties that will bring the Facebook mafia down.

Fine. 'Literally' bringing the Facebook mafia down like that would do.

But only for now.

[0] https://news.ycombinator.com/item?id=28742179

Hopefully permanently.

In this context, I remember youtube+pakistan issue[1]. I also wonder how an AS/BGP manager do his/her job... I imagine a guy/girl changing a text file in a old console. Anyone knows?

[1] https://www.infoworld.com/article/2648947/youtube-outage-und...

They must have shut it off and turned it back on

I keep getting non-dns errors from Hackner News as well. There appears to be some sort of broader incident happening?

It's not just lag, I keep getting the "We're having some trouble serving your request. Sorry!" page.

Edit: HN related thread https://news.ycombinator.com/item?id=28749476

It's funny that hackernews is now overloaded with distracted people ;-)

Also Speedtest.net for me is showing a 503 error page. Seems a large CDN might be having problems. Their status page shows all green. FB and their other sites are also down.

edit: I see it's back up and I've been getting downvoted, here's a screenshot of the error for clarity

https://i.imgur.com/wvhOwwL.png

Checked isitdownrightnow.com and said Netflix was also down. Any chance these are related?

It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections.

I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.

Would be very interesting if they release the RCA to the public

Sir, I don't care who you are, you must open a ticket.

Seems to be affecting all Facebook properties.

some poor engineer is sobbing over a split brain mnesia cluster right now praying to get the thing back up.