Hacker News Re-Imagined

Facebook-owned sites were down

  • 2589 points
  • 9 months ago

  • @nabeards
  • Created a post

Facebook-owned sites were down


@platz 9 months

Replying to @nabeards 🎙

[redacted]

Reply


@throwawaylolx 9 months

Replying to @nabeards 🎙

When I noticed HN was loading slowly, I already knew FB was down.

Reply


@ballenf 9 months

Replying to @nabeards 🎙

One real potential cost to FB here is breaking people's addictions to FB and IG. This might just be the little finger-snap to wake up a sizable chunk of the user base that they life is just a little better during the outage.

Reply


@rootinier 9 months

Replying to @nabeards 🎙

nslookup www.facebook.com 8.8.8.8 Server: 8.8.8.8 Address: 8.8.8.8#53

* server can't find www.facebook.com: SERVFAIL

Reply


@DebtDeflation 9 months

Replying to @nabeards 🎙

I assume all of the tertiary sites that use "Login with Facebook" are broken now too? So glad I never adopted that.

Reply


@pmlittle 9 months

Replying to @nabeards 🎙

This is all total left brain looping and complexity coming home to roost.

Reply


@kaustubhvp 9 months

Replying to @nabeards 🎙

it is always DNS!

Reply


@gprasanth 9 months

Replying to @nabeards 🎙

Suspecting it might be related to the recent letsencrypt cert authority expiring? Was just debugging an issue earlier today and just couldn't help wondering how much of the internet is secured by letsencrypt.

All of the static hosts providing free SSL: vercel, netlify, render, firebase hosting, github pages, heroku etc. ...

It does work on modern browsers and devices but goes terribly broken on a lot of old devices.

Reply


@tomerbd 9 months

Replying to @nabeards 🎙

No like for you

Reply


@bubblehack3r 9 months

Replying to @nabeards 🎙

Their stock is down 5% too. Everything is down for them today ;)

Reply


@typingmonkey 9 months

Replying to @nabeards 🎙

If it is an DNS error, why is the .onion site also offline?

- https://en.wikipedia.org/wiki/Facebook_onion_address

- facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion

Reply


@susahahhaha 9 months

Replying to @nabeards 🎙

add comment

Reply


@aritraghosh007 9 months

Replying to @nabeards 🎙

The down page shows copyright from 2020 smh

Reply


@fartingflamingo 9 months

Replying to @nabeards 🎙

From the archived ramenporn reddit comment thread at [0]:

> This must be incredibly stressful so for your sake I hope you sort it out quickly... but for the world's sake, I hope you fail and make the problem worse before jumping ship followed by every other engineer, leaving it to Zuckerberg to fix himself. But I still hope it's not too stressful for you!

https://archive.is/Idsdl

Reply


@ur-whale 9 months

Replying to @nabeards 🎙

> Facebook-owned sites are down

And the world rejoiced.

Reply


@new-day-rising 9 months

Replying to @nabeards 🎙

Thoughts and prayers...

Reply


@CodeGlitch 9 months

Replying to @nabeards 🎙

Looks like HN is being hit pretty hard right now?

Reply


@mrfusion 9 months

Replying to @nabeards 🎙

Thought experiment: what if they were down for a week and the world completely healed itself?

Reply


@quattrofan 9 months

Replying to @nabeards 🎙

And right now, the world is a slightly better place

Reply


@gunshai 9 months

Replying to @nabeards 🎙

Posting this comment will be like farting into a hurricane, but here goes.

Company like Facebook has a serious problem and their stock drops ... precipitously. CEO of said company instead of selling their equity in their company has taken out loans against their equity in order to decrease their tax burden and cash in on the value of their equity.

What amount of decrease would cause a margin call from lenders for the forced sale of said equity and subsequently the loss of majority stake in their own company? Now obviously only the lenders know this information and assuming I have the rough order of operations correct.

Could this be a potential chink in the armor of founders / CEOs / anyone who takes out low interest loans against the equity they hold in their company? Maybe my understanding of this is too simplified.

Reply


@husainhz7 9 months

Replying to @nabeards 🎙

Whatsapp's down too. Tough month for FB, especially with the leak.

Reply


@TrackerFF 9 months

Replying to @nabeards 🎙

Anyone wanna estimate the cost of total downtime for facebook and instagram, as far as lost ad revenue goes - per minute?

Reply


@emptysea 9 months

Replying to @nabeards 🎙

Interesting, even some open source sites like: https://fbinfer.com are down

but https://glean.software and https://reactjs.org aren't

Reply


@rcarmo 9 months

Replying to @nabeards 🎙

Unsurprisingly, Oculus is down as well, as are most services for the VR headset. So that's 4 major properties right now.

Reply


@tut-urut-utut 9 months

Replying to @nabeards 🎙

Facebook down, WhatsApp down, but Signal still works. Time for a change?

EDIT: Yes, Signal is not federated, but that's what people are at least ready to consider as a WhatsApp alternative. I also created Matrix / Element account, and had 0 contacts using it already.

Reply


@sparrish 9 months

Replying to @nabeards 🎙

Seems to be DNS related.

None of the listed facebook nameservers are resolvable or reachable:

a.ns.facebook.com b.ns.facebook.com c.ns.facebook.com d.ns.facebook.com

Reply


@blablablub 9 months

Replying to @nabeards 🎙

Glad to report that facebook's dns in china is not affected. You can dig facebook.com and the depth of the internet happily reply with a random ip address as usual.

Reply


@ivanmontillam 9 months

Replying to @nabeards 🎙

One thing that triggers my OCD is leaving the Facebook session open, though it's my own computer.

Maybe it's DNS. It's always DNS.

Reply


@yawnxyz 9 months

Replying to @nabeards 🎙

I thought Facebook, Instagram and WhatsApp ran on different infrastructure (and they've been trying for a while to align everything)?

How could they all go down at the same time, if they have different teams of engineers running each product separately?

Could anyone with some background (or person familiar with the matter) explain how their system's set up?

Reply


@cblconfederate 9 months

Replying to @nabeards 🎙

It's like a nuclear bomb exploded on the internet.

Reply


@asduoihfijnu 9 months

Replying to @nabeards 🎙

add comment

Reply


@ionwake 9 months

Replying to @nabeards 🎙

Why is this not the first post on hn? It’s double the points and has over 590 comments?

Reply


@raimille1 9 months

Replying to @nabeards 🎙

Whatsapp down as well

Reply


@jleyank 9 months

Replying to @nabeards 🎙

Somebody published vogon poetry and pictures and the internet routed around the damage...

Reply


@_yo2u 9 months

Replying to @nabeards 🎙

value of fb directly proportional to traffic flow curious about why fb is down ;)

Reply


@EastOfTruth 9 months

Replying to @nabeards 🎙

We can only hope that they will be gone forever... and HN is having major issues at the same time!

Reply


@elboru 9 months

Replying to @nabeards 🎙

Is it just me or HN also feels kinda laggy?

Reply


@StapleHorse 9 months

Replying to @nabeards 🎙

That explain the new contacts in Telegram.

Reply


@klik99 9 months

Replying to @nabeards 🎙

Is this related to the outages from lets encrypt root cert expiring? Probably not since this looks like a DNS issue, but still it's a crazy coincidence that two major internet breaking events happen in the same week

Reply


@tomohawk 9 months

Replying to @nabeards 🎙

Many local governments use FB to get info out.

Events like this show they should use multiple outlets instead of the big monopoly.

Alternatives like gab exist, but its incredibly hard to gain traction against the big monopolies.

Reply


@EB66 9 months

Replying to @nabeards 🎙

Is anyone else seeing knock-on effects at the other major public DNS providers? I'm seeing nslookups sent to 4.2.2.2 and 8.8.8.8 intermittently timeout if the hostname does not belong to a major website. CloudFlare DNS (1.1.1.1) doesn't appear to be impacted. For example:

[root@app ~]# nslookup downforeveryoneorjustme.com 4.2.2.2 ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached

[root@app ~]# nslookup downforeveryoneorjustme.com 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53

Non-authoritative answer: Name: downforeveryoneorjustme.com Address: 172.67.166.187 Name: downforeveryoneorjustme.com Address: 104.21.91.48

[root@app ~]#

Perhaps DNS queries are skyrocketing and overwhelming some of the major public DNS servers.

Reply


@mcintyre1994 9 months

Replying to @nabeards 🎙

I love how understated companies always are about things like this.

> Facebook said: "We are aware some people are having trouble accessing our apps and products. We are working to get things back to normal as quickly as possible and apologise for any inconvenience."

https://www.bbc.co.uk/news/technology-58793174

Reply


@LuisMondragon 9 months

Replying to @nabeards 🎙

Some Oculus Quest owners can't use their device https://www.reddit.com/r/OculusQuest/comments/q18xwy/faceboo...

Reply


@littlecranky67 9 months

Replying to @nabeards 🎙

The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?

Reply


@EGreg 9 months

Replying to @nabeards 🎙

Telegram seems down too, is it down for you?

Reply


@asduoihfijnu 9 months

Replying to @nabeards 🎙

ok mom i commented

Reply


@vvpan 9 months

Replying to @nabeards 🎙

Not an indepth technical comment here but: seeing a tech megacorp go offline for a day makes me very very happy.

Reply


@MrStonedOne 9 months

Replying to @nabeards 🎙

Rumormill is suggesting that facebook badge readers are also down causing issues with trying to get to the servers to manually fix them.

https://twitter.com/sheeraf/status/1445099150316503057?s=21

Reply


@sAbakumoff 9 months

Replying to @nabeards 🎙

rich people serve their revenge for pandora papers

Reply


@zrail 9 months

Replying to @nabeards 🎙

hugops for the engineers having to deal with this. It's incredibly stressful and I personally feel like they deserve some empathy, even if I don't like Facebook.

I wonder if maybe part of the lesson will be to run the root of your authoritative DNS hierarchy on separate infrastructure with a separate domain name. Using facebook.com as your root is cool and all but when that label disappears it causes huge issues.

Reply


@koprulusector 9 months

Replying to @nabeards 🎙

Reddit wasn’t working a few min ago. Broader issue?

Reply


@liendolucas 9 months

Replying to @nabeards 🎙

Does anyone have a reasonable guess on how much money they have already lost?

Reply


@cphoover 9 months

Replying to @nabeards 🎙

now if only tiktok would fail

Reply


@OnceUponADevops 9 months

Replying to @nabeards 🎙

Confirming that we're seeing a major outage with all of our integrations with FB products.

Reply


@elzbardico 9 months

Replying to @nabeards 🎙

Rejoice! The revolution has started!

Yes, I know humor is not welcome in HN

Reply


@mkaszkowiak 9 months

Replying to @nabeards 🎙

This outage is huge. I'm waiting for the write-up, assuming they release one

Reply


@jeffrom 9 months

Replying to @nabeards 🎙

Is facebook being down causing hacker news to get the hug of death??

Reply


@pmlittle 9 months

Replying to @nabeards 🎙

This is all left brain implimentation with looping and classic complexity coming home to roost. As we move through time, we build off of solutions of the past which are solving a problem, but complexity keeps adding on and this is a classic programming/computer science delemma.

Reply


@m0guz 9 months

Replying to @nabeards 🎙

Ironic enough; status.fb.com also down.

Reply


@russellbeattie 9 months

Replying to @nabeards 🎙

It's not hyperbole to say that this is going to literally save lives.

Cutting off Facebook's firehouse of hate and misinformation for just a couple hours is going to have a obvious positive effect on millions of people. At this scale, at least one person will get vaccinated today because they didn't see the wall of ignorance that is FB's news feed.

Maybe we should introduce "digital blue laws", where one day a week, social media is shut down for the overall good of society.

Reply


@oldabc 9 months

Replying to @nabeards 🎙



@boramalper 9 months

Replying to @nabeards 🎙



@cemoktra 9 months

Replying to @nabeards 🎙

Imagine this happening to AWS

Reply


@jader201 9 months

Replying to @nabeards 🎙

FB seems to be finally loading for me, after nearly 6 hours.

This will be a highly discussed topic for a bit.

Reply


@akshayrajp 9 months

Replying to @nabeards 🎙

As are Instagram and WhatsApp

Reply


@foobaw 9 months

Replying to @nabeards 🎙

This is taking a longer time than expected for a company like Facebook - must be serious where a rollback isn't possible or trivial.

Reply


@em3rgent0rdr 9 months

Replying to @nabeards 🎙

How much revenue does facebook loose per hour down?

Reply


@codediesel 9 months

Replying to @nabeards 🎙

still down, people going to signal to have a chat

Reply


@throwaway78981 9 months

Replying to @nabeards 🎙



@husamia 9 months

Replying to @nabeards 🎙

do we know what caused the DNS outage at FB?

Reply


@jacobwilliamroy 9 months

Replying to @nabeards 🎙

Should say "Facebook owned. Sites are down."

Reply


@yawnxyz 9 months

Replying to @nabeards 🎙

Funny enough, I went to https://www.isitdownrightnow.com/ to check if Facebook is down, and isitdownrightnow is down itself... probably from the massive number requests coming to check if Facebook is down

Reply


@j3th9n 9 months

Replying to @nabeards 🎙

I like it, it feels like it's 1999.

Reply


@geocrasher 9 months

Replying to @nabeards 🎙



@dghughes 9 months

Replying to @nabeards 🎙

Twitter seems to be a big buggy now too maybe just a coincidence. User comments under posts are not appearing.

Reply


@attende_domine 9 months

Replying to @nabeards 🎙

  % ping whatsapp.com
  ping: whatsapp.com: Name or service not known
  % ping web.whatsapp.com
  ping: web.whatsapp.com: Name or service not known
  % ping facebook.com
  ping: facebook.com: Name or service not known
  % ping instagram.com
  PING instagram.com (31.13.65.174) 56(84) bytes of data.
  64 bytes from 31.13.65.174 (31.13.65.174): icmp_seq=1 ttl=53 time=110 ms

Reply


@chasd00 9 months

Replying to @nabeards 🎙

can productivity (or emotional stability) for the overall US economy be tracked on a daily basis? I wonder if a wholesale facebook outage would show up on that graph as a brief blip in the positive direction.

Reply


@qualudeheart 9 months

Replying to @nabeards 🎙

If it bleeds we can kill it!

Reply


@underscore_ku 9 months

Replying to @nabeards 🎙

good

Reply


@joelbondurant 9 months

Replying to @nabeards 🎙

LOL! They should have used the USA Fact-Check Algorithm from the Science Ministry.

Reply


@erjjones 9 months

Replying to @nabeards 🎙

Does everyone just buy in that this is just a network change gone wrong? OR could they be mitigating a breach/hack? OR could it be some other theory?

Reply


@smashah 9 months

Replying to @nabeards 🎙

Good, now I can go for lunch.

Reply


@unusximmortalis 9 months

Replying to @nabeards 🎙

for me even this site loads very very slowly. pinging google name server is fast as usual. it could be a more wide problem not just FB related.

Reply


@TedShiller 9 months

Replying to @nabeards 🎙

This is honestly the best feature Facebook has ever developed. I hope it's permanent. It has the following effects: you feel better about yourself, you can spend more time with your family, you are more productive.

Reply


@jontro 9 months

Replying to @nabeards 🎙

Getting everything back up again will probably be a nightmare. Imagine all the internal services trying to reach a consistent state after such a long outage.

Reply


@AzzieElbab 9 months

Replying to @nabeards 🎙

it is probably unrelated but HN is crawling

Reply


@mef 9 months

Replying to @nabeards 🎙

Looks like the routes to their hosted nameservers are down, e.g. A.NS.FACEBOOK.COM

Reply


@liendolucas 9 months

Replying to @nabeards 🎙

Perhaps tomorrow, the brave man or woman responsible for this beautiful screw up will step forward in HN for an outstanding ovation. Whoever did this, thank you! As a souvenir I took a screenshot on my phone.

Reply


@waltbosz 9 months

Replying to @nabeards 🎙

I'm curious if this extended outage will do anything to curb the dopamine addition caused by facebook.

For example, will FB addicts experience a day of repeated failed attempts to get their FB fix, which will then condition them to stop trying.

Reply


@rvnx 9 months

Replying to @nabeards 🎙

Fixed

Reply


@ricardo81 9 months

Replying to @nabeards 🎙

Doesn't seem too clever that Facebook's NS servers are a.ns.facebook.com, b.ns.facebook.com etc. IIRC that kind of setup requires some glue records.

Reply


@imalerba 9 months

Replying to @nabeards 🎙

Seems like Telegram went down with a big whatsapp-is-down hug of death.

Reply


@teddyh 9 months

Replying to @nabeards 🎙

Here we can see why you should not have all your DNS servers in the same AS (in this case, AS32934).

Reply


@marstall 9 months

Replying to @nabeards 🎙

between outage and whistleblower, this has got to be the worst day in facebook's life

Reply


@glanzwulf 9 months

Replying to @nabeards 🎙

Oh no...

Anyway...

Reply


@6510 9 months

Replying to @nabeards 🎙

Reminds me of a story Jack Fresco use to tell were financial workers were unable to get to work because a bridge was not usable. People were worried about terrible consequences if all these important people were unable to do their work. To their surprise life just continued as if nothing changed.

Reply


@emmap21 9 months

Replying to @nabeards 🎙

There is a global outage this morning starting from 8:00 AM. This list has Google, Tiktok, Zoom, Slack and of course FB products and services.

Reply


@rocho 9 months

Replying to @nabeards 🎙

For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL:

    $ dig facebook.com

    ; <<>> DiG 9.16.21 <<>> facebook.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23982
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;facebook.com.   IN A

    ;; Query time: 16 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Mon Oct 04 17:53:00 CEST 2021
    ;; MSG SIZE  rcvd: 41

Reply


@htrp 9 months

Replying to @nabeards 🎙

In the post-mortem, we'll find out that Facebook's alerting and comms systems all run on Facebook. As a result, they can't even coordinate the restart to roll back changes.

Reply


@pixelgeek 9 months

Replying to @nabeards 🎙

I would like that say that after my "burn it down" comments on another Facebook related post that I had nothing to do with this.

Reply


@mro_name 9 months

Replying to @nabeards 🎙

frankly, who cares? Seriously.

Those services are toxic for years now and everybody knows that. Who still uses them occasionally let alone relies on them can't be helped, can they?

Reply


@mdani 9 months

Replying to @nabeards 🎙

Is this in some way connected to the Facebook data leak of 1.5 billion users? The timing seems quite odd that both these things happen around the same time.

Reply


@clipradiowallet 9 months

Replying to @nabeards 🎙

In other news, a bunch of people got a lot more work done today than normal I suspect...

Reply


@polack 9 months

Replying to @nabeards 🎙

After changing the screen resolution all operating systems will prompt the user if the applied settings where correct, otherwise it will time out and reset to last known good setting. Maybe time for the core internet infrastructure to implement something similar? :)

Reply


@Nasreddin_Hodja 9 months

Replying to @nabeards 🎙

Ha-ha!

Reply


@the-dude 9 months

Replying to @nabeards 🎙

It is about time to speculate about sabotage, a disgruntled employee or something more exotic.

All this BGP talk is boring.

Reply


@shockeychap 9 months

Replying to @nabeards 🎙

I feel for the sysadmins who are fighting ulcers and migraines at the moment, but I can't shake feeling that the world is just a little bit better for this small window of time.

Reply


@korethr 9 months

Replying to @nabeards 🎙

At 21:44 UTC, facebook.com resolves for me.

Reply


@AtNightWeCode 9 months

Replying to @nabeards 🎙

So they managed to remove facebook.com from 1.1.1.1 and 8.8.8.8. That is impressive. Not something anyone can achieve in such short time by even trying.

Reply


@segah 9 months

Replying to @nabeards 🎙

My father (https://bit.ly/3acZAAI), who is a certified CCSP Ethical Hacker and formerly worked @ZScaler/Checkpoint/Palo Alto Networks, would say that there are basically two scenarios: someone like him did it intentionally or someone like him did it by mistake.

Any other scenario of outsiders, code updates, etc - basically misses the point of how modern DNS infrastructure works.

Reply


@natas 9 months

Replying to @nabeards 🎙

good ridance

Reply


@jamespwilliams 9 months

Replying to @nabeards 🎙

Is this caused by missing glue records? I can’t resolve any of FB’s nameservers. Anyone know how that could happen?

Reply


@not2b 9 months

Replying to @nabeards 🎙

This is a great argument for the antitrust authorities to break up Facebook. Allowing the big social media companies to buy each other creates a single point of failure. If Instagram and WhatsApp were separate companies, a technical disaster at one would not take out the other two.

Reply


@decrypt 9 months

Replying to @nabeards 🎙

From this tweet: https://twitter.com/BlazejKrajnak/status/1445063232486531099

"Because of missing DNS records for http://Facebook.com, every device with FB app is now DDoSing recursive DNS resolvers. And it may cause overloading ..."

Reply


@rainboiboi 9 months

Replying to @nabeards 🎙

Just wondering - would the engineer who made the mistake be fired?

Reply


@amediauk1 9 months

Replying to @nabeards 🎙

tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  eehub.home [192.168.1.254]
  2     3 ms     3 ms     3 ms  172.16.14.63
  3     *        5 ms     3 ms  213.121.98.145
  4     5 ms     3 ms     4 ms  213.121.98.144
  5    17 ms     8 ms    18 ms  87.237.20.142
  6     8 ms     6 ms     7 ms  lag-107.ear3.London2.Level3.net [212.187.166.149]
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     7 ms     7 ms     6 ms  be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
 10    70 ms    69 ms    70 ms  be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
 11    73 ms    73 ms    74 ms  be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
 12    84 ms    85 ms    84 ms  be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
 13    90 ms    90 ms    90 ms  be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
 14   143 ms   142 ms   143 ms  po111.asw02.sjc1.tfbnw.net [173.252.64.102]
 15   114 ms   119 ms   114 ms  be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
 16   125 ms   126 ms   124 ms  be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
 17    91 ms    92 ms    91 ms  po734.psw03.ord2.tfbnw.net [129.134.35.143]
 18    91 ms    93 ms    90 ms  157.240.36.97
 19    74 ms    74 ms    73 ms  a.ns.facebook.com [129.134.30.12]
Trace complete.

this is what i got now

Reply


@jurajmlich 9 months

Replying to @nabeards 🎙

DNS servers of a major internet provider in the Czech Republic are down now. Probably not a coincidence (other DNS server's stats show increased traffic so my guess is that Vodafone's DNS servers were unable to cope with the increased traffic and crashed https://twitter.com/BlazejKrajnak/status/1445063232486531099).

It's crazy that half the country doesn't have internet because Facebook stopped working.

Reply


@antocv 9 months

Replying to @nabeards 🎙

Its alive!

drill @1.1.1.1 www.facebook.com ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 2172 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; www.facebook.com. IN A

;; ANSWER SECTION: www.facebook.com. 3401 IN CNAME star-mini.c10r.facebook.com. star-mini.c10r.facebook.com. 3403 IN A 31.13.72.36

Reply


@david_acm 9 months

Replying to @nabeards 🎙

pings to a.ns.facebook.com are no longer timing out

Reply


@cryptodan 9 months

Replying to @nabeards 🎙

Hope it's permanent

Reply


@xmpir 9 months

Replying to @nabeards 🎙

Just thinking about all the conspiracy theories you could make of this. Yesterday pandora papers, today the internet stops working.

Reply


@blobbers 9 months

Replying to @nabeards 🎙

well... this is unfortunate:

; <<>> DiG 9.10.6 <<>> facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36072 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

Reply


@amediauk1 9 months

Replying to @nabeards 🎙

tracert 129.134.30.12

Tracing route to a.ns.facebook.com [129.134.30.12] over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  eehub.home [192.168.1.254]
  2     3 ms     3 ms     3 ms  172.16.14.63
  3     \*        5 ms     3 ms  213.121.98.145
  4     5 ms     3 ms     4 ms  213.121.98.144
  5    17 ms     8 ms    18 ms  87.237.20.142
  6     8 ms     6 ms     7 ms  lag-107.ear3.London2.Level3.net [212.187.166.149]
  7     \*        \*        \*     Request timed out.
  8     \*        \*        \*     Request timed out.
  9     7 ms     7 ms     6 ms  be2871.ccr42.lon13.atlas.cogentco.com [154.54.58.185]
 10    70 ms    69 ms    70 ms  be2101.ccr32.bos01.atlas.cogentco.com [154.54.82.38]
 11    73 ms    73 ms    74 ms  be3600.ccr22.alb02.atlas.cogentco.com [154.54.0.221]
 12    84 ms    85 ms    84 ms  be2879.ccr22.cle04.atlas.cogentco.com [154.54.29.173]
 13    90 ms    90 ms    90 ms  be2718.ccr42.ord01.atlas.cogentco.com [154.54.7.129]
 14   143 ms   142 ms   143 ms  po111.asw02.sjc1.tfbnw.net [173.252.64.102]
 15   114 ms   119 ms   114 ms  be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89]
 16   125 ms   126 ms   124 ms  be3038.ccr32.slc01.atlas.cogentco.com [154.54.42.97]
 17    91 ms    92 ms    91 ms  po734.psw03.ord2.tfbnw.net [129.134.35.143]
 18    91 ms    93 ms    90 ms  157.240.36.97
 19    74 ms    74 ms    73 ms  a.ns.facebook.com [129.134.30.12]
Trace complete.

Reply


@wly_cdgr 9 months

Replying to @nabeards 🎙

...Including that one

Reply


@nicbou 9 months

Replying to @nabeards 🎙

Alle Störungen shows a massive spike in problems for every service it keeps track of: https://xn--allestrungen-9ib.de/

Reply


@iseanstevens 9 months

Replying to @nabeards 🎙

Interesting Timing!

Reply


@suyash 9 months

Replying to @nabeards 🎙

Facebook hacked?

Reply


@jleyank 9 months

Replying to @nabeards 🎙

I would have thought that these companies that are richer then $GOD would have (virtual) instances of at least the previous stable version available for situations such as this. It would at least keep their damn doors open and internal communications systems going... Maybe they'll NOW think of such things? What's the cliche, penny wise and pound foolish? Or is it, no need to listen to experienced Network Designers? I can never remember...

Reply


@matsemann 9 months

Replying to @nabeards 🎙

What I find weird is that there is no indication in the app that nothing is working. I just get a cached view of everything I've seen the last few days.

Which is a feature I hate, since it does that all the time even when I have a connection. Says there are 3 comments on a post, when I know there is more. Opening them doesnt show them, and no way to refresh. But going to the web page I can see them.

Reply


@caturopath 9 months

Replying to @nabeards 🎙

What would a full day of WhatsApp outage mean for the world?

Reply


@gillytech 9 months

Replying to @nabeards 🎙

It's about damn time. Hopefully they stay down. It will do the world some good (long term) to have some time away from this platform and platforms like it.

Reply


@htrp 9 months

Replying to @nabeards 🎙

Facebook is proving that it's systemically important by taking the entire site down.....

Zuckerberg is taking his ball and going home unless you stop writing mean things about him /s

Reply


@emmap21 9 months

Replying to @nabeards 🎙

Not only facebook, but also Google, Zoom, Telegram, Youtube and many more internet service/ product/ providers from 8:00 AM today. This is more like internet outage.

Reply


@barbs 9 months

Replying to @nabeards 🎙

I find myself a little bit happy that it's down. I use Facebook quite often, but mostly because everyone else I know uses it. If everyone is forced to find an alternative, that'd be fine by me.

Reply


@totaldude87 9 months

Replying to @nabeards 🎙

wonder how much of internet traffic as a whole is down now..

Reply


@aero-glide2 9 months

Replying to @nabeards 🎙

Does this mean Oculus is unusable now?

Reply


@pat-jay 9 months

Replying to @nabeards 🎙

Finally! A small (or not so small) outage for FB, a large benefit for mankind :)

Reply


@carrja99 9 months

Replying to @nabeards 🎙

Good.

Reply


@bhartzer 9 months

Replying to @nabeards 🎙

Facebook being down makes me think of all of those small businesses who never built websites. They rely on traffic and publicity from their Facebook pages only.

It's so important to diversify, such as building a website.

Reply


@avodonosov 9 months

Replying to @nabeards 🎙

When I click the HN link I am presented with facebook login page. I don't have an account so can not proceed.

Is the article link just https://facebook.com/ ?

Reply


@bitzlab 9 months

Replying to @nabeards 🎙

This morning around 10:00 AM I've been posting on a forum a message in to which I have said that I'm hoping Facebook will get shut down.. Now, Facebook situation you see it might be my fault. Sorry folks, didn't meant that for real.. I still need FB's products! =))

https://natrmd.com

Reply


@nazgulsenpai 9 months

Replying to @nabeards 🎙

I just got the login page. It was fun while it lasted.

Reply


@shmiga 9 months

Replying to @nabeards 🎙

A lot of addict's will now feel how is it to live in the reality not scrolling dump images. I hope this will be tradition at least once a month.

Reply


@keithnoizu 9 months

Replying to @nabeards 🎙

somewhere an engineer is begging a mnesia instance to come back online.

Reply


@teddyh 9 months

Replying to @nabeards 🎙

Interestingly, their .onion site¹ is also down.

https://en.wikipedia.org/wiki/Facebook_onion_address

Reply


@john37386 9 months

Replying to @nabeards 🎙

Imagine if they need access to fb.com email to re-enable the access for the on-site technician.

Reply


@kingkool68 9 months

Replying to @nabeards 🎙



@EGreg 9 months

Replying to @nabeards 🎙

Anyone able to Connect w WhatsApp at the moment?

Reply


@html5web 9 months

Replying to @nabeards 🎙

Great time to take a break from Facebook and Instagram. Use Telegram instead of WhatsApp

Reply


@chadlavi 9 months

Replying to @nabeards 🎙

I recognize that for WhatsApp users around the globe this is probably more than an inconvenience, but the rest of humanity is getting something of a reprieve here.

Reply


@Animats 9 months

Replying to @nabeards 🎙

New York Times has coverage.[1]

"A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo."

[1] https://archive.is/iBzs3

Reply


@Ristovski 9 months

Replying to @nabeards 🎙

Discord [1] is taking a toll from the increased traffic as well:

"We're noticing an elevated level of usage for the time of day and are currently monitoring the performance of our systems. We do not anticipate this resulting in any impact to the service.

We have temporarily disabled typing notifications. We expect these to be re-enabled soon."

[1] https://discordstatus.com/

Reply


@dkarp 9 months

Replying to @nabeards 🎙

Someone save a risky release for 9am on a Monday morning? Decided Friday afternoon was too risky?

Reply


@Animats 9 months

Replying to @nabeards 🎙

There's still no connectivity to Facebook's DNS servers:

    > traceroute a.ns.facebook.com
      traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
      1  dsldevice.attlocal.net (192.168.1.254)  0.484 ms  0.474 ms  0.422 ms
      2  107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1)  1.592 ms  1.657 ms  1.607 ms 
      3  71.148.149.196 (71.148.149.196)  1.676 ms  1.697 ms  1.705 ms
      4  12.242.105.110 (12.242.105.110)  11.446 ms  11.482 ms  11.328 ms
      5  12.122.163.34 (12.122.163.34)  7.641 ms  7.668 ms  11.438 ms
      6  cr83.sj2ca.ip.att.net (12.122.158.9)  4.025 ms  3.368 ms  3.394 ms
      7  * * *
      ...
So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.

Reply


@_joel 9 months

Replying to @nabeards 🎙

Everything is a f*king Facebook problem

Reply


@TedShiller 9 months

Replying to @nabeards 🎙

This solves the disinformation problem

Reply


@spurgu 9 months

Replying to @nabeards 🎙

They seem to be back now.

Reply


@m_coder 9 months

Replying to @nabeards 🎙

I unblocked Facebook right now from my hosts file so I could message someone and couldn't figure out why Facebook failed to load. I tested HN and viola I see that the entire world has sent Facebook requests to 0.0.0.0 lol

Reply


@monkaiju 9 months

Replying to @nabeards 🎙

Gotta post this every time theres a big DNS issue, which seems daily now.

Check out Dug! Its a global DNS propagation/monitoring toolon the CLI: https://github.com/unfrl/dug/

Reply


@jmfldn 9 months

Replying to @nabeards 🎙

Yet another reason to not over-rely on a few big tech companies for the majority of the planet's communication. Forget concerns about competition, monopolies and so on for now (as important as they are), what we want are many social networks, video conferencing apps, messenger apps. Every country should strive to build their own Google or FB, or certainly many more should. State-backed if needed. It's a question of resilience and security as much as anything.

Reply


@ExtraE 9 months

Replying to @nabeards 🎙

Any way to read this without an account?

Reply


@koksik202 9 months

Replying to @nabeards 🎙

my home connection with ISP is down Vodafone Ireland, so I guess they have such a big churn in Vodafone from FB BGP routes that it blew Vodafone network. Is it DNS or routing issue?

Reply


@ctur 9 months

Replying to @nabeards 🎙

It's back as of approx 14:47 PST.

Reply


@pytlicek 9 months

Replying to @nabeards 🎙

Who else sees their deleted messages on WhatsApp that shouldn't be there?

https://twitter.com/Pytlicek/status/1445072626729242637

Reply


@74d-fe6-2c6 9 months

Replying to @nabeards 🎙

I'm pretty sure this has been building up since the morning (Germany). I've had odd connectivity problems to a number of sites including slack for a moment.

Reply


@muthdra 9 months

Replying to @nabeards 🎙

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow.

Reply


@jangrahul 9 months

Replying to @nabeards 🎙

makes me think, why dont porn sites ever go down ?

Reply


@poorjohnmacafee 9 months

Replying to @nabeards 🎙

Teen depression and suicide rates plummeting right now

Reply


@drcongo 9 months

Replying to @nabeards 🎙

Let's hope it's permanent.

Reply


@HellDunkel 9 months

Replying to @nabeards 🎙

I wish it would stay that way.

Reply


@sabujp 9 months

Replying to @nabeards 🎙

India runs on whatsapp. They'll have more backups now.

Reply


@firstSpeaker 9 months

Replying to @nabeards 🎙

Is there any place to see how the overall internet bandwidth usage has changed during this outage?

Reply


@amir-h 9 months

Replying to @nabeards 🎙

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?

Reply


@zanethomas 9 months

Replying to @nabeards 🎙

hahaha, good riddance

Reply


@XiS 9 months

Replying to @nabeards 🎙



@josalhor 9 months

Replying to @nabeards 🎙

facebook.com resolves again!

> ping facebook.com

PING facebook.com (31.13.83.36) 56(84) bytes of data.

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=1 ttl=54 time=12.2 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=2 ttl=54 time=12.1 ms

64 bytes from edge-star-mini-shv-01-mad1.facebook.com (31.13.83.36): icmp_seq=3 ttl=54 time=11.7 ms

Can't yet traceroute to a.ns.facebook.com tho

Reply


@Animats 9 months

Replying to @nabeards 🎙

DNS configuration is becoming a single point of failure. A few weeks ago, many services running out of AWS West 2 failed because the within-the-datacenter DNS system broke down somehow.

Reply


@muthdra 9 months

Replying to @nabeards 🎙

Lichess Android app is also down but not the webpage. Infinity app for Reddit is down. HN is super slow and "having trouble serving requests".

Reply


@v4ult 9 months

Replying to @nabeards 🎙

Signal FTW

Reply


@rootusrootus 9 months

Replying to @nabeards 🎙

And a lot of people seem to be coming to HN to find out why, judging by how laggy HN is getting right now...

Reply


@hotz 9 months

Replying to @nabeards 🎙

The joy that people are getting from this is quite shitty. I hate social media but there are people earning a living working for these companies. Like others have pointed out, businesses and neighborhood watches rely on tech like this. At some point we've all had sites/apps go down, in a situation like that the last thing you want is people enjoying it. The lack of empathy in this thread is telling.

Reply


@peanut_worm 9 months

Replying to @nabeards 🎙

I am sure it is just a symptom of the Facebook outage, but it seems like every website I am going on is slower than usual today.

Reply


@WallyFunk 9 months

Replying to @nabeards 🎙

This would be a golden opportunity to launch your 'Facebook Killer' app. Preferably a social network where people don't pay with their data, but with, you know, a thing called Money.

Reply


@alkonaut 9 months

Replying to @nabeards 🎙

Here is a handy troubleshooting flowchart for megacorp outages:

> Is it a DNS issue? -> yes

It can be used in reverse as a postmortem too.

Reply


@Animats 9 months

Replying to @nabeards 🎙

Outage is top story on CNN and Fox. Facebook is not returning their calls. Sheera Frenkel at the New York Times has been able to get a little more info, but not much.

Now Twitter is starting to have problems with overload.

Reply


@slackfan 9 months

Replying to @nabeards 🎙

So how much do we need to pitch in to keep it down?

Reply


@swayson 9 months

Replying to @nabeards 🎙

This really is a fascinating case-study of what is truly resilient systems. More often than not, they are not centralized.

Reply


@marchingvehicle 9 months

Replying to @nabeards 🎙

Where could the physical data centers be that they need to access? How far away could it be?

Reply


@rstupek 9 months

Replying to @nabeards 🎙

Looks like someone found the light switch and turned everything back on!

Reply


@kossTKR 9 months

Replying to @nabeards 🎙

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue:

>As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.

User is providing live updates of the incident here:

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like...

Reply


@calebm 9 months

Replying to @nabeards 🎙

"When you are strong, appear weak."

Reply


@jurajmlich 9 months

Replying to @nabeards 🎙

It seems it has caused DNS servers crash for one of biggest Czechia's internet provider - Vodafone. Can be unrelated but I doubt it (https://twitter.com/BlazejKrajnak/status/1445063232486531099).

Think of it - half the country doesn't have internet because of this crash, that's terrifying. (Switching DNS servers obviously works but that's not something the general population will do)

Reply


@nabeards 9 months

Replying to @nabeards 🎙

Was receiving an error page, now just a server not responding error.

Reply


@amir-h 9 months

Replying to @nabeards 🎙

Hacker News also got so much slower, is it the load from people hoarding here after not being able to reach FB?

[I'm also getting server error trying to submit this comment]

Reply


@slackfan 9 months

Replying to @nabeards 🎙

So how much cash do we need to pitch in to keep it down?

Reply


@throwaway123x2 9 months

Replying to @nabeards 🎙

The cynic in me wonders if this is related to the Pandora Papers leak

Reply


@tacker2000 9 months

Replying to @nabeards 🎙



@cestith 9 months

Replying to @nabeards 🎙

Out of band management is an important feature for the reliability of your network.

Reply


@MrPatan 9 months

Replying to @nabeards 🎙

Oh no

Reply


@faramarz 9 months

Replying to @nabeards 🎙

What are some of the possible scnarios beyond the DNS issue suggested? (and might it be an attack?)

Reply


@Sophira 9 months

Replying to @nabeards 🎙

Somebody just had their very own "onosecond".

https://www.youtube.com/watch?v=X6NJkWbM1xk

The video is one that Tom Scott published in June 2020 about the worst typo he ever made in one of his prior jobs, and while the Facebook mistake is almost certainly not going to be anything irrecoverable like this one, you can bet that Facebook pride themselves on being available all the time.

Reply


@todd-davies 9 months

Replying to @nabeards 🎙

Perhaps allowing Facebook, WhatsApp and Instagram to merge was efficient after all - now that they have synchronized outages, people finally have a chance to get on with their lives, free of clickbait news and misinformation.

Reply


@randomperson_24 9 months

Replying to @nabeards 🎙

World productivity just grew by 10%

Reply


@38932ur98u 9 months

Replying to @nabeards 🎙

This event should be a good conversation starter on how horrifyingly monopolistic this trifecta of services has on worldwide communication. When I think through a random smattering of people in my contact book, I now have no way of contacting quite a few people at all. That's fucked. I wonder how many important messages, replies, etc will be screwed up due to this.

Reply


@GDC7 9 months

Replying to @nabeards 🎙

Maybe intentional?

Zuck trying to give an example of what a world without FB would look like, kinda saying to detractors what would happen if they had it their way.

Reply


@babuskov 9 months

Replying to @nabeards 🎙

Is HN hit by something as well? It's loading really slow for me.

Reply


@tinyprojects 9 months

Replying to @nabeards 🎙

Oculus is also down

Reply


@rkagerer 9 months

Replying to @nabeards 🎙

The internet felt a little more fresh and clean today.

Reply


@chungy 9 months

Replying to @nabeards 🎙

Even Facebook's Onion site isn't working: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t...

Fascinating simply that it's apparently not just a DNS issue.

Reply


@MrYellowP 9 months

Replying to @nabeards 🎙

I keep trying to submit to HN but I keep getting an error.

What's wrong with the internet?

FaceBook is down.

My friend from Slovenia is having trouble with discord. It eats his messages.

I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.

TrackMania players have talked about having input lag.

ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)

ycombinator turned out to be giving only errors.

Some sites I've found via google results seem to report that they are suffering from slow connections.

Do you have anything to add to this?

Reply


@dclaw 9 months

Replying to @nabeards 🎙

Delete Facebook / Instagram / WhatsApp when it comes back up. They are all trash.

Reply


@mupuff1234 9 months

Replying to @nabeards 🎙

Is there any site that tracks number of users for messaging apps? I'd be really curious to see if signal\telegram\etc are seeing a big bump.

Reply


@pwenzel 9 months

Replying to @nabeards 🎙

I noticed that some websites are loading slowly due to the third party script https://connect.facebook.net/en_US/fbevents.js timing out.

When uBlock Origin is running, this script gets blocked and pages return to feeling snappy.

Reply


@dizzant 9 months

Replying to @nabeards 🎙

Reading the thread, I'm surprised at the number of nearly identical "How much do we have to pay to keep it down? xD" posts I'm seeing, often from throwaway accounts. Some accounts with multiple near-identical posts within the same minute.

Could this be a coordinated smear in HN comments?

Reply


@mnd999 9 months

Replying to @nabeards 🎙

Somebody moved fast.

Reply


@spicybright 9 months

Replying to @nabeards 🎙

Still amazes me their infra team is supposedly the best in the world, and compensated as such, yet things like this happen.

Personally I'm glad FB went down for a few hours, but it's hard to imagine how that would happen in the first place.

Reply


@jangrahul 9 months

Replying to @nabeards 🎙

makes me wonder, why dont porn sites ever seem to go down ?

Reply


@ProjectArcturis 9 months

Replying to @nabeards 🎙

Have they tried turning it off and then turning it back on again?

Reply


@nurhdmsx 9 months

Replying to @nabeards 🎙

Its always a DNS problem

Reply


@TremendousJudge 9 months

Replying to @nabeards 🎙

Whataspp too. This seems pretty big

Reply


@yholio 9 months

Replying to @nabeards 🎙

I had problems with my internet connection and loaded my ISPs site. Strangely, my bill was paid. Even stranger, some sites load while others do not.

Then it hit me: I am so dependent on Facebook owned properties (Whatsapp, Facebook, insta) that a Facebook failure looks to me like an internet failure.

Reply


@neb_b 9 months

Replying to @nabeards 🎙

Also instagram and messenger

Reply


@mindcrime 9 months

Replying to @nabeards 🎙

Okay, let me tell you the difference between Facebook and everyone else, we don't crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire userbase. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don't you get that?

Reply


@MrYellowP 9 months

Replying to @nabeards 🎙

I keep trying to submit to HN but I keep getting an error.

What's wrong with the internet?

FaceBook is down.

My friend from Slovenia is having trouble with discord. It eats his messages.

I can't load photos from my friend in telegram and the messages take a relatively long time - multiple seconds! - to get received.

TrackMania players have talked about having input lag.

ycombinator is really slow and reports an error after submitting. "We're having some trouble serving your request. Sorry!" (lost count of the times i've tried submitting this)

ycombinator turned out to be giving only errors, but now seems to be working occasionally. I can not submit anything, though.

Some sites I've found via google results seem to report that they are suffering from slow connections.

Do you have anything to add to this?

Reply


@lazlee 9 months

Replying to @nabeards 🎙

Hopefully forever.

Reply


@pytlicek 9 months

Replying to @nabeards 🎙

Who else sees their deleted messages on WhatsApp that shouldn't be there?

https://news.ycombinator.com/item?id=28749652

Reply


@dnndev 9 months

Replying to @nabeards 🎙

I am soooo happy I did not even notice! Don’t use any of these apps.

Reply


@infologics 9 months

Replying to @nabeards 🎙



@mrkickling 9 months

Replying to @nabeards 🎙

When I do dig instagram.com I get an A response for this IP: 31.13.65.174 or similar addresses, which leads to an empty page.

Reply


@Paianni 9 months

Replying to @nabeards 🎙

The level of intelligence of the comments in this thread kinda confirm my suspicions that the armchair experts from Reddit et al (including myself) are discovering this site.

Reply


@Animats 9 months

Replying to @nabeards 🎙

Facebook outage is now the top story on CNN and Fox. Facebook stock down 5%. Facebook is not returning calls from Fox, or CNN.

Reply


@zip1234 9 months

Replying to @nabeards 🎙

DNS?

Reply


@orthecreedence 9 months

Replying to @nabeards 🎙

Good riddance.

Reply


@walrus01 9 months

Replying to @nabeards 🎙

downdetector looks like a real mess for it.

I'm going to parrot the other comment here and say nothing of value was lost.

https://downdetector.com/status/facebook/

Reply


@tannhaeuser 9 months

Replying to @nabeards 🎙

Expecting to get messages on WhatsApp alternatives tonight ...

Reply


@nycdatasci 9 months

Replying to @nabeards 🎙

Ignore.

Reply


@stonks 9 months

Replying to @nabeards 🎙

Facebook and Messenger are working now.

Instagram and WhatsApp - not yet.

Reply


@RedShift1 9 months

Replying to @nabeards 🎙

Nothing of value was lost

Reply


@lucidbee 9 months

Replying to @nabeards 🎙

The timing of this is so rich in irony I can't help but wonder if there is an element of internal sabotage. How many FB employees hate FB right now? The latest expose of FB is both effective and truly awful. I can't imagine feeling good about a FB job. And it's gotten worse! Now they look like they can't even keep their websites up.

Reply


@someonehere 9 months

Replying to @nabeards 🎙

When I worked there they were all about open source projects to build it themselves and control the service. Well, when your whole company is run on one DNS service this is going to bite you in the butt.

I only know of a handful of Saas apps they didn’t build internally. Sadly none of those will help them get out of this situation.

Reply


@scumcity 9 months

Replying to @nabeards 🎙

hrm, bgp and dns. It's weird when decades old technology somehow fails like this. The main reason distributed systems is hard is because of the time component. Whenever you add timeouts to an algorithm, everything becomes orders of magnitude more difficult to reason about, as the number of states grows without bound. In any case, this is an epic outage and sad.

Reply


@ionwake 9 months

Replying to @nabeards 🎙

Did I just read that the Facebook IRC fallback went down too?!? I was about to say what’s wrong with freenode ( but yeah on 2nd thoughts let’s not talk about freenode )

Reply


@baalimago 9 months

Replying to @nabeards 🎙

Aren't there places around poorer countries where Facebook is basically an ISP? What about them? They have literally 0 info.

https://tcrn.ch/3kOHco1

Reply


@baby 9 months

Replying to @nabeards 🎙

Wow. I can’t remember the last time whatsapp was done. I pretty much use messenger/instagram/whatsapp to talk to most of my friends and family. I’m happy that I do use other platforms otherwise I would be completely cut off from my parents right now.

Reply


@LuisMondragon 9 months

Replying to @nabeards 🎙

Facebook employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.

https://twitter.com/sheeraf/status/1445099150316503057

Reply


@nwatson 9 months

Replying to @nabeards 🎙

I guess the "prophets" at Victory Channel / Flashpoint called down Holy Fire on the Facebook infrastructure in retribution ... https://youtu.be/FbSkFuvqFdA?t=1127 . (I'm an Evangelical Christian but those folks are nuts ... Mario Murillo, Lance Wallnau, Hank Kunneman, Gene Bailey, etc.)

Reply


@midnightdiesel 9 months

Replying to @nabeards 🎙

Hopefully it never comes back up!

Reply


@freediver 9 months

Replying to @nabeards 🎙

Terrible day for many people. Both working for Facebook and those depending on their services.

Reply


@jacke 9 months

Replying to @nabeards 🎙

They made they own BGP tools and looks like it failed https://www.youtube.com/watch?v=wHfYUbKNEyc

Reply


@taftster 9 months

Replying to @nabeards 🎙

What I think is interesting is the effects of this type of thing across peripheral news sites, like HN. I wonder how much spike HN gets with people rushing here to find out what's going on and to read the (articulate) related discussions.

Reply


@johnbaker92 9 months

Replying to @nabeards 🎙

Let this be permanent - not a huge loss for humanity.

Reply


@fartingflamingo 9 months

Replying to @nabeards 🎙

The whistleblower said she wanted to fix Facebook.

Mission accomplished, I'd say. For now at least.

Reply


@polynomial 9 months

Replying to @nabeards 🎙

"It's always DNS."

Reply


@baalimago 9 months

Replying to @nabeards 🎙

Aren't there places around poorer countries where Facebook is basically an ISP? What about them?

https://tcrn.ch/3kOHco1 2Africa cable, as an example

Reply


@markus_zhang 9 months

Replying to @nabeards 🎙

Just a sidenote I think outlook is down as well.

Reply


@rvz 9 months

Replying to @nabeards 🎙

From [0]

> ...there is no limit to the scandals, leaks, whistleblowers, lawsuits or penalties that will bring the Facebook mafia down.

Fine. 'Literally' bringing the Facebook mafia down like that would do.

But only for now.

[0] https://news.ycombinator.com/item?id=28742179

Reply


@lewich 9 months

Replying to @nabeards 🎙

Hopefully permanently.

Reply


@jose-cl 9 months

Replying to @nabeards 🎙

In this context, I remember youtube+pakistan issue[1]. I also wonder how an AS/BGP manager do his/her job... I imagine a guy/girl changing a text file in a old console. Anyone knows?

[1] https://www.infoworld.com/article/2648947/youtube-outage-und...

Reply


@Bwild 9 months

Replying to @nabeards 🎙

They must have shut it off and turned it back on

Reply


@shkkmo 9 months

Replying to @nabeards 🎙

I keep getting non-dns errors from Hackner News as well. There appears to be some sort of broader incident happening?

It's not just lag, I keep getting the "We're having some trouble serving your request. Sorry!" page.

Edit: HN related thread https://news.ycombinator.com/item?id=28749476

Reply


@jbverschoor 9 months

Replying to @nabeards 🎙

It's funny that hackernews is now overloaded with distracted people ;-)

Reply


@milankragujevic 9 months

Replying to @nabeards 🎙

Also Speedtest.net for me is showing a 503 error page. Seems a large CDN might be having problems. Their status page shows all green. FB and their other sites are also down.

edit: I see it's back up and I've been getting downvoted, here's a screenshot of the error for clarity

https://i.imgur.com/wvhOwwL.png

Reply


@thedudeabides5 9 months

Replying to @nabeards 🎙

Checked isitdownrightnow.com and said Netflix was also down. Any chance these are related?

Reply


@synaesthesisx 9 months

Replying to @nabeards 🎙

It's quite a coincidence for this to coincide with the whistleblower report + rumors of Peter Thiel (perhaps via Palantir?) involved in leveraging FB for the 2022 midterm elections.

I'm not suggesting that this is the case, but a failure of this scale (with internal systems also down) could allow scrubbing of evidence without leaving traces.

Reply


@wejick 9 months

Replying to @nabeards 🎙

Would be very interesting if they release the RCA to the public

Reply


@cheese_van 9 months

Replying to @nabeards 🎙

Sir, I don't care who you are, you must open a ticket.

Reply


@nabeards 9 months

Replying to @nabeards 🎙

Seems to be affecting all Facebook properties.

Reply


@keithnoizu 9 months

Replying to @nabeards 🎙

some poor engineer is sobbing over a split brain mnesia cluster right now praying to get the thing back up.

Reply


About Us

site design / logo © 2022 Box Piper