Cloudflare had a partial outage

Hackernews is my new status page. All others are useless.

So interesting, the actual edge servers work.

But the Key/Value store that all cloudflare's configuration data lives on is giving 500 errors

Still down, here.

Hah. I was seeing a lot of 500 errors across the web and came to HN to see what was up.

I guess this is probably why I can't log into league of legends atm.

Peep, anyone there?

Wow, for me it looked like the world had gone mad. This is a reminder to not only rely on 1.1.1.1 for DNS resolution in PiHole.

I host most of my services locally, but ironically could not connect to my own homelab. I use a dedicated domain with DynDNS and did not configure the network and DNS without reliance on external DNS. Surely it's infinitely more likely for me to make a mistake, right?

Back up now as of about 5 min ago.

I really dislike that they are editing their status messages.

Entry[1] dated "Jun 21, 2022 - 06:43 UTC" has been edited to include more detail after they posted another entry at 06:57 UTC. There seems to be no indication that the message has been altered.

Currently text on the status page may suggest that they identified the problem immediately but it took about 15 minutes. Previously there was a text stating that customers should expect update within 15 minutes. Next message was posted 14 minutes after that but previous message was altered later and nothing indicates this.

Cloudflare, not cool.

[1] https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

Up on my phone (AT&T) but down on my main ISP/desktop (AT&T lightspeed). BGP issues too, maybe?

At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.

DNS API failing with 500

That explains why i couldn't access phoronix

Sooner or later they will start to give a d..n about resiliency :)

This aligned with me debugging a separate issue in a program that programmatically uses the npm registry... that wasn't fun.

I look forward to a technical report on this outage on their blog.

Cloudflare is great everyone, let's not forget how awesome they are.

Can someone link me to some information that explains what Cloudflare is besides being a CDN?

Like I understand how websites can be served using a CDN and how a lot of the internet depends on that... but I don't see how gaming services like Valorant or cloud providers like AWS or chat room like Discord depend on Cloudflare.

Thanks!

I'm placing my bets on a config file with emoji characters

cloudflare has a 100% SLA. This needs revision https://www.cloudflare.com/business-sla/

Hello 8.8.8.8 my old friend.

Seems to be back up again, albeit a bit slow.

Again, why did people decide to centralized like 80% or something of the internet under a single company?

Uh, that’s not good. The negatives of centralisation really smack you in the face.

I’ll start moving my sites away from Cloudflare soon. Not because it’s bad — in fact it has been amazing, but rather to decentralise.

wow so that's why all of the "is it down for me" sites are down lol

I wasted a bunch of time debugging the HTTP 500 errors on my site before I realized everything is 100% OK on my end, and that it's Cloudflare returning the error not my servers.

Looks like it's mostly good now?

If only we could come up with a globally distributed set of networks and systems that could be run by millions of entities that don't rely on each other to keep working. Oh no wait...

Our monitoring systems had influxdb endpoints behind cloudflare. Now we not only lost users but also access to data about the impact of outage.

Wishing Cloudflare ops teams the best to recover fast from this outage. Meanwhile, we urge customers to check out www.cdnreserve.com , and implement a sound CDN backup strategy (auto-failover) when the primary CDN suffers an outage.

- Encrypted DNS seems to be having issues (very slow resolution, if any)

- Having issues connecting to GitHub (Could be they are using CF, or could be DNS issue - but I'm able to connect fine to Google services)

- Twitter loads, but all images fail to resolve

- https://www.cloudflarestatus.com/ loads very slowly, and no assets (CSS, images, etc) load

EDIT from CF :: The issue has been identified and a fix is being implemented. Posted Jun 21, 2022 - 06:57 UTC

Had a hard time trusting Cloudflare since they blocked 8chan due to political activism.

maybe we should start suggesting the use of IPFS/Arweave to host critical infra status pages or similar sites that are occasionally imminently needed

Wow lots of websites are affected, including Medium. The perils of centralization strike again. Though ironically, I noticed that the IPFS website uses cloudflare as well. The actual IPFS network is working just fine though, and I'm not aware of IPFS ever having any global outages. Though then again, I'm not aware of any on bittorrent either

Discord is back, Kickstarter progressed from 500 to an exotic "Error 1016 Origin DNS error".

EDIT: all flushed, Kickstarter works.

jgrahamc, just some feedback about trying to reach support:

1. I could see my site down, including cloudflare.com with nginx 500 errors, via Sydney AU

2. Logged in to dashboard (via Melbourne AU) that worked; and so was thinking it was an issue with Sydney Cloudflare My experience with Cloudflare has been in the past sometimes servers in some regions have issues and its a transient thing.

3. Status page showed no problems, so I went to "Contact support" and went around in circles (really frustrating) via the "Contact support" link moving me between Community forums, Support ticket, etc. I then see Chat is an option is available with a Business plan, so I upgrade to that, hoping for some real-time support to alert of the Sydney issue.

4. Return to the "Contact support" page after upgrading the plan, but the Chat option still not present on the support screen (and help articles say to return to support page and click "Chat" but it never shows up).

5. Come across https://community.cloudflare.com/t/cloudflare-for-teams-chat... searching for why I can't see Chat as an option on the support forum saying they're on paid plans with no chat support and its not showing up, so I just give up assuming its broken

6. Open HackerNews and see its at the top. A few moments later the status page reflects the outage.

I still can't see the Chat option so I've down-graded the plan again.

LMAO of course when every single thing I tried to use won't load or gives me a useless default 500 nginx error page, I find out why here. Figured it was CloudFlare. Single point of failure, not once.

While this sucks, at least we'll have a great writeup to look forward to

Back up?

I had tinkered with my network settings just before this to troubleshoot an entirely unrelated problem so for a minute there I thought I broke everything lol

I think unpkg were affected by this too.

GitLab.com is impacted too as they are also behind Cloudflare.

The entire day yesterday performance with Cloudflare was extremely sluggish. Pages which relied on it, even if it's only loading a JS-file from the CDN, would hang for tens of seconds.

Everything working here in Australia

Another reminder that centralized services like Cloudflare are a bad idea.

ah fuck

It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal).

This is also important for countries with limited connectivity to the Internet, if the PoP in that country looses it's connection back to CF it shuts everything down, so even if the origin is in the next rack over from the PoP, it's un-reachable.

Cloudflare has too many outages.

Their core service (DNS and web proxying) should see an outage once every 10 years or less. Much like Google Search (which is a far more complex service).

Yet it seems we get an outage more frequently than once a year. In my opinion, that makes the service too unreliable to base my business off - it's not like I can failover to another provider while they're down.

My morning started with crunching logs and not finding any errors and slowly panicking.

But we'll, can happen :)

I am guessing this is why substack is down?

Sites are coming back up on my end (NL)

turns out having a central failure point for the entire web was a bad idea

Can confirm, no issues on our infra side. Cloudflare took down the web once again.

This is the official Cloudflare incident URL:

https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

I'm having '500 Internal Server Error' (nginx) on a regular tab, and everything working fine on an icognito tab. Go figure.

This was very educational, all of a sudden I couldn't reach 60% of all websites I normally visit everyday. I guess this is the cost of laziness under the guise of DDOS protection.

I was setting up some DNS for a site, when it suddenly stopped working after 30mins of missing with the settings and googling I gave up come in here, and see this.

My sites that are just using DNS are working fine, it's only those with the orange cloud, proxy turned on that are broken.

Was just about to post. Many sites returning 500 in the UK, and cloudflare seems to be the point of failure, including itself.

I cannot access science.org, quora.com, substack.com at the moment. It shows 500 Internal Server Error. Didn't know why but now it is clear. I guess I just wait until it is fixed.

looks like lot of sites being impacted by this one

Kind of ironic, there was a big "cloudflare is bad and a central point of failure" article on the front page just a couple days ago.

Found it, https://news.ycombinator.com/item?id=31801947

edit: Not that I necessarily agree with the article even in light of there being an outage, cloudflare has been pretty good for us. Just thought it was interesting.

yeah, and if cloudflare could make their anti-bot "verification" interoperable with noscript/basic (x)html browsers, and not force those grotesquely and absurdely massive google (blink/geeko) and apple(webkit) web engines, that would be less criminal.

Yep, same issue. All our services are down, this is very bad. Can't even point directly to the app servers to resolve.

Ironically isitdownrightnow.com is also down

CF's website is down as well. The CF Status page [0] says everything is working, though.

[0] https://www.cloudflarestatus.com/

Very bad start of the day :(

Cloudflare.com now up, and websites are coming up, argo tunnels still down

Same here! Seems to work with FR proxy tho!

Sites returning 500 is one thing, people will understand that's an error. Site can't be found because DNS is out is not one that the generic public will start to debug, but instead they'll walk away from the site, sometimes for good.

Question: how could be (temporary) DNS errors be made nicer?

https://www.cloudflarestatus.com still shows „All Systems Operational“

All sites of mine are down now.

Shouldn't have happened in the first place. Should have had something that worked on their own website to indicate the service is down, not needing to come to a somewhat obscure tech forum to find out the details.

It's times like these when I'm appreciative of the simplicity of the HN tech stack. Was talking to some people on discord when it went down and then noticed some other websites were down. Came right to HN to see 5 different threads about this. Will be curious to see what the cause of the issue turns out to be

It's working in Belgium but not in the Netherlands

Several sites I was trying to access all went down at the same time. Came to Hacker News to see what was up - not disappointed!

*Including America's Cardroom, perhaps the biggest "offshore" US poker site. I can promise you that there are a lot of people who were playing in tournaments that are very unhappy right now. New York here.

https://hn.algolia.com also down now.

Cloudflare Warp connects, then prevents anything else from loading. Thought it was my router

hugops to cf team. <3

Explains why the online training course I was part way through stopped working! Amusing that the quickest diagnosis came from skimming the headlines here :)

All shopify based stores are down

This should hopefully drive home the idea of why HN shouldn't be cheering on Cloudflare's slow takeover of the internet.

That indicates how much of the Internet depends on cloudflare server.

DigitalOcean is down as well.

Linode seems unreachable too (and related hosted VPS). Is it my problem or is it a general failure on Linode?

DNS looks to be ok, but sites that I have proxied through them aren't working.

I'm unable to log into Bitwarden Safari extension. That's an alarming detail... Mobile app still works, fortunately

https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj:

Identified

The issue has been identified and a fix is being implemented.

Posted 5 minutes ago. Jun 21, 2022 - 06:57 UTC

Things like Victorops/Splunk On-call are also down because of this.

This means if your alerts are fired through them, you'll peacefully be sleeping through this incident unless your customers wake you up.

Hmm, I am able to access Cloudflare's own website but sites that are proxied through them give me Nginx's default 500 error page.

Everything is up for me. Short downtime?

Single Point of Failure. LOL.

All things work here? Using cf dns, but even cf.com loads.

Edit: in belgium

I would change the URL to be https://www.cloudflarestatus.com/ intead, the cloudflare.com domain looks to be hosted in a different way.

is it just DNS?

CF status page now showing a wide spread incident.

https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

The worst part is cloud infrastructure companies like DigitalOcean and Linode are both down simply because for some reason they can't build their own infrastructure to not rely on Cloudflare lol.

Is this why 4chan is down?

I believe discord was also affected due to this.. However, I did get messages from my friend in Thailand

My 2 cloudflare pages are working perfectly , https://mustafagroup.site/

https://emaanpainters.co.nz/

"Investigating - Cloudflare is investigating wide-spread issues with our services and/or network.

Users may experience errors or timeouts reaching Cloudflare’s network or services.

We will update this status page to clarify the scope of impact as we continue the investigation. The next update should be expected within 15 minutes."

Everything using Cloudflare is down. Tells you that how much of the internet Cloudflare is part of.

Once in a while web2 is going great with Cloudflare. Until when everyone uses it and it goes down.

Am I wrong to use this as an excuse to not use Cloudflare?

The Cape Town location does not seem to be impacted in any way. Everything works as expected.

This is such a weird outage, cloudflare sites are down on some of my devices but not others.

Oh... this explains why discord is down.

Today a mistake, tomorrow an order from the DOJ. Take heed, Internet.

linear.app -> down notion.so -> down

Wow, this dependency on cloudflare is wide.

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks.

Should be back up everywhere.

Getting intermittent SERVFAIL from 1.1.1.1 DNS as well

I haven't been impacted in Australia at all, but all of my (probably US based) monitors are going off.

Statuspage seems to be useless, I was just trying to get the status via multiple networks and my mobile network. Ironically other downdetector services are also down.

Seems to be a lot of stuff flapping right now. I was able to load a client site that was behind Cloudflare, and now not.

cloudflare.com was returning Connection Refused, then error 522 cloudflarestatus.com was returning Connection Refused, now can't even resolve the IP

My guess would be that a router misconfiguration is being progressively deployed throughout their infrastructure.

EDIT: Continues to look like a cascading failure across their network. 1.1.1.1 is now unreachable for me.

Auth0 login is also not working. Their website is up though

Slack seems to be having trouble sending messages as well. Was trying to let my team know i've acked the request and am unable to do so.

Down here in Japan, too. Thought it was my connection for a bit. sigh

Not having fun right now, but this brings home just how reliant so much of the internet is on a very few very big service providers.

Haha... I got pinged on my phone a site I manage is down, trying to figure out what's wrong with it, noticing other sites down, realizing it's Cloudflare

They're not down worldwide, we're still seeing traffic from some POPs, but it looks like a majority of their POPs are dead.

This feels like a bad config push.

Just noticed this morning.. IRCCloud and Discord are affected by this too. Wonder what else.