Hackernews is my new status page. All others are useless.Reply
So interesting, the actual edge servers work.
But the Key/Value store that all cloudflare's configuration data lives on is giving 500 errorsReply
Still down, here.Reply
Hah. I was seeing a lot of 500 errors across the web and came to HN to see what was up.Reply
I guess this is probably why I can't log into league of legends atm.Reply
Peep, anyone there?
Wow, for me it looked like the world had gone mad. This is a reminder to not only rely on 22.214.171.124 for DNS resolution in PiHole.
I host most of my services locally, but ironically could not connect to my own homelab. I use a dedicated domain with DynDNS and did not configure the network and DNS without reliance on external DNS. Surely it's infinitely more likely for me to make a mistake, right?Reply
Back up now as of about 5 min ago.Reply
I really dislike that they are editing their status messages.
Entry dated "Jun 21, 2022 - 06:43 UTC" has been edited to include more detail after they posted another entry at 06:57 UTC. There seems to be no indication that the message has been altered.
Currently text on the status page may suggest that they identified the problem immediately but it took about 15 minutes. Previously there was a text stating that customers should expect update within 15 minutes. Next message was posted 14 minutes after that but previous message was altered later and nothing indicates this.
Cloudflare, not cool.Reply
Up on my phone (AT&T) but down on my main ISP/desktop (AT&T lightspeed). BGP issues too, maybe?Reply
At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.Reply
DNS API failing with 500Reply
That explains why i couldn't access phoronixReply
Sooner or later they will start to give a d..n about resiliency :)Reply
This aligned with me debugging a separate issue in a program that programmatically uses the npm registry... that wasn't fun.Reply
I look forward to a technical report on this outage on their blog.Reply
Cloudflare is great everyone, let's not forget how awesome they are.Reply
Can someone link me to some information that explains what Cloudflare is besides being a CDN?
Like I understand how websites can be served using a CDN and how a lot of the internet depends on that... but I don't see how gaming services like Valorant or cloud providers like AWS or chat room like Discord depend on Cloudflare.
I'm placing my bets on a config file with emoji charactersReply
Hello 126.96.36.199 my old friend.Reply
Seems to be back up again, albeit a bit slow.Reply
Again, why did people decide to centralized like 80% or something of the internet under a single company?Reply
Uh, that’s not good. The negatives of centralisation really smack you in the face.
I’ll start moving my sites away from Cloudflare soon. Not because it’s bad — in fact it has been amazing, but rather to decentralise.Reply
wow so that's why all of the "is it down for me" sites are down lolReply
I wasted a bunch of time debugging the HTTP 500 errors on my site before I realized everything is 100% OK on my end, and that it's Cloudflare returning the error not my servers.Reply
Looks like it's mostly good now?Reply
If only we could come up with a globally distributed set of networks and systems that could be run by millions of entities that don't rely on each other to keep working. Oh no wait...Reply
Our monitoring systems had influxdb endpoints behind cloudflare. Now we not only lost users but also access to data about the impact of outage.Reply
Wishing Cloudflare ops teams the best to recover fast from this outage. Meanwhile, we urge customers to check out www.cdnreserve.com , and implement a sound CDN backup strategy (auto-failover) when the primary CDN suffers an outage.Reply
- Encrypted DNS seems to be having issues (very slow resolution, if any)
- Having issues connecting to GitHub (Could be they are using CF, or could be DNS issue - but I'm able to connect fine to Google services)
- Twitter loads, but all images fail to resolve
- https://www.cloudflarestatus.com/ loads very slowly, and no assets (CSS, images, etc) load
EDIT from CF :: The issue has been identified and a fix is being implemented. Posted Jun 21, 2022 - 06:57 UTCReply
Had a hard time trusting Cloudflare since they blocked 8chan due to political activism.Reply
maybe we should start suggesting the use of IPFS/Arweave to host critical infra status pages or similar sites that are occasionally imminently neededReply
Wow lots of websites are affected, including Medium. The perils of centralization strike again. Though ironically, I noticed that the IPFS website uses cloudflare as well. The actual IPFS network is working just fine though, and I'm not aware of IPFS ever having any global outages. Though then again, I'm not aware of any on bittorrent eitherReply
Discord is back, Kickstarter progressed from 500 to an exotic "Error 1016 Origin DNS error".
EDIT: all flushed, Kickstarter works.Reply
jgrahamc, just some feedback about trying to reach support:
1. I could see my site down, including cloudflare.com with nginx 500 errors, via Sydney AU
2. Logged in to dashboard (via Melbourne AU) that worked; and so was thinking it was an issue with Sydney Cloudflare My experience with Cloudflare has been in the past sometimes servers in some regions have issues and its a transient thing.
3. Status page showed no problems, so I went to "Contact support" and went around in circles (really frustrating) via the "Contact support" link moving me between Community forums, Support ticket, etc. I then see Chat is an option is available with a Business plan, so I upgrade to that, hoping for some real-time support to alert of the Sydney issue.
4. Return to the "Contact support" page after upgrading the plan, but the Chat option still not present on the support screen (and help articles say to return to support page and click "Chat" but it never shows up).
5. Come across https://community.cloudflare.com/t/cloudflare-for-teams-chat... searching for why I can't see Chat as an option on the support forum saying they're on paid plans with no chat support and its not showing up, so I just give up assuming its broken
6. Open HackerNews and see its at the top. A few moments later the status page reflects the outage.
I still can't see the Chat option so I've down-graded the plan again.Reply
LMAO of course when every single thing I tried to use won't load or gives me a useless default 500 nginx error page, I find out why here. Figured it was CloudFlare. Single point of failure, not once.Reply
While this sucks, at least we'll have a great writeup to look forward toReply
I had tinkered with my network settings just before this to troubleshoot an entirely unrelated problem so for a minute there I thought I broke everything lolReply
I think unpkg were affected by this too.Reply
GitLab.com is impacted too as they are also behind Cloudflare.Reply
The entire day yesterday performance with Cloudflare was extremely sluggish. Pages which relied on it, even if it's only loading a JS-file from the CDN, would hang for tens of seconds.Reply
Everything working here in AustraliaReply
Another reminder that centralized services like Cloudflare are a bad idea.Reply
It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal).
This is also important for countries with limited connectivity to the Internet, if the PoP in that country looses it's connection back to CF it shuts everything down, so even if the origin is in the next rack over from the PoP, it's un-reachable.Reply
Cloudflare has too many outages.
Their core service (DNS and web proxying) should see an outage once every 10 years or less. Much like Google Search (which is a far more complex service).
Yet it seems we get an outage more frequently than once a year. In my opinion, that makes the service too unreliable to base my business off - it's not like I can failover to another provider while they're down.Reply
My morning started with crunching logs and not finding any errors and slowly panicking.
But we'll, can happen :)Reply
I am guessing this is why substack is down?Reply
Sites are coming back up on my end (NL)Reply
turns out having a central failure point for the entire web was a bad ideaReply
Can confirm, no issues on our infra side. Cloudflare took down the web once again.Reply
This is the official Cloudflare incident URL:Reply
I'm having '500 Internal Server Error' (nginx) on a regular tab, and everything working fine on an icognito tab. Go figure.Reply
This was very educational, all of a sudden I couldn't reach 60% of all websites I normally visit everyday. I guess this is the cost of laziness under the guise of DDOS protection.Reply
I was setting up some DNS for a site, when it suddenly stopped working after 30mins of missing with the settings and googling I gave up come in here, and see this.
My sites that are just using DNS are working fine, it's only those with the orange cloud, proxy turned on that are broken.Reply
Was just about to post. Many sites returning 500 in the UK, and cloudflare seems to be the point of failure, including itself.Reply
I cannot access science.org, quora.com, substack.com at the moment. It shows 500 Internal Server Error. Didn't know why but now it is clear. I guess I just wait until it is fixed.Reply
looks like lot of sites being impacted by this oneReply
Kind of ironic, there was a big "cloudflare is bad and a central point of failure" article on the front page just a couple days ago.
edit: Not that I necessarily agree with the article even in light of there being an outage, cloudflare has been pretty good for us. Just thought it was interesting.Reply
yeah, and if cloudflare could make their anti-bot "verification" interoperable with noscript/basic (x)html browsers, and not force those grotesquely and absurdely massive google (blink/geeko) and apple(webkit) web engines, that would be less criminal.Reply
Yep, same issue. All our services are down, this is very bad. Can't even point directly to the app servers to resolve.Reply
Ironically isitdownrightnow.com is also downReply
Very bad start of the day :(
Cloudflare.com now up, and websites are coming up, argo tunnels still downReply
Same here! Seems to work with FR proxy tho!Reply
Sites returning 500 is one thing, people will understand that's an error. Site can't be found because DNS is out is not one that the generic public will start to debug, but instead they'll walk away from the site, sometimes for good.
Question: how could be (temporary) DNS errors be made nicer?Reply
All sites of mine are down now.Reply
Shouldn't have happened in the first place. Should have had something that worked on their own website to indicate the service is down, not needing to come to a somewhat obscure tech forum to find out the details.Reply
It's times like these when I'm appreciative of the simplicity of the HN tech stack. Was talking to some people on discord when it went down and then noticed some other websites were down. Came right to HN to see 5 different threads about this. Will be curious to see what the cause of the issue turns out to beReply
It's working in Belgium but not in the NetherlandsReply
Several sites I was trying to access all went down at the same time. Came to Hacker News to see what was up - not disappointed!
*Including America's Cardroom, perhaps the biggest "offshore" US poker site. I can promise you that there are a lot of people who were playing in tournaments that are very unhappy right now. New York here.Reply
Cloudflare Warp connects, then prevents anything else from loading. Thought it was my routerReply
hugops to cf team. <3Reply
Explains why the online training course I was part way through stopped working! Amusing that the quickest diagnosis came from skimming the headlines here :)Reply
All shopify based stores are downReply
This should hopefully drive home the idea of why HN shouldn't be cheering on Cloudflare's slow takeover of the internet.Reply
That indicates how much of the Internet depends on cloudflare server.Reply
DigitalOcean is down as well.Reply
Linode seems unreachable too (and related hosted VPS). Is it my problem or is it a general failure on Linode?Reply
DNS looks to be ok, but sites that I have proxied through them aren't working.Reply
I'm unable to log into Bitwarden Safari extension. That's an alarming detail... Mobile app still works, fortunatelyReply
The issue has been identified and a fix is being implemented.
Posted 5 minutes ago. Jun 21, 2022 - 06:57 UTCReply
Things like Victorops/Splunk On-call are also down because of this.
This means if your alerts are fired through them, you'll peacefully be sleeping through this incident unless your customers wake you up.Reply
Hmm, I am able to access Cloudflare's own website but sites that are proxied through them give me Nginx's default 500 error page.Reply
Everything is up for me. Short downtime?Reply
Single Point of Failure. LOL.Reply
All things work here? Using cf dns, but even cf.com loads.
Edit: in belgiumReply
is it just DNS?Reply
CF status page now showing a wide spread incident.Reply
The worst part is cloud infrastructure companies like DigitalOcean and Linode are both down simply because for some reason they can't build their own infrastructure to not rely on Cloudflare lol.Reply
Is this why 4chan is down?Reply
I believe discord was also affected due to this.. However, I did get messages from my friend in ThailandReply
"Investigating - Cloudflare is investigating wide-spread issues with our services and/or network.
Users may experience errors or timeouts reaching Cloudflare’s network or services.
We will update this status page to clarify the scope of impact as we continue the investigation. The next update should be expected within 15 minutes."Reply
Everything using Cloudflare is down. Tells you that how much of the internet Cloudflare is part of.
Once in a while web2 is going great with Cloudflare. Until when everyone uses it and it goes down.Reply
Am I wrong to use this as an excuse to not use Cloudflare?Reply
The Cape Town location does not seem to be impacted in any way. Everything works as expected.Reply
This is such a weird outage, cloudflare sites are down on some of my devices but not others.Reply
Oh... this explains why discord is down.Reply
Today a mistake, tomorrow an order from the DOJ. Take heed, Internet.Reply
linear.app -> down notion.so -> down
Wow, this dependency on cloudflare is wide.Reply
Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks.
Should be back up everywhere.Reply
Getting intermittent SERVFAIL from 188.8.131.52 DNS as wellReply
I haven't been impacted in Australia at all, but all of my (probably US based) monitors are going off.Reply
Statuspage seems to be useless, I was just trying to get the status via multiple networks and my mobile network. Ironically other downdetector services are also down.Reply
Seems to be a lot of stuff flapping right now. I was able to load a client site that was behind Cloudflare, and now not.
cloudflare.com was returning Connection Refused, then error 522 cloudflarestatus.com was returning Connection Refused, now can't even resolve the IP
My guess would be that a router misconfiguration is being progressively deployed throughout their infrastructure.
EDIT: Continues to look like a cascading failure across their network. 184.108.40.206 is now unreachable for me.Reply
Auth0 login is also not working. Their website is up thoughReply
Slack seems to be having trouble sending messages as well. Was trying to let my team know i've acked the request and am unable to do so.Reply
Down here in Japan, too. Thought it was my connection for a bit. sighReply
Not having fun right now, but this brings home just how reliant so much of the internet is on a very few very big service providers.Reply
Haha... I got pinged on my phone a site I manage is down, trying to figure out what's wrong with it, noticing other sites down, realizing it's CloudflareReply
They're not down worldwide, we're still seeing traffic from some POPs, but it looks like a majority of their POPs are dead.
This feels like a bad config push.Reply
Just noticed this morning.. IRCCloud and Discord are affected by this too. Wonder what else.Reply