Hacker News Re-Imagined

Cloudflare Reverse Proxies Are Dumping Uninitialized Memory

  • 3238 points
  • β€’ 4 years ago

  • @tptacek
  • Created a post

Cloudflare Reverse Proxies Are Dumping Uninitialized Memory


@snek β€’ 4 years

Replying to @tptacek πŸŽ™

gg to project0

Reply


@stevenhubertron β€’ 4 years

Replying to @tptacek πŸŽ™

Well now I have a great response to the sales guy that bugs me everyday.

Reply


@glasz β€’ 4 years

Replying to @tptacek πŸŽ™

ppl have learned _nothing_. how much does it take to just STOP TO MITM YOUR OWN SERVICES?

Reply


@stephenr β€’ 4 years

Replying to @tptacek πŸŽ™

Who wants to bet this won't change a lot of developer's making statements like "I use <Insert HTTPS offering CDN> so my site is secure"

Reply


@wereHamster β€’ 4 years

Replying to @tptacek πŸŽ™

So.. when are we going to stop using unsafe languages which allows these kinds of memory corruption or leaks? If this is not reason enough, what else needs to happen before people realise that whatever language the cloudflare proxy is written in is a really bad one?

Reply


@Karupan β€’ 4 years

Replying to @tptacek πŸŽ™

This is huge and CF is certainly downplaying the issue. To be clear, I think the kind of tech that they deal with is extremely complex, which makes it ever harder to test or uncover them easily. And they have been reasonably good with disclosures (prior to this incident).

When I was evaluating CF for a small personal app, I really thought hard about using a public reverse proxy and decided that it wasn't worth it for the scale I was dealing with. No one can predict these security issues, but I sure am glad I didn't go with them!

Reply


@mrep β€’ 4 years

Replying to @tptacek πŸŽ™

What is the optimal balance between centralization and decentralization? Most people in this thread are complaining about how using a big centralized service (cloudfare) causes so much damage when security issues come up, and yet I have seen many people advocate using a single password manager (like 1password) to which this exact type of huge security problem can happen (your password manager is the single point of security failure which can comprimise all of your accounts!!!).

What is the optimal solution???

Reply


@XorNot β€’ 4 years

Replying to @tptacek πŸŽ™

Incidents like this remind me that the password problem is only partially solved by password managers: most of the internet (i.e. if you're not my bank) needs a simple, easy to script protocol that allows me to automate the process of rolling a lot of passwords.

Reply


@SadWebDeveloper β€’ 4 years

Replying to @tptacek πŸŽ™

Oh boy what a great week... first we have SHA-1 getting a fast-track to the obsolete hashes and now cloudflare is f*cking everyone because they tried to obfuscate emails from websites and fail to "test every edge" case... whats next is the question.

Reply


@hendzen β€’ 4 years

Replying to @tptacek πŸŽ™

I think this bug is kind of an indictment of Ragel. It has some great ideas, but since the generated code is so low level - and allows arbitrary blocks of code to be executed in the guts of the parser, bugs like these can result in this horrible memory issues - particularly since the generated code is often used to parse untrusted user input.

Reply


@thurston β€’ 4 years

Replying to @tptacek πŸŽ™

Author of Ragel here.

An experienced Ragel programmer would know that when you start setting the EOF pointer you are enabling code paths that never executed before. Like, potentially buggy ones. Eek!

Reply


@mderazon β€’ 4 years

Replying to @tptacek πŸŽ™

So did anyone find out why so many Google accounts got "action required" alerts yesterday ?

Reply


@Blackthorn β€’ 4 years

Replying to @tptacek πŸŽ™

This is probably gonna get buried at this point, but one thing I'm surprised about is this seems like yet another parser bug. Why are we still using hand-written parsers? Even if you're Very Smart, you'll probably get it wrong. We have parser generators for a lot of things. Even for mostly unparseable garbage like wild-type HTML we have pretty good libraries for handling it. Fresh hand-written parsers are just bombs waiting to explode.

Reply


@alkonaut β€’ 4 years

Replying to @tptacek πŸŽ™

Just stop using pointer arithmetic and manually managed buffers for anything security/safety related already.

Had this proxy been written in nearly any other language it wouldn't have had this vulnerability, like so many similar vulnerabilities.

Using ML or Rust or Java or whatever doesn't magically make all vulnerabilities disappear but it sure makes those that are intrinsic to C disappear. And that's not just a few.

There is just no excuse.

Reply


@tptacek β€’ 4 years

Replying to @tptacek πŸŽ™

Oh, my god.

Read the whole event log.

If you were behind Cloudflare and it was proxying sensitive data (the contents of HTTP POSTs, &c), they've potentially been spraying it into caches all across the Internet; it was so bad that Tavis found it by accident just looking through Google search results.

The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep everyone at work late today. And, this morning, Google announced the SHA-1 collision, which everyone (including the insiders who leaked that the SHA-1 collision was coming) thought was the big announcement.

Nope. A SHA-1 collision, it turns out, is the minor security news of the day.

This is approximately as bad as it ever gets. A significant number of companies probably need to compose customer notifications; it's, at this point, very difficult to rule out unauthorized disclosure of anything that traversed Cloudflare.

Reply


@yclept β€’ 4 years

Replying to @tptacek πŸŽ™

for easy firewalling and i'm sure a fun internet experience https://www.cloudflare.com/ips-v4

Reply


@mixedbit β€’ 4 years

Replying to @tptacek πŸŽ™

There is a huge fleet of compromised machines out there that belong to botnets. Soon we will see the botnets operators extracting content from these compromised machines browser caches to hunt for data leaked in this incident. Clearing search engines caches is just not enough, all secrets need to be replaced.

Reply


@dantiberian β€’ 4 years

Replying to @tptacek πŸŽ™

From Twitter:

"@taviso their post-mortem indicates this would've been exploitable only 4 days prior to your initial contact. Is that info invalid?" - https://twitter.com/pmoust/status/834916647873961984

"@pmoust Yes, they worded it confusingly. It was exploitable for months, we have the cached data." - https://twitter.com/taviso/status/834918182640996353

Reply


@dreamcompiler β€’ 4 years

Replying to @tptacek πŸŽ™

Only inherently unsafe languages like C make it possible for an amateur-hour HTML parsing blunder to spew secrets all over the Internet. If you can't be bothered to check your return codes, at least use a language that doesn't multiply the damage from that mistake a million-fold.

Reply


@chousuke β€’ 4 years

Replying to @tptacek πŸŽ™

I used the lastpass CLI tool and some UNIX tools to do a tentative check of which of my domains might be affected. Something like the following should work okay:

    lpass ls | egrep -o '[a-z]+\.[a-z]+' | sort > mydomains.sorted
    sort sorted_unique_cf.txt > cf_really_sorted
    comm -12 mydomains.sorted cf_really_sorted
It's not perfect (since it will only look at the lastpass item description, not the actual URL, and will only match foo.tld type domains), but it still found a number of domains for me

Reply


@nikisweeting β€’ 4 years

Replying to @tptacek πŸŽ™

I've compiled a list of 7,385,121 domains served through cloudflare using several scrapers. https://github.com/pirate/sites-using-cloudflare

The full list is available for download here (23mb) https://github.com/pirate/sites-using-cloudflare/raw/master/...

I will be updating it as I find more domains.

Reply


@homakov β€’ 4 years

Replying to @tptacek πŸŽ™

Chrome marking Cloudflare HTTPS as "Secure" must be turned into something different, like "Not So Secure" or whatever. Secure = end to end.

Cloudflare is MitM by design. Chrome and others must not tolerate it. This vulnerability is just another reason to do it asap.

Reply


@CapacitorSet β€’ 4 years

Replying to @tptacek πŸŽ™

>Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.

Considering the amount and sensitivity of the data they handle, I'm not sure a t-shirt is an appropriate top-tier reward.

Reply


@ThrustVectoring β€’ 4 years

Replying to @tptacek πŸŽ™

Maybe I'm being a bit too paranoid, but shouldn't your services be set up in a way that doesn't let Cloudflare touch that sort of sensitive data in the first place? You can't distrust everything, of course, but "compromised reverse-proxy acts as a MITM by logging and exfiltrating sensitive information" seems like it ought to be in the threat model of service providers.

Reply


@buildops β€’ 4 years

Replying to @tptacek πŸŽ™

Time to move to Incapsula

Reply


@cwisecarver β€’ 4 years

Replying to @tptacek πŸŽ™

This sounds to me like an object lesson in "Why you shouldn't write your own HTML parser."

Every time I see a dev trying to parse HTML with a custom solution or regex or anything other than a proven OSS library designed to parse HTML I recoil reflexively. Sure, maybe you don't need a parser to see if that strong tag is properly closed but the alternative is ...

Reply


@symlinkk β€’ 4 years

Replying to @tptacek πŸŽ™

I've been going through Google's and Bing's caches for about 2 hours looking for leaked credentials and I don't see much - many results don't have an option to view a cached copy. I think Google and Bing are wiping any cache entry that are affected by this vulnerability.

Reply


@_pmf_ β€’ 4 years

Replying to @tptacek πŸŽ™

From the incident report at https://blog.cloudflare.com/incident-report-on-memory-leak-c... (not the article):

> About a year ago we decided that the Ragel-based parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain.

I'd assume that at this point, customers would like to have a little more than a vague promise.

Reply


@tete β€’ 4 years

Replying to @tptacek πŸŽ™

Wow, I only recently had a discussion about "What if this happens?". Great timing to make a point. Unique "told you so" opportunity, but I actually am sad that this happened. Millions of people wasting time on password changes and related things again. :(

And now off to resetting a lots of password and checking where OTPs are possible.

Reply


@AYBABTME β€’ 4 years

Replying to @tptacek πŸŽ™

This comes around to me as something that just shouldn't have happened. CloudFlare are pretty big on Go, as far as I can tell (and I guess Lua for scripting nginx). Why was this parsing package written in a non memory-safe language? Parsing is one of those "obvious" things easy to mess up; the likelihood of a custom, hand written parser being buggy is pretty high. If it's somehow understood that your library is likely to have bugs, why do it in C/C++, where bugs often lead to bleeding memory? In a shop that's already fluent in Go, where they have the institutional knowledge to do it safely? Sure performance is not going to be the same, but with some care it'll get pretty close.

Sorry I hate to just be a coach commentator. Obviously hindsight is 20/20. Still I think there's a lesson here.

Reply


@hatsunearu β€’ 4 years

Replying to @tptacek πŸŽ™

>(It took every ounce of strength not to call this issue "cloudbleed")

and some chap did it anyways. yay, i guess.

Reply


@andrewvijay β€’ 4 years

Replying to @tptacek πŸŽ™

Can somebody please explain what exactly do they mean by a dump here? How is the leak happening? Is it something like we get extra data than we ask for in a request?

Reply


@stevebmark β€’ 4 years

Replying to @tptacek πŸŽ™

Is there a government body that can enforce fines over this? Or is a class action lawsuit the only way to seek damages?

Reply


@frankmoodie β€’ 4 years

Replying to @tptacek πŸŽ™

Question: what about the %99 of the internet users who have no idea what SSL/HTTP/any other web tech is ? How are they even going to be notified?

Reply


@benevol β€’ 4 years

Replying to @tptacek πŸŽ™

Well, keep centralizing and this is what you get, sooner or later.

Also, mono-cultures have always been a very bad idea, not just in agriculture.

Reply


@chm β€’ 4 years

Replying to @tptacek πŸŽ™

Some important parts:

    The examples we're finding are so bad, I cancelled some
    weekend plans to go into the office on Sunday to help
    build some tools to cleanup. I've informed cloudflare
    what I'm working on. I'm finding private messages from
    major dating sites, full messages from a well-known
    chat service, online password manager data, frames from
    adult video sites, hotel bookings. We're talking full
    https requests, client IP addresses, full responses,
    cookies, passwords, keys, data, everything.

    Cloudflare pointed out their bug bounty program, but I
    noticed it has a top-tier reward of a t-shirt.

    Cloudflare did finally send me a draft. It contains an  
    excellent postmortem, but severely downplays the risk
    to customers.

Reply


@Soarnrobertson β€’ 4 years

Replying to @tptacek πŸŽ™

This article is beginning to look like a whole bunch of people talking about a leak and not saying that they would use that data for vicious things.

Reply


@meowface β€’ 4 years

Replying to @tptacek πŸŽ™

Never been so relieved my company uses a different CDN...

Reply


@dmitrygr β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudflare's announcement, as it is currently worded, deserves the understatement-of-the-centry award.

Reply


@ers35 β€’ 4 years

Replying to @tptacek πŸŽ™

Hacker News uses Cloudflare: http://bgp.he.net/dns/news.ycombinator.com#_ipinfo

Add the following to your hosts file to bypass Cloudflare and access HN directly:

  50.22.90.248 news.ycombinator.com

Reply


@dorianm β€’ 4 years

Replying to @tptacek πŸŽ™

Here is a list of domains where I found public leaked data: http://doma.io/2017/02/24/list-of-affected-cloudbleed-domain...

Reply


@danvdragos β€’ 4 years

Replying to @tptacek πŸŽ™

How was https traffic leaked? Cloudflare, in order to offer its services, acts like a man in the middle and internally decrypts https traffic [0]

[0]: https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflar...

Reply


@sparkling β€’ 4 years

Replying to @tptacek πŸŽ™

This might be the time to point out the CloudFlare watch blog: http://crimeflare.com/

Reply


@clebio β€’ 4 years

Replying to @tptacek πŸŽ™

So, two of the three hard problems in computer science (fencepost and cache invalidation)?

Reply


@danvdragos β€’ 4 years

Replying to @tptacek πŸŽ™

How was https trafic leaked? Cloudflare, in order to offer its services, acts like a man in the middle and internally decrypts https trafic [0]

[0]: https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflar...

Reply


@fulafel β€’ 4 years

Replying to @tptacek πŸŽ™

Yet another strong argument for end-to-end security. Terminate in the middle, and you risk things like this.

Hopefully people will learn something from today.

Reply


@hehheh β€’ 4 years

Replying to @tptacek πŸŽ™

I'm a little drunk so please forgive me if I'm way off base here or if I'm ultimately describing a service that already exists.

Unless I'm mistaken, CloudFlare's services necessarily require they act as a MITM. Would it be possible or practical change the DDoS protection service such that it uses an agent on the customer's end (the CF customer) that relays relevant data to CF, instead of having CF MITM all data?

As it is now, we have:

  End user <-> CF MITM to inspect packet data <-> CF Customer site
where CF uses the data discovered through MITM (and other metadata such as IP) to determine if the end user is a bad actor.

What if we, instead, had something like:

  End user <-> CF TCP proxy <-> CF Customer site
                   ^                    |
                   |                    v
             CF decision agent <-- CF metadata ingest
The CF captive portal would not work with this but they could still shut down regular ol boring TCP DDoSes.

Reply


@borplk β€’ 4 years

Replying to @tptacek πŸŽ™

It says their bug bounty program has a top-tier reward of a t-shirt? Wow ... don't go bankrupt Cloudflare.

Reply


@secfirstmd β€’ 4 years

Replying to @tptacek πŸŽ™

"and even plaintext API requests from a popular password [1Password] manager that were sent over https"

Plaintext?

Reply


@enraged_camel β€’ 4 years

Replying to @tptacek πŸŽ™

Your comment got flagged and killed, which I thought was bullshit so I vouched for it.

Because you're correct: if CF's info sec team is "very very good at their jobs", how did this incident happen?

Reply


@Soarnrobertson β€’ 4 years

Replying to @tptacek πŸŽ™

Welp, time to move and get a different IP again :\

Reply


@hkjgkjy β€’ 4 years

Replying to @tptacek πŸŽ™

One of the reasons I prefer paying with Bitcoin over credit card, is that when using cryptocurrency I don't have to give them they key to my account - instead they give me an inbox that I send the value into.

Guessing a lot of credit card details are ripe for picking in the data they leaked.

Reply


@jhgjklj β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudflare please stop asking me if i am a robot and then ask to pick the store board posts for ever. What kind of idiot coded that, asking me always.

Reply


@chiefalchemist β€’ 4 years

Replying to @tptacek πŸŽ™

Let's be honest. There are holes. More than we care to admit. The truth, if embrassed, could undermine the world's economy. It's just question of when.

Reply


@matthewowen β€’ 4 years

Replying to @tptacek πŸŽ™

So, they know which sites leaked data in responses. It sounds like they can also say categorically that some sites won't have been affected (if they don't share any infrastructure with the sites that could have leaked data).

Will Cloudflare be explicitly notifying customers about whether data from their site could have been leaked by this bug?

Reply


@kmfrk β€’ 4 years

Replying to @tptacek πŸŽ™

So how does one find or generate a list of companies using CloudFlare to figure out how you're affected - kinda like HaveIBeenPwned.com?

Reply


@pjmlp β€’ 4 years

Replying to @tptacek πŸŽ™

Time for the C. A. R. Hoare's weekly quote, taking time to reflect on what happened since 1981 regarding computer security on system languages.

The first principle was security: The principle that every syntactically incorrect program should be rejected by the compiler and that every syntactically correct program should give a result or an error message that was predictable and comprehensible in terms of the source language program itself. Thus no core dumps should ever be necessary. It was logically impossible for any source language program to cause the computer to run wild, either at compile time or at run time. A consequence of this principle is that every occurrence of every subscript of every subscripted variable was on every occasion checked at run time against both the upper and the lower declared bounds of the array. Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to - they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law.

-- Turing Award lecture 1981

Reply


@brilliantcode β€’ 4 years

Replying to @tptacek πŸŽ™

how far back does this affect websites on cloudflare? I removed mine a year ago because I was using it for the SSL.

This will put the final lid on cloudflare anyhow. Sticking with AWS.

Reply


@jacquesm β€’ 4 years

Replying to @tptacek πŸŽ™

Hm. Not so good. The main website that I log in to that uses CloudFlare is this one.

Reply


@jitbit β€’ 4 years

Replying to @tptacek πŸŽ™

Webmasters and App-devs running on CloudFlare. You (at least) have to "force-logout" your users that have "remember me" cookie set.

At least change the cookie name so the token stops working. For example, in ASP.NET - change the "forms-auth" name in the web.config file. etc etc.

Reply


@vermontdevil β€’ 4 years

Replying to @tptacek πŸŽ™

I got an email from Cloudflare and here's an excerpt about the # of sites affected by this.

Not sure what to make of it - the low number of domains affected.

====================================

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

Reply


@Globz β€’ 4 years

Replying to @tptacek πŸŽ™

I was planning on moving my website over DigitalOcean and I now http://www.doesitusecloudflare.com/?url=www.digitalocean.com is telling me that they are affected by cloudbleed, I guess I should wait it out...

Reply


@xt00 β€’ 4 years

Replying to @tptacek πŸŽ™

Wow apparently they never fuzzed their input and looked at the output. A malformed html input should be about the easiest possible thing to try... yeouch...

Reply


@steven_pack β€’ 4 years

Replying to @tptacek πŸŽ™

If only there were a systems programming language, offering c-like performance with memory guarantees and well suited to high throughput network servers that would catch this class of bugs at compile-time [1] [2]

[1] https://www.rust-lang.org/en-US/ [2] Self declared rust fanboy

Reply


@aioprisan β€’ 4 years

Replying to @tptacek πŸŽ™

CloudFlare's disclosure severely downplays the impact that this can have on their customers. We're going to close our account shortly.

Reply


@jkells β€’ 4 years

Replying to @tptacek πŸŽ™

My first thought was relief, thank god I'm not using Cloudflare.

Where would you even start to address this? Everything you've been serving is potentially compromised, API keys, sessions, personal information, user passwords, the works.

You've got no idea what has been leaked. Should you reset all your user passwords, cycle all or your keys, notify all your customers that there data may have been stolen?

My second thought after relief was the realization that even as a consumer I'm affected by this, my password manager has > 100 entries what percentage of them are using CloudFlare? Should I change all my passwords?

What an epic mess. This is the problem with centralization, the system is broken.

Reply


@curuinor β€’ 4 years

Replying to @tptacek πŸŽ™

Can we start a list of affected right now? I found:

OKCupid

Uber

people claiming 1Password, can't find

Reddit

Lyft

Yelp

Pingdom

Digital Ocean

Montecito Bank and Trust

Reply


@DanielDent β€’ 4 years

Replying to @tptacek πŸŽ™

There are still a lot of results with leaked data in Google's Cache and they are pretty easy to find..

Some possible queries: "CF-Int-Brand-ID", nginx-cache "Certisign Certificadora Digital",

Once you find one, you can look through the results for unusual strings/headers which you can use to find more results.

Many results have clearly been removed from Google's cache, but.. many also have not.

Reply


@no_protocol β€’ 4 years

Replying to @tptacek πŸŽ™

Is there a list of sites potentially affected?

I'm assuming I need to change my passwords on a significant number of sites. So far none of them have alerted me to a potential breach. Would love to have a head start.

Reply


@ComputerGuru β€’ 4 years

Replying to @tptacek πŸŽ™

Some day, the world will wake up to the fact that we've taken the beauty of a decentralized internet and willingly traded it in for a single-point-of-failure design.

I will refrain from any criticism of Cloudflare and what I think about this because they're going through hell as it is. But everyone else is fair game. The higher a level of service you centralize, the more you stand to lose.

Reply


@loeg β€’ 4 years

Replying to @tptacek πŸŽ™

Anyone know which password manager uses Cloudflare? Just trying to figure out if I'm affected.

Reply


@piker β€’ 4 years

Replying to @tptacek πŸŽ™

Given that the plaintext is cached (or feared to be), is googling/binging one's passwords a bad way to check for pwnage?

Reply


@pmahoney β€’ 4 years

Replying to @tptacek πŸŽ™

I haven't found a clear answer to this:

CloudFlare has multiple SSL configurations:

> Flexible SSL: There is an encrypted connection between your website visitors and Cloudflare, but not from Cloudflare to your server.

> Full SSL: Encrypts the connection between your website visitors and Cloudflare, and from Cloudflare to your server

(I'll add Full SSL mode still involves CloudFlare terminating SSL (decrypting) before re-encrypting to communicate to your server)

If I am running in Full SSL mode, is (or was) my data vulnerable to being leaked?

Reply


@xenadu02 β€’ 4 years

Replying to @tptacek πŸŽ™

Another day, another C memory safety bug that completely breaks all security everywhere.

We're definitely doomed to repeat the same mistakes over and over.

Reply


@DannyBee β€’ 4 years

Replying to @tptacek πŸŽ™

So, does the t-shirt say: "I found a zero-day bug in cloudflare and all i got was this lousy

X-Uber-token:

X-Uber-latitude:

... "

Reply


@bovermyer β€’ 4 years

Replying to @tptacek πŸŽ™

Well, my day tomorrow is going to be busy. So's my evening tonight, I guess.

Reply


@implr β€’ 4 years

Replying to @tptacek πŸŽ™



@hkjgkjy β€’ 4 years

Replying to @tptacek πŸŽ™

HaveIBeenPwnd must be having a great day today!

Reply


@bcl β€’ 4 years

Replying to @tptacek πŸŽ™

Here's a simple little Rust app to check a list of domains for CF usage -- https://github.com/bcl/uses-cf

Reply


@bartkappenburg β€’ 4 years

Replying to @tptacek πŸŽ™

From a cloudflare employee:

"We were working to disclose the bug as quickly as possible, but wanted to clean up search engine caches before it became public because we felt we had a duty of care to ensure that this private information was removed from public view. We were comfortable that we had time as Google Project Zero initially gave us a 90 day disclosure window (as can still be seen in their incident tracker), however after a couple of days, they informed us that they felt that 7 days was more appropriate. Google Project Zero ended up disclosing this information after only 6 days."

Reply


@jcwayne β€’ 4 years

Replying to @tptacek πŸŽ™

Makes me wonder if the Great Firewall has a caching layer.

Reply


@johnhenry β€’ 4 years

Replying to @tptacek πŸŽ™

"Cloudbleed".

Reply


@apple4ever β€’ 4 years

Replying to @tptacek πŸŽ™

I have yet to receive an email about this. Very disappointed that I had to find out via another source 12 hours after the blog post was up.

Reply


@afandian β€’ 4 years

Replying to @tptacek πŸŽ™

In addition to comments here calling the words 'memory leak' disingenuous because it's technically correct but underplays the problem, I'm now seeing articles in non-technical publications referring to the incident as a "leak".

In the wider world the word "leak" doesn't mean memory access patterns, it means deliberate sabotage.

The headline in "The Verge" is "Password and dating site messages leaked by internet giant Cloudflare". That's technically correct too, but also gives completely the wrong message.

Simpler, proactive messaging from Cloudfront might have helped here.

Reply


@helper β€’ 4 years

Replying to @tptacek πŸŽ™

I was able to get a few hits from a quick google search that are still in google's webcache.

Reply


@jtchang β€’ 4 years

Replying to @tptacek πŸŽ™

This is scary stuff. Any key/password that you used on a cloudflare site should be considered compromised.

That's a crapton of keys.

Reply


@mabbo β€’ 4 years

Replying to @tptacek πŸŽ™

Signs you are about to have a bad time: Tavis Ormandy publicly tweets that he urgently needs someone from your security team to contact him, and no, the public disclosure form won't do.

Reply


@planetix β€’ 4 years

Replying to @tptacek πŸŽ™

So time to reset password and logout of all mobile apps to get new authorization tokens?

Reply


@snikeris β€’ 4 years

Replying to @tptacek πŸŽ™

Anyone have any additional information about this bit from the comments:

> and even plaintext API requests from a popular password manager that were sent over https (!!).

Reply


@cypherpunks01 β€’ 4 years

Replying to @tptacek πŸŽ™

"We also undertook other search expeditions looking for potentially leaked information on sites like Pastebin and did not find anything."

How comforting!

Reply


@rdl β€’ 4 years

Replying to @tptacek πŸŽ™

Neither this thread nor the Cloudflare blog post include concise steps for customers who were exposed.

There's an argument for changing secrets (user passwords, API keys, etc.) for potentially affected sites, plus of course investigating logs for any anomalous activity. It would be nice if there were a guide for affected users, maybe a supplemental blog post.

(and yet again: thank you Google for Project Zero!)

Reply


@philip1209 β€’ 4 years

Replying to @tptacek πŸŽ™

Interesting. Cloudflare uses a lot of Go, which should hypothetically be memory safe. Was this system in Go? If so, I would be interested in seeing proof of concept code for a vulnerability like this.

Reply


@_wmd β€’ 4 years

Replying to @tptacek πŸŽ™

Step 1) MITM the entire Internet, undermining its SSL infrastructure, build a business around it

Step 2) leak cleartext from said MITM'd connections to the entire Internet

I recently noted that in some ways Cloudflare are probably the only entity to have ever managed to cause more damage to popular cryptography since the 2008 Debian OpenSSL bug (thanks to their "flexible" ""SSL"" """feature"""), but now I'm certain of it.

"Trust us" doesn't fly any more, this simply isn't good enough. Sorry, you lost my vote. Not even once

edit: why the revulsion? This bug would have been caught with valgrind, and by the sounds of it, using nothing more complex than feeding their httpd a random sampling of live inputs for an hour or two

Reply


@Soarnrobertson β€’ 4 years

Replying to @tptacek πŸŽ™

So, would LastPass be involved in this at all? Do tey use CloudFlare?

Reply


@manigandham β€’ 4 years

Replying to @tptacek πŸŽ™

Side note: HackerNews uses CloudFlare.

Reply


@joepie91_ β€’ 4 years

Replying to @tptacek πŸŽ™

This is probably a good moment to recall the article I published a while ago about how CloudFlare is actively putting the web at risk: http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-hav...

This is precisely why. The only thing that surprises me about this, is that it was an accidental disclosure rather than a breach. Other than that, this was completely to be expected.

EDIT: Also, this can't be repeated enough: EVERYBODY IS AFFECTED. Change your passwords, everywhere, right now. Don't wait for vendors to notify you.

Anything could have irrevocably leaked, and you have no way of knowing for sure, so assume the worst.

Reply


@jlgaddis β€’ 4 years

Replying to @tptacek πŸŽ™

Anyone wrote a script yet that checks the top 1M (or so) web sites to find out which use Cloudflare? It would help with knowing what secrets I need to change (as an end user -- I'm not a Cloudflare customer, thank $deity).

Reply


@anc84 β€’ 4 years

Replying to @tptacek πŸŽ™

Are the Chinks worse than the Russkis though????! /s

Reply


@tkachenko β€’ 4 years

Replying to @tptacek πŸŽ™

Small service to check if your site is POSSIBLE affected to CloudFlare data leaks https://cloudflareleaks.webtls.com/

Reply


@soheil β€’ 4 years

Replying to @tptacek πŸŽ™

What's the rationale behind sending user PII through a CDN? Presumably that is useful to that one user only so a CDN wouldn't be super useful in distributing the load across its edges. Also doesn't CDN caching kinda defeat the purpose of having SSL?

Reply


@kogir β€’ 4 years

Replying to @tptacek πŸŽ™

I'm not 100% clear: Only three features were affected, and only sites with one or more of those features enabled leaked data into their pages.

But was the leaked data similarly limited to only the sites with the features enabled? Or could it have come from any request - even an entirely unrelated site?

Reply


@sambull β€’ 4 years

Replying to @tptacek πŸŽ™

RIP Cloudflare 2017.. took you long enough

Reply


@orasis β€’ 4 years

Replying to @tptacek πŸŽ™

Our CNAME pointing to github pages was down on Cloudflare today with a 1014 error. I'm guessing they broke some other stuff while scrambling to fix this privacy issue? Not a good day for them.

Reply


@techolic β€’ 4 years

Replying to @tptacek πŸŽ™

Is there an International Day of Internet Security? I think we should make today that day.

Reply


@hacknat β€’ 4 years

Replying to @tptacek πŸŽ™

I'm surprised to learn that people with real security concerns are using Cloudflare. I put it in front of my blog, but I would never use it in front of something that has sensitive data. I just don't get how companies like Zendesk could be so stupid. I barely blame Cloudflare. If you think terminating SSL with a CDN is a good idea you get what you deserve.

Reply


@omgtehlion β€’ 4 years

Replying to @tptacek πŸŽ™

What bothers me is not the bug itself, but the fact that so much sites and apps terminate SSL at cloudflare that NSA/FBI/other-3-letter-agency does not need to come after any separate company, but just needs to tap cloudflare and call it a day.

Reply


@artursapek β€’ 4 years

Replying to @tptacek πŸŽ™

Holy shit, this could be a company-ending event. For CloudFlare or any of its clients.

Reply


@actuator β€’ 4 years

Replying to @tptacek πŸŽ™

I wrote a script which checks the domains you have visited from your chrome history to see if they use Cloudflare by checking if the header `cf-ray` is present in their response headers: https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d...

Found my bank's site on it. :(

Reply


@djhworld β€’ 4 years

Replying to @tptacek πŸŽ™

Can someone explain to me why they were parsing HTML in the first place? That's the bit I don't fully understand, but I've not got experience of what Cloudflare does, I thought they were a CDN

Reply


@a3n β€’ 4 years

Replying to @tptacek πŸŽ™

Password managers are mentioned.

I looked on the lastpass blog (s/www/blog/), nothing about this. Is it just too early?

Reply


@nkkollaw β€’ 4 years

Replying to @tptacek πŸŽ™

Could some kind soul do an ELI5?

I'm not lazy, it's just overwhelming trying to figure out what's actually going on with all these comments...

Reply


@edoceo β€’ 4 years

Replying to @tptacek πŸŽ™

Phase 3 is Profit!

Reply


@the_common_man β€’ 4 years

Replying to @tptacek πŸŽ™

How does such a simple bug not get picked by auto tests, ci or end to end tests? I am baffled. Since we are behind cloudflare, I am not sure what I should tell my manager now. I lack the technical know how to parse that extremely technical article. Are we supposed to just assume all our traffic that passed via cloudflare is possibly compromised?

It's also a bit sad that travis has to contact cloudflare by twitter. Seriousy?

Edit: https://twitter.com/taviso/status/832744397800214528 is the tweet in question

Reply


@acd β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudflare is also breaking anonymous surfing by throwing captchas at you. Security wise they do DDoS ok but not WAF which Incapsula does a lot better. When I mean better I mean protection against exploits.

Reply


@HugoDaniel β€’ 4 years

Replying to @tptacek πŸŽ™

So its not only the tor browser experience that sucks with cloudflare.

Reply


@faragon β€’ 4 years

Replying to @tptacek πŸŽ™

Does anyone know if there is a way for mapping virtual addresses to areas with zeroes and replacing it with the memset to 0 on write access, so software could be still efficient without calling calloc() instead of malloc()? (i.e. memset to 0 only for actually written zones)

Reply


@fagnerbrack β€’ 4 years

Replying to @tptacek πŸŽ™

TL;DR for the lazy ones:

> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

This is huge.

I mean, seriously, this is REALLY HUGE.

Reply


@mordant β€’ 4 years

Replying to @tptacek πŸŽ™

Apparently, the only way to change one's Uber password is to use the 'Forgot password' path on their login page.

So, I clicked on that - and I get a 500 error from NGINX.

My guess is that a lot of services are going to be overwhelmed by the sheer volume of password reset requests, thus preventing users from resetting their passwords.

Reply


@jjoe β€’ 4 years

Replying to @tptacek πŸŽ™

Everyone: change your HN password asap!

Reply


@daxfohl β€’ 4 years

Replying to @tptacek πŸŽ™

Anyone know of a way to google for your passwords (assuming you have strong, unique passwords) to see if they've been exposed anywhere, without exposing them?

Reply


@rickdmer β€’ 4 years

Replying to @tptacek πŸŽ™

I created a Chrome extension that searchs your bookmarks for sites that use Cloudflare: https://chrome.google.com/webstore/detail/cloudbleed-bookmar...

Reply


@spydum β€’ 4 years

Replying to @tptacek πŸŽ™

People are going to lambast CF for downplaying the impact, and there could be merit in that.

However, I really want to say I am absolutely impressed with both Project Zero AND Cloudflare on so many fronts, from clarity of communication, to collaboration, and rapid response. So many other organizations would have absolutely tanked when presented with this problem. Huge kudos for CF guys understanding the severity and aligning resources to make the fixes.

In terms of P0 and Tavis though, holy crap. Where the heck would we be without these guys? Truly inspiring !

Reply


@Rican7 β€’ 4 years

Replying to @tptacek πŸŽ™

Yeaaaaa, this isn't good.

This is what CloudBleed looks like, in the wild: https://gfycat.com/ElatedJoyousDanishswedishfarmdog

A random HTTP request's data and other data injected into an HTTP response from Cloudflare.

Sick.

Reply


@Xorlev β€’ 4 years

Replying to @tptacek πŸŽ™

> One of the advantages of being a service is that bugs can go from reported to fixed in minutes to hours instead of months. The industry standard time allowed to deploy a fix for a bug like this is usually three months; we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes.

Great, that makes me feel so much better! I'm sorry, don't try to put a cherry on the top when you've just leaked PII and encrypted communications.

Additionally, most vendors in the industry aren't deployed in front of quite as much traffic as CloudFlare is. It's a miracle that ProjectZero managed to find the issue.

Reply


@mattbee β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudflare isn't just a security hole in the middle of the internet, they're a protection racket.

If you wanted to pay to DDoS a site, search for "booter" and you'll get a list of sites that will take another site off the internet for money with a flood of traffic.

quezstresser.com webstresser.co topbooter.co instabooter.com booter.xyz critical-boot.com top10booters.com betabooter.com databooter.com

etc. etc. - from the first 30 results I could find 2 booter sites that weren't hosted by Cloudflare.

But hey, pay Cloudflare and your site too can be safe from DDoS attacks...

Reply


@user5994461 β€’ 4 years

Replying to @tptacek πŸŽ™

> The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

1) From the metrics I recalled when I interviewed there, and assuming the given probability is correct, that means a potential of 100k-200k paged with private data leaked every day.

2) What's the probably that a page is served to a cache engine? Not a clue. Let's assume 1/1000.

3) That puts a bound around a hundred leaked pages saved per day into caches.

4) Do the cache only provide the latest version of a page? I think most do but not all. Let's ignore that aspect.

5) What's the probably that a page contains private user information like auth tokens? Maybe 1/10?

6) So, that's 10 pages saved per day into the internet search caches.

7) That's on par with their announcement: "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains." Well, not that we know for how long this was running.

8) Now, I don't want to downplay the issue, but leaking an dozen tokens per day is not that much of a disaster. Sure it's bad, but it's not remotely close to the leak of the millennia and it's certainly not internet scale leak.

9) For the record, CloudFlare serves over one BILLION human beings. Given the tone and the drama I expected way more data from this leak. This is a huge disappointment.

Happy Ending: You were probably not affected.

Reply


@askvictor β€’ 4 years

Replying to @tptacek πŸŽ™

Could this be the reason behind having to reauth my Google accounts in the past couple of days? I.e. did Google invalidate all auth tokens in case they leaked via a third party website via CF?

Reply


@SandB0x β€’ 4 years

Replying to @tptacek πŸŽ™

Can someone provide a lay-person's explanation of the issue and its implications?

Reply


@joeyh β€’ 4 years

Replying to @tptacek πŸŽ™

The bug was in cloudflare servers, not code run on customer's own web servers, right?

Reply


@tr32q423 β€’ 4 years

Replying to @tptacek πŸŽ™

The root cause is apparently coming from auto-generated code that causes buffer overrun:

    /* generated code */
    if ( ++p == pe )
        goto _test_eof;

With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains.

The examples in the report shows Uber, okcupid , etc. It would be good to know the full list, to know what password might have been compromised.

https://blog.cloudflare.com/incident-report-on-memory-leak-c...

Reply


@ta2987 β€’ 4 years

Replying to @tptacek πŸŽ™

We need an official and comprehensive list of domains served by Cloudflare throughout the affected period.

Reply


@willtim β€’ 4 years

Replying to @tptacek πŸŽ™

If you must write your HTML parser in C/C++, then you should expect buffer overruns.

Reply


@packetized β€’ 4 years

Replying to @tptacek πŸŽ™

Salient question at this point: Did Cloudflare have any systems in place that would allow themselves to identify queries that were abusing this defect?

Reply


@codezero β€’ 4 years

Replying to @tptacek πŸŽ™

Why do they need to add google analytics to random people's web pages?

Reply


@coindork β€’ 4 years

Replying to @tptacek πŸŽ™

And it shall be called Cloudbleed.

Reply


@Sami_Lehtinen β€’ 4 years

Replying to @tptacek πŸŽ™

I made similar site too, but with geolocation, tags, and fully threaded replies and private messages. Like & Dislike - As well as machine learning which will dig most interesting posts for you. As well as score near by posts higher etc. But nobody cared. So I'll be shutting it down in 6 months. (Domain expires)

Reply


@throwaway7959 β€’ 4 years

Replying to @tptacek πŸŽ™

Can anyone ELI5 what's going on?

Reply


@aerovistae β€’ 4 years

Replying to @tptacek πŸŽ™

I noticed StackOverflow is on the list of compromised sites. I sign into that via my google account. Does this mean I need to change Google credentials?

Reply


@csomar β€’ 4 years

Replying to @tptacek πŸŽ™

Oh boy, this is bad as fuck. Major bitcoin exchanges were affected and these are exchanges where if you can login, you might be able to withdraw the cash irreversibly for ever.

I'm trying to figure out how bad this is; and a part from the exchanges I'm using which other sensitive sites are concerned.

Reply


@dkarapetyan β€’ 4 years

Replying to @tptacek πŸŽ™

Every piece of dependency in your stack is a vulnerability vector. I feel like this is the only sane assumption to make these days. Yesterday I was thinking of doing some stuff with cloudflare and today I'm reading this report.

The modern web requires a paranoid attitude.

Reply


@abalone β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudfare's bug bounty maximum reward[1]:

1. Recognition on our Hall of Fame.

2. A limited edition CloudFlare bug hunter t-shirt. CloudFlare employees don't even have this shirt. It's only for you all. Wear it with pride: you're part of an exclusive group.

3. 12 months of CloudFlare's Pro or 1 month of Business service on us.

4. Monetary compensation is not currently offered under this program.

Guessing they're gonna reconsider #4 at this point.

[1] https://hackerone.com/cloudflare

Reply


@ffjffsfr β€’ 4 years

Replying to @tptacek πŸŽ™

Does anyone know answer to this question someone is asking there at the end? Is it related?

> could you tell us why a lot of people had to re-authenticate their Google accounts on their devices all of the sudden? It may not have been related, but Google definitely did something that had us all re-authenticate.

I too had to reauthenticate and was very worried because it was first time I had to do this, I thought something bad happened with my account and it was very suspicious.

Reply


@sparkling β€’ 4 years

Replying to @tptacek πŸŽ™

I know what Cloudflare is but i don't quite understand the underlying issue.

Can someone explain in simpler terms what happened here and how it a) affects sites using Cloudflare and b) Users accessing sites with Cloudflare?

Reply


@tannhaeuser β€’ 4 years

Replying to @tptacek πŸŽ™

Holy sh*t. Is this the end of Cloudflare with the trust being absolutely destroyed and lawsuits coming in? Can't say I'm sad for them. Cloudflare sells you DDOS protection, and hosts (eg. masks the IP of) the very DDOSers to protect against themselves, which I find bordering on the criminal.

Hosters like Hetzner, OVH have for a year now offered DDOS protection (I'm guessing it's heuristic rate limiting, but they won't tell details b/c that would make it trivial to workaround it, so they say). Could someone characterize their offering and tell me if it's any good?

To those spinning a story against C programming here: it is entirely possible (trivial, even) to isolate address spaces between requests, and has been for like 25 years (CGI programming) and more. When you absolutely must use a long running, single-address space service container, OpenBSD's httpd shows how to do it right (goes to great lengths to randomize/re-initialize memory etc.). I agree, though, that using straight C isn't a good choice for the latter.

Reply


@kyledrake β€’ 4 years

Replying to @tptacek πŸŽ™

Friendly reminder that Cloudflare willingly hosts the top DDoS-for-hire attack sites, and refuses to take them down when they are reported.

Run WHOIS on them, it's almost 100% behind Cloudflare: https://www.google.com/#q=ddos+booter

I would be less concerned about the fact that Cloudflare is spraying private data all over the internet if people weren't being coerced into it by a racket.

We won't have a decentralized web anymore if this keeps going. The entire internet will sit behind a few big CDNs and spray private data through bugs and FISA court wire taps. God help us all if this happens.

Reply


@DangerousPie β€’ 4 years

Replying to @tptacek πŸŽ™

Has anybody else actually received an email from Cloudflare about this? I'm a paying customer, but haven't heard anything from them yet. I hope they don't expect they can leave it at a random blog post that will go by unnoticed?

Reply


@ReedJessen β€’ 4 years

Replying to @tptacek πŸŽ™

Reddit just told me my account was compromised

Reply


@Tiksi β€’ 4 years

Replying to @tptacek πŸŽ™

> Many of the logged urls contained query strings from https requests that I don't think they intended to share.

I guess this confirms a few things.

- The complete query strings are logged,

- They don't appear to be too concerned with who accesses the logs internally or have a process that limits the access, and

- They're willing to send those logs out to a random person.

Reply


@fagnerbrack β€’ 4 years

Replying to @tptacek πŸŽ™

Cloudflare blog post related to this incident: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

Reply


@jgrahamc β€’ 4 years

Replying to @tptacek πŸŽ™



About Us

site design / logo Β© 2021 Box Piper