Hacker News Re-Imagined

Google's infamous internal 2010 “I just want to serve 5TB” video now public

  • 1130 points
  • 18 days ago

  • @raldi
  • Created a post
  • • 1 comments

Google's infamous internal 2010 “I just want to serve 5TB” video now public


@sergiotapia 18 days

Replying to @raldi 🎙

Much more infamous.

"Mongo DB Is Web Scale" - https://www.youtube.com/watch?v=b2F-DItXtZs

Reply


@zzt123 17 days

Replying to @raldi 🎙

Was this video ever used by Google for anything outside of internal before now? I swear I’ve seen this before (it’s still hilarious!) but I never worked at Google.

Reply


@jboggan 18 days

Replying to @raldi 🎙

Eagerly awaiting the "GoogFellas" video leak.

Reply


@loxias 18 days

Replying to @raldi 🎙

And now, in 2021, Google has inflicted their "clarity" on the rest of the world. I miss jobs from the 2000s, the jobs where you were paid to write software for a living.

You know, engineering! Given a task, or set of requirements, develop software on your computer, software which eventually runs on the customer's computer, where it's used to solve the customer's problem.

My most recent full time employment a year ago was at a great company. Healthy culture, some of the most talented coworkers I've ever had the pleasure to work along side.

Over the year I lasted there, I used for the first time: Docker, Golang, Kubernetes, Terraform, Gitlab, Saltstack, Prometheus, (and probably other middleware that my brain has GCd to free space). I was barely able to get anything done. At least, it always felt that way.

Maybe I'm just an idiot, I don't know. I'd accept it if true! What I do know is that I used to be able to build things for people, be compensated well for it, and get satisfaction from a customer liking what I've built. It was simple.

In this brave new world, with containers, pods and this and that and the other thing, where it can take months before one even understands enough primitives to do a "hello world".... how can anything ever get done?? How can anything inventive, creative, or experimental emerge from our industry when the develop/test/improve cycle has gone from minutes to weeks or months?

I don't know what the future looks like, but the present strikes me as unsustainable in the long run.

(edit: Wow! I expected this to be downvoted to oblivion, not my highest rated comment on the site...)

<tiny>(Forgive this shameless self promotion: if, dear reader-who-is-a-hiring-manager, you have a paid role for a lowly but experienced systems engineer who doesn't know anything about "web" or "apps" or "social" but is quite adroit with C/C++ (and a few others), most "sciencey/mathy" type problems, signal processing, firmware, network protocols, automation/scripting, and more, ... email is welcome!)</>

Reply


@dpryden 18 days

Replying to @raldi 🎙

Non-Googler: What do all those words mean?

Noogler: Haha, this video is so funny!

L4 SWE: (Crying because the video is so true)

L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.

L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...

Reply


@nojvek 18 days

Replying to @raldi 🎙

The borgman readability approvers makes me chuckle.

At Stripe, there were language approvers. Only those blessed could approve PRs. Even XML had a set of approvers. I had fun time getting hold of an XML approver.

Reply


@codewiz 17 days

Replying to @raldi 🎙

10 years ago, on my first week at Google, my manager suggested I watch this video to get a fairly accurate idea of what the team does :-)

Reply


@sunyc 18 days

Replying to @raldi 🎙

I actually have Borgmon readability! Peer bonus pls.

Reply


@dang 18 days

Replying to @raldi 🎙

Recent and related:

I don’t know how to count that low - https://news.ycombinator.com/item?id=28988281 - Oct 2021 (259 comments)

especially this comment: https://news.ycombinator.com/item?id=29032656

Reply


@mlindner 18 days

Replying to @raldi 🎙

Wow I saw this somewhere a long time ago. But I don't remember where and in what context.

Reply


@b20000 18 days

Replying to @raldi 🎙

meanwhile google docs is still a slow piece of shit.

Reply


@Zababa 18 days

Replying to @raldi 🎙

There's almost some kind of irony on uploading that to youtube, a feeling of "why can't I deploy my service as easily as people can upload videos to youtube?".

Reply


@silentsea90 18 days

Replying to @raldi 🎙

Actively working towards being a Xoogler so I don't have to live in this dystopia.

Reply


@devnull3 18 days

Replying to @raldi 🎙

At 2:05 the green dude asks if you think your users are scum and do you hate them.

The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.

Reply


@bluefox 18 days

Replying to @raldi 🎙

BitTorrent existed since 2001. Get on with the times.

Reply


@martini333 18 days

Replying to @raldi 🎙

Google: The Sunk Cost Fallacy

Reply


@GauntletWizard 18 days

Replying to @raldi 🎙

Holy crap! I've been asking for this for forever[1]. Thank you to the leaker!

[1] https://news.ycombinator.com/item?id=21786729

Reply


@Spivak 18 days

Replying to @raldi 🎙

I’m so confused, isn’t this just like basic highly available infrastructure mixed with a toxic SRE culture?

I want to serve 5TB!

Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.

And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.

Are people not used to what it really takes to “just run some code?”

Reply


@cromwellian 18 days

Replying to @raldi 🎙

As a Googler, it's often easier for me to setup a GCP consumer account, AWS, or Heroku account to demo something, compared to using anything internal. I remember the most annoying situation was like 10 years ago when me and other engineers ported Quake 2 to run in Chrome, we were in a time crunch to demo it multiplayer, and I ended up setting up an AWS account to serve it. But then I left it running and forgot and ended up getting a few hundred dollars billed to me because the Quake2 server was chewing CPU.

Reply


@didip 18 days

Replying to @raldi 🎙

What is "Borgmon readability" and why was it important. I think that's one of the punch line of the video.

Reply


@Lammy 18 days

Replying to @raldi 🎙

Personally I'm still waiting for a copy of "Pusher, I have a hotfix."

Reply


@nunez 18 days

Replying to @raldi 🎙

still hilarious

Reply


@ChuckMcM 17 days

Replying to @raldi 🎙

Wow, way to trigger my Google PTSD. :-)

I think the only thing I would have added would be using some service that recently was changed so that someone could get promoted but does the same thing the old one did but with different bugs.

Reply


@birken 18 days

Replying to @raldi 🎙

Hey... those of us that worked on Google's internal Bigtable service worked very hard so you didn't have a file to a ticket to set up replication between your Bigtable cells.

The rest does seem about accurate though.

Reply


@gcampos 18 days

Replying to @raldi 🎙

What exactly are these "peer bonuses"? Is it real? Is it what I think it is? Do people actually use them as bargain chips?

Reply


@dekhn 18 days

Replying to @raldi 🎙

Thank you, whomever did this! I asked for it in a comment recently.

This video basically is making fun of a common situation of Google at the time, where a person wants to serve up some data for analytics, but the sysadmins expect the person to follow a process intended for much more complex and high availability services run by teams of skilled engineers.

It parodies SRE as a BOFH sysadmin, even though in general SRE are quite easygoing and helpful.

It helped poked fun at a number of overly stuffy processes and also helped push people to make hosting modest datasets (like this 5TB one) easier.

Reply


@Reschivon 17 days

Replying to @raldi 🎙

> It was a site where you could type in a script and it would do text-to-speech and actually > animated some goofy characters to lip-sync the words for you

Does anyone still have the code for generating "Fox girl and Broccoli man" videos? I'm thinking of starting a small revival of this meme.

Reply


@Wonnk13 18 days

Replying to @raldi 🎙

One of the few things I miss about my time there...

Never did get Java readability :(

Reply


@jazzyjackson 18 days

Replying to @raldi 🎙

what am I looking at here

EDIT: it has been explained to me: https://rachelbythebay.com/w/2021/10/30/5tb/

Reply


@opinion-is-bad 18 days

Replying to @raldi 🎙

The multiple repetitions of “This is Google” hit home for me. I never worked as a software engineer so much of the rest is out of scope to my experience, but the constant idolization of Google, and by proxy each other for working at such a place, eventually changed from feeling coy to cultish.

Reply


@xiphias2 18 days

Replying to @raldi 🎙

I wish I would have had this video before 2010. I got paged at night every time there was a PCR failover, and I didn’t know what to do with it. This video is better than all the extensive documentation that we had.

Reply


@cletus 18 days

Replying to @raldi 🎙

Ah, this takes me back (disclaimer: Xooger, 2010-2017). It's painful and funny because it's true. Or was true.

Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.

Reply


@romwell 18 days

Replying to @raldi 🎙

Ah, as a xoogler, I can finally enjoy this video without tears :)

Reply


@sbpayne 18 days

Replying to @raldi 🎙

I'm so glad I can see this again. I forgot how much I missed this.

Reply


@metanonsense 18 days

Replying to @raldi 🎙

Move fast and break things! And while you are at it, please, don't break anything.

Reply


@mseepgood 18 days

Replying to @raldi 🎙

This monotonous speech synthesis is annoying to listen to. The delivery of the jokes is awful. Who can sit through a 3 min video like that?

Reply


@nunez 18 days

Replying to @raldi 🎙

this is making me miss memegen

google had its downs, but wasting hours on memegen was not one of them

Reply


@slac 18 days

Replying to @raldi 🎙

I have the t-shirt!

Reply


@jamestimmins 18 days

Replying to @raldi 🎙

As an external user who has found Google's services to be incomprehensible, it's nice to know it is (was) equally as painful internally.

Reply


@frakkingcylons 18 days

Replying to @raldi 🎙

For anyone else who'd rather read than watch this video, here's the transcript (from YT's auto-generated captions): https://pastebin.com/8UrFftM6

Reply


@throwaway20371 18 days

Replying to @raldi 🎙

These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.

Reply


@Imnimo 18 days

Replying to @raldi 🎙

What I don't get is why they wouldn't just use MongoDB. MongoDB is web-scale.

Reply


@anshumankmr 18 days

Replying to @raldi 🎙

What is this exactly?

Reply


@SilasX 18 days

Replying to @raldi 🎙

Wow, kind of funny that Xtranormal now lives on in the few viral videos that were made with it.

Here's where the company is now (the original domain is used for something else now):

https://en.wikipedia.org/wiki/Nawmal

Reply


@raldi 18 days

Replying to @raldi 🗣

Background: https://rachelbythebay.com/w/2021/10/30/5tb/

This video was hugely influential on changing the way Google does internal tools and operations.

Reply


@treebog 18 days

Replying to @raldi 🎙

What does “serve 5TB” refer to? They expect 5TB of network bandwidth over some time period (a month?)? Or their database takes up 5TB on disk?

Reply


@shoeshoeshoey 18 days

Replying to @raldi 🎙

Facebook had its own meme: "Pusher I need a hotfix"

Reply


@adminscoffee 17 days

Replying to @raldi 🎙

it's funny how you have to have a dictionary in order to understand how to navigate google's infrastructure

Reply


@w0mbat 18 days

Replying to @raldi 🎙

When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable. Eventually it all got rewritten by other people to do things properly like the video says.

Reply


@kgin 18 days

Replying to @raldi 🎙

Only if you think your users are scum. Do you think your users are scum? Do you? Why do you hate your users?

Reply


@jbverschoor 18 days

Replying to @raldi 🎙

omg so toxic

Reply


@jamestimmins 18 days

Replying to @raldi 🎙

As a friend of mine explained why she left Google a year or so ago, "I got tired of emailing 30 people to try to figure out who owned a single variable."

Reply


@taldo 18 days

Replying to @raldi 🎙

Ah, the laughs (Xoogler since 2020). It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.

Reply


@mikewarot 18 days

Replying to @raldi 🎙

I didn't realize this was made by Google in the first place when I saw it a few days ago. I hope things are simpler now, but I doubt it.

Reply


@smartician 18 days

Replying to @raldi 🎙

It's not much different today. Nowadays you'll also need privacy review, accessibility review, security review, and diversity & inclusion review.

Reply


@jedberg 18 days

Replying to @raldi 🎙

The same conversation at Netflix 10 years ago:

I want to serve 5TB of data.

Ok, spin up an instance in AWS and put it there.

I want it production ready.

Ok, replicate it to a second instance. If it breaks we'll page you to fix it.

The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.

Reply


@CephalopodMD 18 days

Replying to @raldi 🎙

Broccoli man is incredible. I hope someone leaks the rest...

Reply


About Us

site design / logo © 2021 Box Piper