Was this video ever used by Google for anything outside of internal before now? I swear I’ve seen this before (it’s still hilarious!) but I never worked at Google.Reply
Eagerly awaiting the "GoogFellas" video leak.Reply
And now, in 2021, Google has inflicted their "clarity" on the rest of the world. I miss jobs from the 2000s, the jobs where you were paid to write software for a living.
You know, engineering! Given a task, or set of requirements, develop software on your computer, software which eventually runs on the customer's computer, where it's used to solve the customer's problem.
My most recent full time employment a year ago was at a great company. Healthy culture, some of the most talented coworkers I've ever had the pleasure to work along side.
Over the year I lasted there, I used for the first time: Docker, Golang, Kubernetes, Terraform, Gitlab, Saltstack, Prometheus, (and probably other middleware that my brain has GCd to free space). I was barely able to get anything done. At least, it always felt that way.
Maybe I'm just an idiot, I don't know. I'd accept it if true! What I do know is that I used to be able to build things for people, be compensated well for it, and get satisfaction from a customer liking what I've built. It was simple.
In this brave new world, with containers, pods and this and that and the other thing, where it can take months before one even understands enough primitives to do a "hello world".... how can anything ever get done?? How can anything inventive, creative, or experimental emerge from our industry when the develop/test/improve cycle has gone from minutes to weeks or months?
I don't know what the future looks like, but the present strikes me as unsustainable in the long run.
(edit: Wow! I expected this to be downvoted to oblivion, not my highest rated comment on the site...)
<tiny>(Forgive this shameless self promotion: if, dear reader-who-is-a-hiring-manager, you have a paid role for a lowly but experienced systems engineer who doesn't know anything about "web" or "apps" or "social" but is quite adroit with C/C++ (and a few others), most "sciencey/mathy" type problems, signal processing, firmware, network protocols, automation/scripting, and more, ... email is welcome!)</>Reply
Non-Googler: What do all those words mean?
Noogler: Haha, this video is so funny!
L4 SWE: (Crying because the video is so true)
L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.
L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...Reply
The borgman readability approvers makes me chuckle.
At Stripe, there were language approvers. Only those blessed could approve PRs. Even XML had a set of approvers. I had fun time getting hold of an XML approver.Reply
10 years ago, on my first week at Google, my manager suggested I watch this video to get a fairly accurate idea of what the team does :-)Reply
I actually have Borgmon readability! Peer bonus pls.Reply
Wow I saw this somewhere a long time ago. But I don't remember where and in what context.Reply
meanwhile google docs is still a slow piece of shit.Reply
There's almost some kind of irony on uploading that to youtube, a feeling of "why can't I deploy my service as easily as people can upload videos to youtube?".Reply
Actively working towards being a Xoogler so I don't have to live in this dystopia.Reply
At 2:05 the green dude asks if you think your users are scum and do you hate them.
The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.Reply
BitTorrent existed since 2001. Get on with the times.Reply
Google: The Sunk Cost FallacyReply
I’m so confused, isn’t this just like basic highly available infrastructure mixed with a toxic SRE culture?
I want to serve 5TB!
Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.
And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.
Are people not used to what it really takes to “just run some code?”Reply
As a Googler, it's often easier for me to setup a GCP consumer account, AWS, or Heroku account to demo something, compared to using anything internal. I remember the most annoying situation was like 10 years ago when me and other engineers ported Quake 2 to run in Chrome, we were in a time crunch to demo it multiplayer, and I ended up setting up an AWS account to serve it. But then I left it running and forgot and ended up getting a few hundred dollars billed to me because the Quake2 server was chewing CPU.Reply
What is "Borgmon readability" and why was it important. I think that's one of the punch line of the video.Reply
Personally I'm still waiting for a copy of "Pusher, I have a hotfix."Reply
Wow, way to trigger my Google PTSD. :-)
I think the only thing I would have added would be using some service that recently was changed so that someone could get promoted but does the same thing the old one did but with different bugs.Reply
Hey... those of us that worked on Google's internal Bigtable service worked very hard so you didn't have a file to a ticket to set up replication between your Bigtable cells.
The rest does seem about accurate though.Reply
What exactly are these "peer bonuses"? Is it real? Is it what I think it is? Do people actually use them as bargain chips?Reply
Thank you, whomever did this! I asked for it in a comment recently.
This video basically is making fun of a common situation of Google at the time, where a person wants to serve up some data for analytics, but the sysadmins expect the person to follow a process intended for much more complex and high availability services run by teams of skilled engineers.
It parodies SRE as a BOFH sysadmin, even though in general SRE are quite easygoing and helpful.
It helped poked fun at a number of overly stuffy processes and also helped push people to make hosting modest datasets (like this 5TB one) easier.Reply
> It was a site where you could type in a script and it would do text-to-speech and actually > animated some goofy characters to lip-sync the words for you
Does anyone still have the code for generating "Fox girl and Broccoli man" videos? I'm thinking of starting a small revival of this meme.Reply
One of the few things I miss about my time there...
Never did get Java readability :(Reply
The multiple repetitions of “This is Google” hit home for me. I never worked as a software engineer so much of the rest is out of scope to my experience, but the constant idolization of Google, and by proxy each other for working at such a place, eventually changed from feeling coy to cultish.Reply
I wish I would have had this video before 2010. I got paged at night every time there was a PCR failover, and I didn’t know what to do with it. This video is better than all the extensive documentation that we had.Reply
Ah, this takes me back (disclaimer: Xooger, 2010-2017). It's painful and funny because it's true. Or was true.
Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.Reply
Ah, as a xoogler, I can finally enjoy this video without tears :)Reply
I'm so glad I can see this again. I forgot how much I missed this.Reply
Move fast and break things! And while you are at it, please, don't break anything.Reply
This monotonous speech synthesis is annoying to listen to. The delivery of the jokes is awful. Who can sit through a 3 min video like that?Reply
this is making me miss memegen
google had its downs, but wasting hours on memegen was not one of themReply
I have the t-shirt!Reply
As an external user who has found Google's services to be incomprehensible, it's nice to know it is (was) equally as painful internally.Reply
These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.Reply
What I don't get is why they wouldn't just use MongoDB. MongoDB is web-scale.Reply
What is this exactly?Reply
What does “serve 5TB” refer to? They expect 5TB of network bandwidth over some time period (a month?)? Or their database takes up 5TB on disk?Reply
Facebook had its own meme: "Pusher I need a hotfix"Reply
it's funny how you have to have a dictionary in order to understand how to navigate google's infrastructureReply
When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable. Eventually it all got rewritten by other people to do things properly like the video says.Reply
Only if you think your users are scum. Do you think your users are scum? Do you? Why do you hate your users?Reply
omg so toxicReply
As a friend of mine explained why she left Google a year or so ago, "I got tired of emailing 30 people to try to figure out who owned a single variable."Reply
Ah, the laughs (Xoogler since 2020). It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.Reply
I didn't realize this was made by Google in the first place when I saw it a few days ago. I hope things are simpler now, but I doubt it.Reply
It's not much different today. Nowadays you'll also need privacy review, accessibility review, security review, and diversity & inclusion review.Reply
The same conversation at Netflix 10 years ago:
I want to serve 5TB of data.
Ok, spin up an instance in AWS and put it there.
I want it production ready.
Ok, replicate it to a second instance. If it breaks we'll page you to fix it.
The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.Reply
Broccoli man is incredible. I hope someone leaks the rest...Reply