There's a decent chance that this is also just an issue within a JSON parsing library that someone at R* decided to throw in and use. The parsing library was probably never intended to handle such large blobs, and R* didn't want to mess with the the OSS code.
I wonder if it might even be an incompatible OSS license?Reply
Red Dead redemption 2 online also does this exact same thing the first time it did it I shut the game off and then charged back my credit card I dealt with it for GTA and I'm just not going to deal with it anymore.Reply
Its a bit disheartening to think about the person-centuries of time wasted on what wouldn't be unfair to characterize as sloppy code.Reply
He updated his article and added an option to donate him. Check the bottom of his website! ;)Reply
Face palm. Thanks for doing this, maybe Rockstar will finally fix this bullshit.Reply
The part that puzzles me the most was this comment about sscanf:
> To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this.
Is this true? Is sscanf really O(N) on the size of the string? Why does it need to call strlen in the first place?Reply
I remember when someone explained here eloquently and inticately explained why Twitter constantly breaks in various ways, such at saying I'm not allowed to see a tweet, but then I merely hit reload and it works... easy fix posted around a year ago at least but still not implemented. Don't hold your breath.Reply
That was a really, really fun read.Reply
Really nice hacking; well done.
I wonder who t0st is. He lives in Moscow according to Twitter. This type of work is usually done to find vulnerability in code e.g. buffer overflows. Injecting a piece of code to program machine code is a technique used in system hacking or software piracy. I can bet t0st is working or has worked for FSB. Anyway, not that is matters... it is still fun to think about itReply
This is some first rate debugging, and great writing to boot. I hope Rockstar sees this, fixes the bug and then pays this fella something. Great post, thanks for sharing!Reply
> Normally Luke would group the same functions together but since I don’t have debugging symbols I had to eyeball nearby addresses to guess if it’s the same place.
I really enjoyed this article, but I do have a question about this part. Why would a single function be listed at mutliple addresses?Reply
I'm going to make the valid assumption that someone, at some point, was assigned to fix this. The issue here, IMHO, is why they didn't. It comes down to how complicated it was to fix this from the organizational perspective, and not from the technical perspective. I'll explain.
First, a disclaimer: I have no idea how much Rockstar employees are paid, nor how their workdays look like. I don't know what their team sizes are, where they worked before, or who manages who and in what fashion. I actually don't know anything about the Rockstar engineering organization at all.
I am also not a GTA player (or, more accurately, haven't been since San Andreas came out many moons ago). This is my perspective as someone who has worked for various organizations in his life ( SWE-centered ones but also other, more "traditional" ones too).
We're all familiar with technical debt - it's a well established concept by now. Reducing it is part of the normal workload for well-functioning organizations, and (good) tech leads think about it often.
What isn't talked about as often is the "organizational" debt. Some things are "verboten" - you don't touch them, you don't talk about them, and you run like hell if you're ever assigned to deal with them.
Every large enough company has (at least) one of those. It might be a critical service written in a language nobody in the team knows anymore. Maybe it's a central piece of infra that somebody wrote and then left, and it's seen patch after patch ever since. These are things that end your career at the company if the BLAME points to you after you attempted to fix them.
I have a gut feeling - not based on anything concrete, as I mentioned - that the loading part for GTA Online might be one of those things. If someone breaks the process that loads the game - that's a big deal. No one would be able to play. Money would be lost, and business-folk will come knocking.
So sure, there might be some mitigations in place - if some part fails to load they allow the game to load anyways, and then attempt to fix it mid-fly. It's not black and white. But it feels like one of those things, and so people might have just been running like hell from it for years and years. Teams change. Projects change hands. People leave, more people join. It's life in the industry.
I would be REALLY interested in learning how software orgs deal with these types of behemoths in their projects. I have yet to find someone who knows how to - methodically and repetitively - break these apart when they appear.Reply
Doesn't surprise me at all. It's an O(n^2) algorithm (strlen called in a loop) in a part of the code where N is likely much smaller in the test environment (in-app purchases).
Overwatch is another an incredibly popular game with obvious bugs (the matchmaking time) front and center. And gamers are quick to excuse it as some sort of incredibly sophisticated matchmaking - just like the gamers mentioned in OP.
It's easy to to say it's something about gamers / gaming / fandom - but I have a locked down laptop issued by my bigcorp which is unbelievably slow. I'd bet a dollar there's a bug in the enterprise management software that spins the CPU. A lot of software just sucks and people use it anyway.Reply
Reading things like these is bittersweet. One one hand, I am glad to see that the art of figuring out “why is this thing slow” is still alive and well, even in the face of pushback from multiple fronts. On the other hand, it’s clear that the bar is continually rising for people who are looking to do this as a result of that pushback. Software has always had a bottleneck of the portion of the code written by the person with the least knowledge or worst priorities, but the ability to actually work around this as an end user has gotten harder and harder.
The first hurdle is the technical skill required: there has always been closed source software, but these days the software is so much more complex, often even obfuscated, that the level of knowledge necessary to diagnose an issue and fix it has gone up significantly. It used to be that you could hold and entire program in your head and fix it by patching a couple bytes, but these days things have many, many libraries and you may have to do patching at much stranger boundaries (“function level but when the arguments are these values and the call stack is this”). And that’s not to say anything of increasing codesigning/DRM schemes that raise the bar for this kind of thing anyways.
The other thing I’ve been seeing is that the separation between the perceived skills of software authors and software users has increased, which I think has discouraged people from trying to make sense of the systems they use. Software authors are generally large, well funded teams, and together they can make “formidable” programs, which leads many to forget that code is written by individuals who write bugs like individuals. So even when you put in the work to find the bug there will be people who refuse to believe you know what you are doing on account of “how could you possibly know better than $GIANT_CORPORATION”.
If you’re looking for ways to improve this, as a user you should strive to understand how the things you use work and how you might respond to it not working–in my experience this is a perpetually undervalued skill. As a software author you should look to make your software introspectable, be that providing debug symbols or instructions on how users can diagnose issues. And from both sides, more debugging tools and stories like this one :)Reply
I played through GTA V, enjoyed it, and tried out the online mode afterward.
I've logged in exactly twice. Load times like that may be worth it to a hardcore gamer, but I have no patience for it. There's no shortage of entertainment available to someone with a PC, a reasonable internet connection, and a modicum of disposable income. Waste my time and I'll go elsewhere for my entertainment.Reply
I am absolutely shocked about this finding. The amount of money on microtransactions Rockstar lost because of this single issue must be gigantic. The amount of people that got turned off by the loading times over the years is massive. It's mind boggling.Reply
I always cringe when I come across any kind of loop with unnecessary polynomial or even exponential complexity, even when it is found in a non-critical part of the code.
It offends my taste as a software engineer and I always fix these.Reply
I think the developers knew it was slow, but they assumed that it was because 'parsing JSON is slow and we can't do anything about it'. Sometimes you're just too sure what's wrong.Reply
How many bets they'll leave it like this on the current version and fix it for the PS5 version, to show how dramatic of a change it is between consoles?Reply
Even after cutting loading by 70% it still take a minute? I haven't played any AAA titles for a long time. But even 30s is way too long. Especially I used to play with HDD. Considering modern SSD can be 30x Faster in Seq Read and Random Read up to 200x.
Is 1 min loading time even normal? Why did it take so long? I never played GTA Online so could someone explain?Reply
This is awesome! Great write up. Just loved it. Thank you :)Reply
woo open sourceReply
Incredible work. Hopefully R* acknowledge this and compensate you in some way. I won’t be holding my breath.
Maybe set up a donation page? I’d be more than happy to send some beer money your way for your time spent on this!Reply
I hate to be the guy but I have to. I don't know how to accomplish this but if the part in question was open source someone would have noticed.Reply
Spicy hot take: the root cause here is the awful c++ library ecosystem.Reply
Red Dead Redemption 2 introduced a bug where the exe would reduce the volumne on start to 19%.
Its now at least 4 month old.
So whats the problem? When you tab+alt out of fullscreen, to change your soundvolumen everytime you start the game, you have to rechange the graphics configuration as well.
I solved it by .net code i found on github which would run fur 5 minutes and would put the volumne up as soon as it ifinds the rdd2 process...Reply
Fantastic research and write-upReply
> the problems shouldn’t take more than a day for a single dev to solve
It bothers me that so many of us developers are annoyed by managers saying stupid stuff like "shouldn't be too hard" about things they don't understand, but then other developers walk into the same trap.
Yes, it looks simple at the surface. It most likely isn't. The problem is not that it looks simple, the problem is that you assume it is simple, probably because you don't have the full context.Reply
I'm a complete noob at all this, si there an easy way to try this out at all? This is an awesome find!Reply
Developers probably knew about this, but executives didn't care when getting int monetary calculations.Reply
I'm a zero to no knowledge on this in terms of implementation, is there like an easy execute at all?Reply
Wow. I always assumed that profiling would be part of the pre-release test processes for AAA games...Reply
Let me guess, when they started this JSON was small and this parsing code worked fine for that small input but over time the JSON got bigger and bigger and here we are.Reply
Wow, many people argue how optimized GTA was and then this. I wonder how much money they lost because of this. I often stopped playing because it just took too long to load.Reply
Why in the world would you roll your own JSON parser?
For reference, I just ran a 10MB file through the JSON parser I use in Python and it finished in 0.047s (vs 190s)Reply
I love reading about this type of hacking/debugging, where they disassemble the binary and patch it.
Can anyone recommend a youtube video where I can watch (not necessarily learn) people doing this sort of work? I'm in the mood for some vicarious hacking :-)Reply
This article is great. As a developer, I've always been curious about what the heck GTAO was doing that could take so long to loadReply
Maybe someone already asked(but the comments are huge). Is there anyway to also trim down the JSON file? Either on disk, or when it is fetched from remote? If you don't care about transactions then you could remove it completely.Reply
Such silly mistakes causing such a huge delay for millions of gamers over 6 years.Reply
Wow this is absolutely fantastic investigation! I learned alot! Thank you.Reply
I gave up on playing GTA:O because everything took so long to load, having never spent a dime. I have to imagine there is so much lost revenue because of this bug; I hate to be harsh but it is truly an embarrassment that someone external to R* debugged and fixed this before they did (given they had 6 years!).Reply
I have a friend who works at Rockstar. I have forwarded the blog post to him.Reply
I once came across a game that rendered progress bar frames at fixed points during the loading process. With VSync enabled this would significantly slow things down compared to what a fast computer could manage otherwise.Reply
Hey T0ST, thanks a mill for releasing this! I've reduced my online loading time from 120s to 40s thanks to this! Since I have the epic version, I used BigBaseV2-fix where datlimabean04 has adapted (and dare I say slightly improved?) your code. If you like, you could also link to the repo where he has integrated that: https://bitbucket.org/gir489/bigbasev2-fix/commits/branch/lo...
Also the post where he mentions adding the branch with your fix is here: https://www.unknowncheats.me/forum/3078353-post937.htmlReply
Hopefully they completely ignored this and other issues in GTA5 online because they have been working on GTA6 all these years /end sarcasm
But honestly, on a large enough team with a large enough backlog much worse things slip through the cracks. Doesn't excuse this at all though.
Also kudos on the write up! Enjoyed reading itReply
One of the most entertaining post I read lately. Congratulations for the job, and the clarity of the explanations! Reading the story, everything looks simple where it sure is not. Lot to learn inside, thanks to the author for that.Reply
Please don't compute speed-up like that.
Old time: 6 minutes (360 seconds). New time: 1 minute and 50 seconds (110 seconds). Speed-up calculated by article author: 1-110/360=0.694 (69.4% saved). Speed-up calculated by me: 360/110=3.27 (3 times faster).
Please calculate it the other way around. It makes great difference when you say you made something 10× faster than when you say you saved 90% of work even if both mean exactly the same thing.
Bruce Dawson has great article about this: https://randomascii.wordpress.com/2018/02/04/what-we-talk-ab...Reply
I haven’t played GTAO in a couple years, but I vaguely remember using the suspend process option of windows task manager to jump into online much faster.
Googling gave me this result, which sounds about right for what I remember. https://www.gfinityesports.com/grand-theft-auto/gta-online-h...
I’m not certain it’s still a valid workaround, and it’s not nearly as sophisticated as the OP method, but at least everyone can do it :)Reply
This is really cool - how did you develop the background knowledge to solve this? I'm trying to learn more about low-level stuff and I would have no idea how to approach solving a problem like thisReply
* There’s a single thread CPU bottleneck while starting up GTA Online
* It turns out GTA struggles to parse a 10MB JSON file
* The JSON parser itself is poorly built / naive and
* After parsing there’s a slow item de-duplication routine
(Copied from the END of the article)Reply
I find it interesting that, with ~63k entries, they didn't just use a bloom filter to do a lookup on if the entity they're working with has already been seen. granted, they stoll need to store the data, but I think a bloom filter would be a more effective way to test if the item exists alreadyReply
"They’re parsing JSON. A whopping 10 megabytes worth of JSON with some 63k item entries."
Ahh. Modern software rocks.Reply
this is amazing. good work! unbelievableReply
The author's frustration with GTA Online reminds me of my own experience with Borderlands 2 taking over 1 minute for its "searching for downloadable content" loading phase. I hate DLC, so having to wait for it was especially aggravating.
I did not ever consider inspecting obfuscated assembly and hooking functions to fix the problem myself. Very impressive work!Reply
I played the game on PS3 and PC. The loading time at launch for PS3 (and later for PC, albeit on SSD) wasn't great, but it also wasn't nearly this terrible.
From a game programming perspective, I'm sure at launch there were very few extras to obtain, so this method ran fast and didn't raise any red flags.
But as time has worn on they've added a ton of extra items and it's become a real problem. What it does show is that probably most of their team are working on creating new things they can sell, vs testing and maintaining the codebase for the last N years.Reply
Super impressive. I am generally not impressed with community hacks because, as every comment thread discusses, they are generally hacks that do not support all players and thus can’t be shipped.
But this seems pretty damn clean. And it’s egregious that no one at R* took a week to investigate and fix “correctly”.Reply
This is such an awesome article, and I love it. Thanks to the author for the great digging and explanation.Reply
Micro transactions were probably added in late stages of development by someone pressured to get the job done quickly and the developers were savvy enough to believe that the added overhead to slow initial loading was standard fare that users already accepted. Not enough time to optimize, forced to ship, and good enough.Reply
I bet if the devs did more leetcode this would have never happened /sReply
Also note that the way he fixed it, strlen only caches the last call and returns quickly on an immediate second call with the same string.
Another reason why C styled null terminated strings suck. Use a class or structure and store both the string pointer and its length.
I have seen other programs where strlen was gobbling up 95% of execution time.Reply
Yep, O(n^2) has the problem that no matter how fast you upgrade your hardware it would still lag.
Another pet peeve of mine is Civ 6's loading time for a saved game is atrocious. I'm sure there's a O(n^2) loop in there somewhere.Reply
Maybe even more surprising to me is that sscanf() relies on strlen().
I would have expected libc to take that use case in consideration and use a different algorithm when the string exceeds a certain size. Even if the GTA parser is not optimal, I would blame libc here. The worst part is that some machines may have an optimized libc and others don't, making the problem apparent only in some configuration.
I believe standard libraries should always have a reasonable worst case by default. It doesn't have to be perfectly optimized, but I think it is important to have the best reasonable complexity class, to avoid these kinds of problems. The best implementations usually have several algorithms for different cases. For example, a sort function may do insertion (n^2, good for small n) -> quicksort (avg. nlog(n), worst n^2, good overall) -> heapsort (guaranteed nlog(n), slower than quicksort except in edge cases). This way, you will never hit n^2 but not at the cost of slow algorithm for the most common cases.
The pseudo hash table is all GTA devs fault though.Reply
This is why people should use commonly-available packages instead of rolling their own version of whatever dumb algorithm they think they can write. This happens all the time. Bugs have been fixed by others, but everyone is too smart to use someone else’s code.Reply
It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.
I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.
This online gamemode alone made $1 billion in 2017 alone.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.
The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.
(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)Reply
Well, that's embarrassing. I can't even imagine the level of shame I would feel if I had written the offending code.
But, you know, premature optimization yadda yadda.Reply
As others have pointed out, this is a good illustration of why you need accurate data during development. I bet that the data set used during development was dirty with duplicates and way too small. It was faster and more convenient to just code defensive garbage than to be the annoying one nagging to another team about the state of data. So this was written, probably along with a TODO comment, and then forgotten about, and ultimately shipped. I've done this same thing. Not with any real consequences, but still. It's what happens when time is of the essence.
How it remained undetected for so long is really weird though. Surely they must've had a massive amount of complaints about the loading times. I completely stopped playing because of them. How could they not have investigated the root cause, atleast once, in six years?Reply
Holy cow, I'm a very casual gamer, I was excited about the game but when it came out I decided I don't want to wait that long and I'll wait until they sort it out. 2 years later it still sucked. So I abandoned it. But.. this... ?! This is unbelievable. I'm certain that many people left this game because of the waiting time. Then there are man-years wasted (in a way different than desired).
Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020.
Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.Reply
Really great work. Here's to hoping someone at Rockstar notices. You didn't even need access to the original sourcecode to debug it! Quite impressive my friend. Thanks for the great write-up!Reply
Honestly, while this horrible code is mildly offensive to me, I'm pretty impressed by this person's persistence. It's one thing to identify a bug in a compiled program, but it's another to patch it without fully understanding what's going on. Caching strlen was a particularly clever trick that sidestepped a bunch of more complicated solutions.Reply
BTC mining is using a significant amount of global energy production every year now. The more cynical might say that ultimately it's a waste of human effort, but I have to say that this bug (and it's most definitely a bug) should be patched (and should've been patched) to save the kilowatts of power wasted parsing mostly the same data.Reply
A lot of comments here kind of brush off the fact that this has been going on for _years_! I would expect quadratic behaviour to slip here and there in non real time code places. We make mistakes after all.
I've had to revisit production code from time to time to discover my or my colleague assumptions about library functions were not actually true.
But for someone in the possession of the source code this looks like _minutes_ worth of profiling time! With a stellar technical accomplishment that GTA5 truly is, you would expect tons of really talented devs that could have traced this one down on their lunch break.
The fact that they didn't speaks volumes about organisational structure within R*. I think some higher up managers _really_ need some stern talking to!Reply
Excellent investigation, and an elegant solution.
There's a "but" though: you might end up getting banned from GTA Online altogether if this DLL injection is detected by some anti-cheat routine. The developers should really fix it on their end.Reply
There’s one thing that made me curious. The author said the load times when the game came out were awful as well.
So has that JSON always been 10mb or has it grown a lot over time?
Has the load time crept up since launch or has it gone down?Reply
think about it, how much electricity wasted
millions of players, billions of minutes wasted
when you scale things, every bit of optimizations IS MENDATORY!
Love performance issues like this, good read.Reply
This is quite impressive though it is important to remember that this came out for machines like the Xbox360 and the PlayStation 3. I am not convinced these modifications would “fit” on those machines without significant thrashing/paging and significant memory usage to the point where other negative effects come into play. Great read and effort regardless.Reply
I loved so much reading it, I was thinking that if someone were to write fictional, sherlock holmes like fantasy story where our sherlock would take some (maybe fictional) widely used software at each episode, investigate it like this, and reveal some (fictional) grand bug in the end- I'd totally read it.
Yeah I know it sounds stupid, but I suspect real Sherlock Holmes was inspired by a true story like this one too, and at least some contemporary detectives started to enjoy reading them.Reply
The Github link is dead, DMCA or incorrect link?Reply
It doesn't explain some things to me. Maybe I read the article inattentively, but which stage of the loading hangs for a very long time? Loading screen, or city overview when joining a session? In my experience, on modern hardware, the loading screen goes very quickly (a minute or two).
What's worth noting is that the city overview can hang for 20-30 minutes during connecting. At the same time, suspending the GTAV.exe for 20-30 seconds allows you to immediately throw the player into an empty public session. It is unlikely caused by slow parsing of json, more like the suspend causes udp timeout when connecting to the slow session hosts.Reply
I wonder how long Rockstar will take to respond/fix.Reply
not surprising - the game industry is absolutely notorious for cutting corners. didn't know they cut corners this much though.
will r* fix it? maybe, especially since some person literally did half of the work for them. But given R* is a large company, this probably wont be fixed for a long time, and GTAO is probably being maintained by the lowest bid contractor group.
They probably have all of their full time assets working on the next iteration of GTA.Reply
Imagine all the time that people are wasting if they are sitting there waiting for it to load. I wonder how many lifespans it adds up to?Reply
I bought a PlayStation a few years go to play GTA..
Never tried consoles before, but figured the point was that it would be fast and hassle free.
GTA was fun, but after completing it I never load it, just to play for fun, because PS load times + GTA load times are too much.
Now I just play 20 year old games under wine on Linux.. that's less of a hassle, and I don't have to touch windows.Reply
Thank god. I always suspected that those loading times were cause by some retarded implementation detail. GTA5 is not that complex to justify that kind of loading times. Even the hardware has scaled massively since their launch and it doesn't even matter.Reply
I can't believe what Rockstar did here. I and 3 friends stop playing GTA V cause of the unbelievable stupid loading times.Reply
This is my favorite read on HN, ever. So cool!Reply
Parsing text is always expensive on the CPU, that's why it's better to often prefer binary files when it's possible.
It's also the reason the DOM is so slow, in my view.
I remember spotting a lot of fast JSON parsers around, but again, there doesn't seem to be any popular, open, flexible, binary file format out there.
Meanwhile, games are always larger and larger, machine learning requires more and more data.
There is ALWAYS money, battery and silicon to be saved by improving performance.Reply
I have absolute respect for people who are able to do that kind of things. A few years back I've tried to play with reverse engineering but I never managed to get the skills.Reply
Very cool analysis. Its unbelievable that this is still unfixed. I don't play GTA anymore mostly because of the load times for every single action.Reply
This is why I come to HN, I was going to skip this because I thought it was about video games, but really glad to have read it, and loved every line of the article.
So much to get from this.
Even if you don't have the source, you can make a change if you are annoyed enough.
If you don't like something, and the source code is out there, really go contribute.
Performance matters, know how to profile and if using an external dependency, then figure out their implementation details.
Algorithms & Data structures matter, often I see devs talking about how it doesn't matter much but the difference between using a hashmap vs array is evident.
Attentive code reviews matter, chances are they gave this to a junior dev/intern, and it worked with a small dataset and no one noticed.Reply
Hi this is Roman from Rockstar.Reply
This should be tweeted to their official channel or someone inside company (on twitter) who has ability let relevant Rockstar dev know about it...Reply
I bet this feature, relating to in game purchasing options, was retro-fitted by some non-core dev team who delivered it to spec and fixed costReply
Pretty strange that nobody at Rockstar bothered to run the games loading phase in a profiler, this problem would have been super easy to observe if they had.Reply
Hey OP, what would it take to persuade you to have a look at Hearthstone next?Reply
[joke]so, they say that c++ is faster than java[/joke]Reply
I like this page design. It looks familiar. Does anyone know where this design comes from? Or if it's original?Reply
scary to think of the carbon emissions this bug has generated over the years.Reply
GTA:O shows advertisements for in-game purchases on the loading screen. How many advertisements you see is a function of how long the loading screen takes.
Something tells me this "problem" was discovered long ago at Rockstar HQ, and quietly deemed not to be a problem worth solving.Reply
Someone hire this person!Reply