Hacker News Re-Imagined

How I cut GTA Online loading times by 70%

  • 3883 points
  • 8 months ago

  • @kuroguro
  • Created a post

How I cut GTA Online loading times by 70%


@Parsnip1 8 months

Replying to @kuroguro 🎙

There's a decent chance that this is also just an issue within a JSON parsing library that someone at R* decided to throw in and use. The parsing library was probably never intended to handle such large blobs, and R* didn't want to mess with the the OSS code.

I wonder if it might even be an incompatible OSS license?

Reply


@lumberingjack 8 months

Replying to @kuroguro 🎙

Red Dead redemption 2 online also does this exact same thing the first time it did it I shut the game off and then charged back my credit card I dealt with it for GTA and I'm just not going to deal with it anymore.

Reply


@VectorLock 8 months

Replying to @kuroguro 🎙

Its a bit disheartening to think about the person-centuries of time wasted on what wouldn't be unfair to characterize as sloppy code.

Reply


@lidotajs 8 months

Replying to @kuroguro 🎙

He updated his article and added an option to donate him. Check the bottom of his website! ;)

Reply


@johnnyAghands 8 months

Replying to @kuroguro 🎙

Face palm. Thanks for doing this, maybe Rockstar will finally fix this bullshit.

Reply


@ufo 8 months

Replying to @kuroguro 🎙

The part that puzzles me the most was this comment about sscanf:

> To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this.

Is this true? Is sscanf really O(N) on the size of the string? Why does it need to call strlen in the first place?

Reply


@daodedickinson 8 months

Replying to @kuroguro 🎙

I remember when someone explained here eloquently and inticately explained why Twitter constantly breaks in various ways, such at saying I'm not allowed to see a tweet, but then I merely hit reload and it works... easy fix posted around a year ago at least but still not implemented. Don't hold your breath.

Reply


@lionhearted 8 months

Replying to @kuroguro 🎙

That was a really, really fun read.

Reply


@mkuklik 8 months

Replying to @kuroguro 🎙

Really nice hacking; well done.

I wonder who t0st is. He lives in Moscow according to Twitter. This type of work is usually done to find vulnerability in code e.g. buffer overflows. Injecting a piece of code to program machine code is a technique used in system hacking or software piracy. I can bet t0st is working or has worked for FSB. Anyway, not that is matters... it is still fun to think about it

Reply


@mstade 8 months

Replying to @kuroguro 🎙

This is some first rate debugging, and great writing to boot. I hope Rockstar sees this, fixes the bug and then pays this fella something. Great post, thanks for sharing!

Reply


@petros786 8 months

Replying to @kuroguro 🎙

Brilliant...

Reply


@JoshMcguigan 8 months

Replying to @kuroguro 🎙

> Normally Luke would group the same functions together but since I don’t have debugging symbols I had to eyeball nearby addresses to guess if it’s the same place.

I really enjoyed this article, but I do have a question about this part. Why would a single function be listed at mutliple addresses?

Reply


@tomgs 8 months

Replying to @kuroguro 🎙

I'm going to make the valid assumption that someone, at some point, was assigned to fix this. The issue here, IMHO, is why they didn't. It comes down to how complicated it was to fix this from the organizational perspective, and not from the technical perspective. I'll explain.

First, a disclaimer: I have no idea how much Rockstar employees are paid, nor how their workdays look like. I don't know what their team sizes are, where they worked before, or who manages who and in what fashion. I actually don't know anything about the Rockstar engineering organization at all.

I am also not a GTA player (or, more accurately, haven't been since San Andreas came out many moons ago). This is my perspective as someone who has worked for various organizations in his life ( SWE-centered ones but also other, more "traditional" ones too).

We're all familiar with technical debt - it's a well established concept by now. Reducing it is part of the normal workload for well-functioning organizations, and (good) tech leads think about it often.

What isn't talked about as often is the "organizational" debt. Some things are "verboten" - you don't touch them, you don't talk about them, and you run like hell if you're ever assigned to deal with them.

Every large enough company has (at least) one of those. It might be a critical service written in a language nobody in the team knows anymore. Maybe it's a central piece of infra that somebody wrote and then left, and it's seen patch after patch ever since. These are things that end your career at the company if the BLAME points to you after you attempted to fix them.

I have a gut feeling - not based on anything concrete, as I mentioned - that the loading part for GTA Online might be one of those things. If someone breaks the process that loads the game - that's a big deal. No one would be able to play. Money would be lost, and business-folk will come knocking.

So sure, there might be some mitigations in place - if some part fails to load they allow the game to load anyways, and then attempt to fix it mid-fly. It's not black and white. But it feels like one of those things, and so people might have just been running like hell from it for years and years. Teams change. Projects change hands. People leave, more people join. It's life in the industry.

I would be REALLY interested in learning how software orgs deal with these types of behemoths in their projects. I have yet to find someone who knows how to - methodically and repetitively - break these apart when they appear.

Reply


@will4274 8 months

Replying to @kuroguro 🎙

Doesn't surprise me at all. It's an O(n^2) algorithm (strlen called in a loop) in a part of the code where N is likely much smaller in the test environment (in-app purchases).

Overwatch is another an incredibly popular game with obvious bugs (the matchmaking time) front and center. And gamers are quick to excuse it as some sort of incredibly sophisticated matchmaking - just like the gamers mentioned in OP.

It's easy to to say it's something about gamers / gaming / fandom - but I have a locked down laptop issued by my bigcorp which is unbelievably slow. I'd bet a dollar there's a bug in the enterprise management software that spins the CPU. A lot of software just sucks and people use it anyway.

Reply


@saagarjha 8 months

Replying to @kuroguro 🎙

Reading things like these is bittersweet. One one hand, I am glad to see that the art of figuring out “why is this thing slow” is still alive and well, even in the face of pushback from multiple fronts. On the other hand, it’s clear that the bar is continually rising for people who are looking to do this as a result of that pushback. Software has always had a bottleneck of the portion of the code written by the person with the least knowledge or worst priorities, but the ability to actually work around this as an end user has gotten harder and harder.

The first hurdle is the technical skill required: there has always been closed source software, but these days the software is so much more complex, often even obfuscated, that the level of knowledge necessary to diagnose an issue and fix it has gone up significantly. It used to be that you could hold and entire program in your head and fix it by patching a couple bytes, but these days things have many, many libraries and you may have to do patching at much stranger boundaries (“function level but when the arguments are these values and the call stack is this”). And that’s not to say anything of increasing codesigning/DRM schemes that raise the bar for this kind of thing anyways.

The other thing I’ve been seeing is that the separation between the perceived skills of software authors and software users has increased, which I think has discouraged people from trying to make sense of the systems they use. Software authors are generally large, well funded teams, and together they can make “formidable” programs, which leads many to forget that code is written by individuals who write bugs like individuals. So even when you put in the work to find the bug there will be people who refuse to believe you know what you are doing on account of “how could you possibly know better than $GIANT_CORPORATION”.

If you’re looking for ways to improve this, as a user you should strive to understand how the things you use work and how you might respond to it not working–in my experience this is a perpetually undervalued skill. As a software author you should look to make your software introspectable, be that providing debug symbols or instructions on how users can diagnose issues. And from both sides, more debugging tools and stories like this one :)

Reply


@EdSchouten 8 months

Replying to @kuroguro 🎙

Both of these issues are good candidates for https://accidentallyquadratic.tumblr.com

Reply


@Zak 8 months

Replying to @kuroguro 🎙

I played through GTA V, enjoyed it, and tried out the online mode afterward.

I've logged in exactly twice. Load times like that may be worth it to a hardcore gamer, but I have no patience for it. There's no shortage of entertainment available to someone with a PC, a reasonable internet connection, and a modicum of disposable income. Waste my time and I'll go elsewhere for my entertainment.

Reply


@JyB 8 months

Replying to @kuroguro 🎙

I am absolutely shocked about this finding. The amount of money on microtransactions Rockstar lost because of this single issue must be gigantic. The amount of people that got turned off by the loading times over the years is massive. It's mind boggling.

Reply


@linuxhansl 8 months

Replying to @kuroguro 🎙

I always cringe when I come across any kind of loop with unnecessary polynomial or even exponential complexity, even when it is found in a non-critical part of the code.

It offends my taste as a software engineer and I always fix these.

Reply


@Geee 8 months

Replying to @kuroguro 🎙

I think the developers knew it was slow, but they assumed that it was because 'parsing JSON is slow and we can't do anything about it'. Sometimes you're just too sure what's wrong.

Reply


@w_for_wumbo 8 months

Replying to @kuroguro 🎙

How many bets they'll leave it like this on the current version and fix it for the PS5 version, to show how dramatic of a change it is between consoles?

Reply


@ksec 8 months

Replying to @kuroguro 🎙

Even after cutting loading by 70% it still take a minute? I haven't played any AAA titles for a long time. But even 30s is way too long. Especially I used to play with HDD. Considering modern SSD can be 30x Faster in Seq Read and Random Read up to 200x.

Is 1 min loading time even normal? Why did it take so long? I never played GTA Online so could someone explain?

Reply


@fl0tingh0st 8 months

Replying to @kuroguro 🎙

This is awesome! Great write up. Just loved it. Thank you :)

Reply


@awinter-py 8 months

Replying to @kuroguro 🎙

woo open source

Reply


@wun0ne 8 months

Replying to @kuroguro 🎙

Incredible work. Hopefully R* acknowledge this and compensate you in some way. I won’t be holding my breath.

Maybe set up a donation page? I’d be more than happy to send some beer money your way for your time spent on this!

Reply


@noobermin 8 months

Replying to @kuroguro 🎙

I hate to be the guy but I have to. I don't know how to accomplish this but if the part in question was open source someone would have noticed.

Reply


@raggi 8 months

Replying to @kuroguro 🎙

Spicy hot take: the root cause here is the awful c++ library ecosystem.

Reply


@Blumkratz 8 months

Replying to @kuroguro 🎙

Red Dead Redemption 2 introduced a bug where the exe would reduce the volumne on start to 19%.

Its now at least 4 month old.

So whats the problem? When you tab+alt out of fullscreen, to change your soundvolumen everytime you start the game, you have to rechange the graphics configuration as well.

I solved it by .net code i found on github which would run fur 5 minutes and would put the volumne up as soon as it ifinds the rdd2 process...

Reply


@Aissen 8 months

Replying to @kuroguro 🎙

It would make a great addition to the Accidentally Quadratic blog: https://accidentallyquadratic.tumblr.com/ (haven't been updated in ~2 years, but maybe the author is around).

Reply


@cwojno 8 months

Replying to @kuroguro 🎙

Fantastic research and write-up

Reply


@capableweb 8 months

Replying to @kuroguro 🎙

> the problems shouldn’t take more than a day for a single dev to solve

It bothers me that so many of us developers are annoyed by managers saying stupid stuff like "shouldn't be too hard" about things they don't understand, but then other developers walk into the same trap.

Yes, it looks simple at the surface. It most likely isn't. The problem is not that it looks simple, the problem is that you assume it is simple, probably because you don't have the full context.

Reply


@bedbuggy221 8 months

Replying to @kuroguro 🎙

I'm a complete noob at all this, si there an easy way to try this out at all? This is an awesome find!

Reply


@endoelenar58 8 months

Replying to @kuroguro 🎙

Developers probably knew about this, but executives didn't care when getting int monetary calculations.

Reply


@bedbuggy221 8 months

Replying to @kuroguro 🎙

I'm a zero to no knowledge on this in terms of implementation, is there like an easy execute at all?

Reply


@trollied 8 months

Replying to @kuroguro 🎙

Wow. I always assumed that profiling would be part of the pre-release test processes for AAA games...

Reply


@ankurdhama 8 months

Replying to @kuroguro 🎙

Let me guess, when they started this JSON was small and this parsing code worked fine for that small input but over time the JSON got bigger and bigger and here we are.

Reply


@faebi 8 months

Replying to @kuroguro 🎙

Wow, many people argue how optimized GTA was and then this. I wonder how much money they lost because of this. I often stopped playing because it just took too long to load.

Reply


@fastball 8 months

Replying to @kuroguro 🎙

Why in the world would you roll your own JSON parser?

For reference, I just ran a 10MB file through the JSON parser I use in Python and it finished in 0.047s (vs 190s)

Reply


@ed25519FUUU 8 months

Replying to @kuroguro 🎙

I love reading about this type of hacking/debugging, where they disassemble the binary and patch it.

Can anyone recommend a youtube video where I can watch (not necessarily learn) people doing this sort of work? I'm in the mood for some vicarious hacking :-)

Reply


@dekerta 8 months

Replying to @kuroguro 🎙

This article is great. As a developer, I've always been curious about what the heck GTAO was doing that could take so long to load

Reply


@rewsiffer 8 months

Replying to @kuroguro 🎙

Maybe someone already asked(but the comments are huge). Is there anyway to also trim down the JSON file? Either on disk, or when it is fetched from remote? If you don't care about transactions then you could remove it completely.

Reply


@sim_card_map 8 months

Replying to @kuroguro 🎙

Unbelievable.

Such silly mistakes causing such a huge delay for millions of gamers over 6 years.

Reply


@deevolution 8 months

Replying to @kuroguro 🎙

Wow this is absolutely fantastic investigation! I learned alot! Thank you.

Reply


@kevinventullo 8 months

Replying to @kuroguro 🎙

I gave up on playing GTA:O because everything took so long to load, having never spent a dime. I have to imagine there is so much lost revenue because of this bug; I hate to be harsh but it is truly an embarrassment that someone external to R* debugged and fixed this before they did (given they had 6 years!).

Reply


@josalhor 8 months

Replying to @kuroguro 🎙

I have a friend who works at Rockstar. I have forwarded the blog post to him.

Reply


@account42 8 months

Replying to @kuroguro 🎙

I once came across a game that rendered progress bar frames at fixed points during the loading process. With VSync enabled this would significantly slow things down compared to what a fast computer could manage otherwise.

Reply


@alvarezsyndrome 8 months

Replying to @kuroguro 🎙

Hey T0ST, thanks a mill for releasing this! I've reduced my online loading time from 120s to 40s thanks to this! Since I have the epic version, I used BigBaseV2-fix where datlimabean04 has adapted (and dare I say slightly improved?) your code. If you like, you could also link to the repo where he has integrated that: https://bitbucket.org/gir489/bigbasev2-fix/commits/branch/lo...

Also the post where he mentions adding the branch with your fix is here: https://www.unknowncheats.me/forum/3078353-post937.html

Reply


@dstola 8 months

Replying to @kuroguro 🎙

Hopefully they completely ignored this and other issues in GTA5 online because they have been working on GTA6 all these years /end sarcasm

But honestly, on a large enough team with a large enough backlog much worse things slip through the cracks. Doesn't excuse this at all though.

Also kudos on the write up! Enjoyed reading it

Reply


@xeyownt 8 months

Replying to @kuroguro 🎙

One of the most entertaining post I read lately. Congratulations for the job, and the clarity of the explanations! Reading the story, everything looks simple where it sure is not. Lot to learn inside, thanks to the author for that.

Reply


@MarekKnapek 8 months

Replying to @kuroguro 🎙

Please don't compute speed-up like that.

Old time: 6 minutes (360 seconds). New time: 1 minute and 50 seconds (110 seconds). Speed-up calculated by article author: 1-110/360=0.694 (69.4% saved). Speed-up calculated by me: 360/110=3.27 (3 times faster).

Please calculate it the other way around. It makes great difference when you say you made something 10× faster than when you say you saved 90% of work even if both mean exactly the same thing.

Bruce Dawson has great article about this: https://randomascii.wordpress.com/2018/02/04/what-we-talk-ab...

Reply


@ohwaitnvm 8 months

Replying to @kuroguro 🎙

I haven’t played GTAO in a couple years, but I vaguely remember using the suspend process option of windows task manager to jump into online much faster.

Googling gave me this result, which sounds about right for what I remember. https://www.gfinityesports.com/grand-theft-auto/gta-online-h...

I’m not certain it’s still a valid workaround, and it’s not nearly as sophisticated as the OP method, but at least everyone can do it :)

Reply


@powerfulclick 8 months

Replying to @kuroguro 🎙

This is really cool - how did you develop the background knowledge to solve this? I'm trying to learn more about low-level stuff and I would have no idea how to approach solving a problem like this

Reply


@punnerud 8 months

Replying to @kuroguro 🎙

tl;dr

* There’s a single thread CPU bottleneck while starting up GTA Online

* It turns out GTA struggles to parse a 10MB JSON file

* The JSON parser itself is poorly built / naive and

* After parsing there’s a slow item de-duplication routine

(Copied from the END of the article)

Reply


@ozzmotik 8 months

Replying to @kuroguro 🎙

I find it interesting that, with ~63k entries, they didn't just use a bloom filter to do a lookup on if the entity they're working with has already been seen. granted, they stoll need to store the data, but I think a bloom filter would be a more effective way to test if the item exists already

Reply


@tyingq 8 months

Replying to @kuroguro 🎙

"They’re parsing JSON. A whopping 10 megabytes worth of JSON with some 63k item entries."

Ahh. Modern software rocks.

Reply


@high_byte 8 months

Replying to @kuroguro 🎙

this is amazing. good work! unbelievable

Reply


@schemescape 8 months

Replying to @kuroguro 🎙

The author's frustration with GTA Online reminds me of my own experience with Borderlands 2 taking over 1 minute for its "searching for downloadable content" loading phase. I hate DLC, so having to wait for it was especially aggravating.

I did not ever consider inspecting obfuscated assembly and hooking functions to fix the problem myself. Very impressive work!

Reply


@animal531 8 months

Replying to @kuroguro 🎙

I played the game on PS3 and PC. The loading time at launch for PS3 (and later for PC, albeit on SSD) wasn't great, but it also wasn't nearly this terrible.

From a game programming perspective, I'm sure at launch there were very few extras to obtain, so this method ran fast and didn't raise any red flags.

But as time has worn on they've added a ton of extra items and it's become a real problem. What it does show is that probably most of their team are working on creating new things they can sell, vs testing and maintaining the codebase for the last N years.

Reply


@forrestthewoods 8 months

Replying to @kuroguro 🎙

Super impressive. I am generally not impressed with community hacks because, as every comment thread discusses, they are generally hacks that do not support all players and thus can’t be shipped.

But this seems pretty damn clean. And it’s egregious that no one at R* took a week to investigate and fix “correctly”.

Reply


@sailfast 8 months

Replying to @kuroguro 🎙

This is such an awesome article, and I love it. Thanks to the author for the great digging and explanation.

Reply


@say_it_as_it_is 8 months

Replying to @kuroguro 🎙

Micro transactions were probably added in late stages of development by someone pressured to get the job done quickly and the developers were savvy enough to believe that the added overhead to slow initial loading was standard fare that users already accepted. Not enough time to optimize, forced to ship, and good enough.

Reply


@dvdbloc 8 months

Replying to @kuroguro 🎙

I bet if the devs did more leetcode this would have never happened /s

Reply


@z92 8 months

Replying to @kuroguro 🎙

Also note that the way he fixed it, strlen only caches the last call and returns quickly on an immediate second call with the same string.

Another reason why C styled null terminated strings suck. Use a class or structure and store both the string pointer and its length.

I have seen other programs where strlen was gobbling up 95% of execution time.

Reply


@ww520 8 months

Replying to @kuroguro 🎙

Yep, O(n^2) has the problem that no matter how fast you upgrade your hardware it would still lag.

Another pet peeve of mine is Civ 6's loading time for a saved game is atrocious. I'm sure there's a O(n^2) loop in there somewhere.

Reply


@GuB-42 8 months

Replying to @kuroguro 🎙

Maybe even more surprising to me is that sscanf() relies on strlen().

I would have expected libc to take that use case in consideration and use a different algorithm when the string exceeds a certain size. Even if the GTA parser is not optimal, I would blame libc here. The worst part is that some machines may have an optimized libc and others don't, making the problem apparent only in some configuration.

I believe standard libraries should always have a reasonable worst case by default. It doesn't have to be perfectly optimized, but I think it is important to have the best reasonable complexity class, to avoid these kinds of problems. The best implementations usually have several algorithms for different cases. For example, a sort function may do insertion (n^2, good for small n) -> quicksort (avg. nlog(n), worst n^2, good overall) -> heapsort (guaranteed nlog(n), slower than quicksort except in edge cases). This way, you will never hit n^2 but not at the cost of slow algorithm for the most common cases.

The pseudo hash table is all GTA devs fault though.

Reply


@thitcanh 8 months

Replying to @kuroguro 🎙

This is why people should use commonly-available packages instead of rolling their own version of whatever dumb algorithm they think they can write. This happens all the time. Bugs have been fixed by others, but everyone is too smart to use someone else’s code.

Reply


@breakingcups 8 months

Replying to @kuroguro 🎙

It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.

I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.

This online gamemode alone made $1 billion in 2017 alone.

Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.

Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.

The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.

(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)

Reply


@luckystarr 8 months

Replying to @kuroguro 🎙

Well, that's embarrassing. I can't even imagine the level of shame I would feel if I had written the offending code.

But, you know, premature optimization yadda yadda.

Reply


@mekkkkkk 8 months

Replying to @kuroguro 🎙

As others have pointed out, this is a good illustration of why you need accurate data during development. I bet that the data set used during development was dirty with duplicates and way too small. It was faster and more convenient to just code defensive garbage than to be the annoying one nagging to another team about the state of data. So this was written, probably along with a TODO comment, and then forgotten about, and ultimately shipped. I've done this same thing. Not with any real consequences, but still. It's what happens when time is of the essence.

How it remained undetected for so long is really weird though. Surely they must've had a massive amount of complaints about the loading times. I completely stopped playing because of them. How could they not have investigated the root cause, atleast once, in six years?

Reply


@comboy 8 months

Replying to @kuroguro 🎙

Holy cow, I'm a very casual gamer, I was excited about the game but when it came out I decided I don't want to wait that long and I'll wait until they sort it out. 2 years later it still sucked. So I abandoned it. But.. this... ?! This is unbelievable. I'm certain that many people left this game because of the waiting time. Then there are man-years wasted (in a way different than desired).

Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020.

Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.

Reply


@sova 8 months

Replying to @kuroguro 🎙

Really great work. Here's to hoping someone at Rockstar notices. You didn't even need access to the original sourcecode to debug it! Quite impressive my friend. Thanks for the great write-up!

Reply


@singhrac 8 months

Replying to @kuroguro 🎙

Honestly, while this horrible code is mildly offensive to me, I'm pretty impressed by this person's persistence. It's one thing to identify a bug in a compiled program, but it's another to patch it without fully understanding what's going on. Caching strlen was a particularly clever trick that sidestepped a bunch of more complicated solutions.

Reply


@eptcyka 8 months

Replying to @kuroguro 🎙

BTC mining is using a significant amount of global energy production every year now. The more cynical might say that ultimately it's a waste of human effort, but I have to say that this bug (and it's most definitely a bug) should be patched (and should've been patched) to save the kilowatts of power wasted parsing mostly the same data.

Reply


@seer 8 months

Replying to @kuroguro 🎙

A lot of comments here kind of brush off the fact that this has been going on for _years_! I would expect quadratic behaviour to slip here and there in non real time code places. We make mistakes after all.

I've had to revisit production code from time to time to discover my or my colleague assumptions about library functions were not actually true.

But for someone in the possession of the source code this looks like _minutes_ worth of profiling time! With a stellar technical accomplishment that GTA5 truly is, you would expect tons of really talented devs that could have traced this one down on their lunch break.

The fact that they didn't speaks volumes about organisational structure within R*. I think some higher up managers _really_ need some stern talking to!

Reply


@4cao 8 months

Replying to @kuroguro 🎙

Excellent investigation, and an elegant solution.

There's a "but" though: you might end up getting banned from GTA Online altogether if this DLL injection is detected by some anti-cheat routine. The developers should really fix it on their end.

Reply


@AJRF 8 months

Replying to @kuroguro 🎙

There’s one thing that made me curious. The author said the load times when the game came out were awful as well.

So has that JSON always been 10mb or has it grown a lot over time?

Has the load time crept up since launch or has it gone down?

Reply


@Shadonototro 8 months

Replying to @kuroguro 🎙

think about it, how much electricity wasted

millions of players, billions of minutes wasted

when you scale things, every bit of optimizations IS MENDATORY!

#saveOurPlanet

Reply


@oatz 8 months

Replying to @kuroguro 🎙

Love performance issues like this, good read.

Reply


@DarkByte 8 months

Replying to @kuroguro 🎙

This is quite impressive though it is important to remember that this came out for machines like the Xbox360 and the PlayStation 3. I am not convinced these modifications would “fit” on those machines without significant thrashing/paging and significant memory usage to the point where other negative effects come into play. Great read and effort regardless.

Reply


@captain_price7 8 months

Replying to @kuroguro 🎙

I loved so much reading it, I was thinking that if someone were to write fictional, sherlock holmes like fantasy story where our sherlock would take some (maybe fictional) widely used software at each episode, investigate it like this, and reveal some (fictional) grand bug in the end- I'd totally read it.

Yeah I know it sounds stupid, but I suspect real Sherlock Holmes was inspired by a true story like this one too, and at least some contemporary detectives started to enjoy reading them.

Reply


@TamDenholm 8 months

Replying to @kuroguro 🎙

The Github link is dead, DMCA or incorrect link?

Reply


@arisudesu 8 months

Replying to @kuroguro 🎙

It doesn't explain some things to me. Maybe I read the article inattentively, but which stage of the loading hangs for a very long time? Loading screen, or city overview when joining a session? In my experience, on modern hardware, the loading screen goes very quickly (a minute or two).

What's worth noting is that the city overview can hang for 20-30 minutes during connecting. At the same time, suspending the GTAV.exe for 20-30 seconds allows you to immediately throw the player into an empty public session. It is unlikely caused by slow parsing of json, more like the suspend causes udp timeout when connecting to the slow session hosts.

Reply


@Taylor_OD 8 months

Replying to @kuroguro 🎙

I wonder how long Rockstar will take to respond/fix.

Reply


@xyst 8 months

Replying to @kuroguro 🎙

not surprising - the game industry is absolutely notorious for cutting corners. didn't know they cut corners this much though.

will r* fix it? maybe, especially since some person literally did half of the work for them. But given R* is a large company, this probably wont be fixed for a long time, and GTAO is probably being maintained by the lowest bid contractor group.

They probably have all of their full time assets working on the next iteration of GTA.

Reply


@CTDOCodebases 8 months

Replying to @kuroguro 🎙

Imagine all the time that people are wasting if they are sitting there waiting for it to load. I wonder how many lifespans it adds up to?

Reply


@gfodor 8 months

Replying to @kuroguro 🎙

Hacker News

Reply


@mattbillenstein 8 months

Replying to @kuroguro 🎙

Christ.

Reply


@jonasfj 8 months

Replying to @kuroguro 🎙

I bought a PlayStation a few years go to play GTA..

Never tried consoles before, but figured the point was that it would be fast and hassle free.

GTA was fun, but after completing it I never load it, just to play for fun, because PS load times + GTA load times are too much.

Now I just play 20 year old games under wine on Linux.. that's less of a hassle, and I don't have to touch windows.

Reply


@holyknight 8 months

Replying to @kuroguro 🎙

Thank god. I always suspected that those loading times were cause by some retarded implementation detail. GTA5 is not that complex to justify that kind of loading times. Even the hardware has scaled massively since their launch and it doesn't even matter.

Reply


@progx 8 months

Replying to @kuroguro 🎙

I can't believe what Rockstar did here. I and 3 friends stop playing GTA V cause of the unbelievable stupid loading times.

Reply


@hisfastness 8 months

Replying to @kuroguro 🎙

This is my favorite read on HN, ever. So cool!

Reply


@jokoon 8 months

Replying to @kuroguro 🎙

Parsing text is always expensive on the CPU, that's why it's better to often prefer binary files when it's possible.

It's also the reason the DOM is so slow, in my view.

I remember spotting a lot of fast JSON parsers around, but again, there doesn't seem to be any popular, open, flexible, binary file format out there.

Meanwhile, games are always larger and larger, machine learning requires more and more data.

There is ALWAYS money, battery and silicon to be saved by improving performance.

Reply


@baby 8 months

Replying to @kuroguro 🎙

Flamegraphs ffs!

Reply


@c7DJTLrn 8 months

Replying to @kuroguro 🎙

Sad that the source (https://github.com/tostercx/GTAO_Booster_PoC) has been taken down, I wanted to give this a try myself. I imagine a lot of GTA Online players would.

Reply


@polote 8 months

Replying to @kuroguro 🎙

I have absolute respect for people who are able to do that kind of things. A few years back I've tried to play with reverse engineering but I never managed to get the skills.

Reply


@pid_0 8 months

Replying to @kuroguro 🎙

Very cool analysis. Its unbelievable that this is still unfixed. I don't play GTA anymore mostly because of the load times for every single action.

Reply


@segmondy 8 months

Replying to @kuroguro 🎙

This is why I come to HN, I was going to skip this because I thought it was about video games, but really glad to have read it, and loved every line of the article.

So much to get from this.

Even if you don't have the source, you can make a change if you are annoyed enough.

If you don't like something, and the source code is out there, really go contribute.

Performance matters, know how to profile and if using an external dependency, then figure out their implementation details.

Algorithms & Data structures matter, often I see devs talking about how it doesn't matter much but the difference between using a hashmap vs array is evident.

Attentive code reviews matter, chances are they gave this to a junior dev/intern, and it worked with a small dataset and no one noticed.

Reply


@Amadhoun9 8 months

Replying to @kuroguro 🎙

Wowwwww

Reply


@T3RMINATED 8 months

Replying to @kuroguro 🎙

Hi this is Roman from Rockstar.

Reply


@ameyv 8 months

Replying to @kuroguro 🎙

This should be tweeted to their official channel or someone inside company (on twitter) who has ability let relevant Rockstar dev know about it...

Reply


@1watts 8 months

Replying to @kuroguro 🎙

I bet this feature, relating to in game purchasing options, was retro-fitted by some non-core dev team who delivered it to spec and fixed cost

Reply


@TinkersW 8 months

Replying to @kuroguro 🎙

Pretty strange that nobody at Rockstar bothered to run the games loading phase in a profiler, this problem would have been super easy to observe if they had.

Reply


@estomagordo 8 months

Replying to @kuroguro 🎙

Hey OP, what would it take to persuade you to have a look at Hearthstone next?

Reply


@coold 8 months

Replying to @kuroguro 🎙

[joke]so, they say that c++ is faster than java[/joke]

Reply


@ce1lingisr00f 8 months

Replying to @kuroguro 🎙

I like this page design. It looks familiar. Does anyone know where this design comes from? Or if it's original?

Reply


@bagacrap 8 months

Replying to @kuroguro 🎙

scary to think of the carbon emissions this bug has generated over the years.

Reply


@avar 8 months

Replying to @kuroguro 🎙

GTA:O shows advertisements for in-game purchases on the loading screen. How many advertisements you see is a function of how long the loading screen takes.

Something tells me this "problem" was discovered long ago at Rockstar HQ, and quietly deemed not to be a problem worth solving.

Reply


@azhenley 8 months

Replying to @kuroguro 🎙

Someone hire this person!

Reply


About Us

site design / logo © 2021 Box Piper