[2020] GGG pls defragment your own ggpk before deploying it to Steam

The reason we don't do that is that it would mean redownloading almost the entire game again every patch.

The steam patcher breaks down big files into blocks and then only gives you the blocks that change. If you try rearrnaging the internals of a single file then basically every block of the file will be changed.

All of the "garbage" in the file is old file versions. By not removing them it makes the patch sizes smaller for existing users at the expense of making new full downloads by new users larger.

Of course, the big single file approach isn't good for steams patcher generally. The main reason is because steam makes a full copy of the entire file when it patches it, patches the copy, then moves it back over the original again. This is to prevent corruption in the case where, for example, you lost power while steam was in the middle of patching a game. This means that patching a 1KB file makes you computer rewrite the entire many-gigabyte file which is why it's slow.

So for that reason, we will be changing the resource system for steam over to use the method we use for consoles, which will significantly improve the situation. Not only will it improve all the patching issues on steam, but it will also come with a pretty significant improvement to loading speed of non-SSD drives. The console version attempts to bundle related files together into files that can be loaded all at the same time.

It's just taking some time because it's not as high priority as other issues that people have been working on. The main blocker is that we have the make the non-steam patcher aware of this new system too. Also, this would of course require a complete game redownload when we do the patch, so we want to make sure we only do it when we have a pretty significant expansion already.

It will be before PoE2.

Unpacked, Path of Exile is hundreds of thousands of tiny files. Many of them are tiny hand edited or tool generated text files that might just be 1KB or so. Packing them together is important for performance so you don't have to do so many disk seeks while loading files.

But so long as your files are still fairly large (32 MB or so per file for example) then the performance of seperate files is totally fine.

For conveniance of the original patcher, it was simple to have one big file. And it was a fairly common thing to do at the time ~13 years ago. Our patcher was modeled on the Guild Wars 1 patcher, and it's quite similar, but blizzard had it's MPQ files and so on. The big single file is structured such that it's very easy for us to work out the minimum set of files that need to be patched. We just check a single hash if nothing has changed, and only explore the file tree to the parts that have changed if not.

We didn't add steam support until the game was already out.

It will be for all users.

We need to get the existing non-steam patching system to be able to patch content for the new system first though.

I don't really like that approach because most people just have steam running in the background and auto patching all the time, and if you don't use steam's patching then you don't get that. The game wouldn't patch until you run it.

The new way will be both friendly for steam, and have higher performance anyway.

The Vulkan renderer was moved up in priority instead of this because the shader related stalls were getting worse.

Since the patch that changes the format will also require a full game redownload, we also plan on fixing some serious problems with the texture compressor we use at the same time, so it has to wait until that stuff is done too.