devtrackers.gg
Julia: We’re back with new Tech Tuesdays! WOOO!
To start off, you may have (or have not) noticed we’ve released a couple chunky hotfixes recently instead of the extended times between updates, and there’s another fix on the horizon. Hotfixes are normally reserved for absolutely game breaking issues, but… A pledge to make the game smoother is a pledge, you see, so across two of the following Tech Tuesdays we will explain why we have decided to speed up the development process with hotfixes between standard updates.
Hi, I’m Murillo and I’m the lead developer on Worlds Adrift.
A big part of running an early access game is responding to things going wrong as quickly as possible. There is still room for improvement, but I’m going to outline some of the challenges we face when delivering hotfixes and some of the actions we took to minimize the time our players are affected by issues. How to Hotfix Hotter!
The best way to not have players affected by issues is, of course, to catch them before they make it live! However, on a project of the magnitude of Worlds Adrift, that is not an easy feat to achieve. Having many interconnected moving parts means every change you make can potentially have far reaching and hard to predict implications which cause bugs or exploits to show up in areas you wouldn’t expect. Thuntomite Effect, if you will.
Since testing all parts of the game takes is intensely time consuming, we have to be smart about what we target and how we go about it. At the end of the day, though, there are way more players than testers, so statistically, you folks will find a lot of stuff we just didn’t have the time to. So, something made it past our net for one reason or another, and it turns out to be a very bothersome/important enough issue that we need to hotfix. What do?
The first step is figuring out what’s causing it. This is highly dependent on the issue; being able to reproduce it ourselves is the most important factor for how quickly we identify where it’s coming from, but in a lot of cases all we have to go on is information provided by players. (Report your issues and give us your output logs!)
In either case, a lot of sleuthing ensues – a process which by itself is more than enough for its own blog post.
Once we figure out and fix the issue, we have to go through another round of testing the game to make sure no undesirable collateral effects were introduced by the fix. This round of testing has a bit of malleability on how thorough it needs to be compared to a full release, which ultimately comes down to how dangerous we feel the fix is and how urgent a solution to the problem feels; we’re trying to improve the players’ experience, but it’s a hard balancing game of how much we want to make sure things are still solid and how much of a hurry the problem puts us in. Finally, once we validate everything to the level we decided we need to, the fix goes live. If another bug becomes the biggest grievance in the community, and if it’s bad enough that we think it can’t wait, we rinse and repeat the whole process. It’s a constant between all those steps is that the time they take is not really in our control (well, if you give us a pass and consider that introducing bugs is not our fault in the first place… :D).
Some issues are easier to diagnose than others, others take so long that we can’t figure them out in time for a hotfix and it becomes better to lump them in with a full update; many fixes are so small that it takes longer to read their description than to fix them, a few end up sprawling into multiple systems getting touched; and some require us to just check we can still log into the game, while others require us to test the game as thoroughly as with any normal release. So, if those factors are not in our control, what else can we do to make sure the patches come out as fast as possible? Well, there are many small factors that affect this, but I’ll focus on two that had the highest impact for us specifically.
First, when you are dealing with an MMO, build times can quickly balloon, and suddenly you realize you lose a lot of time between making a change to the game code and having a build in your hands that you can test your changes on. Because of that, we modified our systems and processes to minimize this time; selectively building only the parts of the game we’re changing (e.g. if we’re only making changes to the game logic servers, there’s no reason to build the physics servers), caching artifacts that we are not touching (the islands, for example, were a time hog for a long time due to their sheer numbers, until we separated them as much as possible from the rest of the project), and just generally optimizing things so that less time is wasted (saving even a few seconds of build time compounds really fast when you have several people doing several builds a day). None of these changes were made with the sole intention of allowing for faster hotfixes, but the benefit of it came nonetheless.
Finally (and this is more tangential to the process of fixing bugs), another thing that made us able to hotfix faster was being more disciplined about what we wanted to fix. It’s easy to be tempted to bundle in all the fixes your developers have made in parallel along with the one or two bugs that are actually urgent. After all, the more things you fix, the better, right?
However, this often would lead to delay after delay of the patch: it took longer to test the build, because you had more things that need to be validated; one of the fixes might have introduced a new issue which needed to be fixed in the hotfix, which took more time and meant it needed to be re-tested, which then tempted us to add the other few bugfixes that got completed in the meanwhile, and… Well, this could quickly snowball into days and weeks of delay, and in the meanwhile the urgent issues were still being a problem for players.
By isolating only the fixes that are crucial, all of these problems are mitigated, and we end up being able to reevaluate later if the other bugfixes are important enough to warrant a hotfix or if they can wait until the next big update, which makes things more efficient.
As mentioned before, there is room for improvement; but as part of our commitment to become as quick as we can when responding to problems that are affecting the enjoyment players get out of the game, we are constantly looking for more areas where we can make changes that further allow us to move faster when hotfixing. So keep sending Kirk reports and logs.
As always, see you in the skies,
Murillo
The post Tech Tuesday: Hotter Hotfixes appeared first on Worlds Adrift.
