[2021] The Balancing Act that is Game Stability

A lil somethin somethin: You can find the details for this event on the announcement page here.

From the day we launched Second Extinction into Early Access, one area of the game that we realized we needed to improve a lot was stability. The number of crashes reported was far more than we were expecting.

You might ask yourself, how do games with plenty of crashes get released?
We are a small team, and we are all on similar hardware and software setups, so with the variety of PC hardware (and software) that exists, there's plenty of things that can slip through our testing, unfortunately. Before we launched into Early access, we did compatibility testing, and those results were very positive. Then when we actually launched, we received many reports of crashes, so our coverage wasn't as good as we had hoped for. We chose to release with Vulkan for our graphics rendering, and that's seemingly a choice that has caused us some problems. Third-party software that injects into the Vulkan driver is something we often see as culprits in our crash reports.

Another thing we did not expect was that we got a lot of players with machines below our minimum hardware requirements playing the game. We have made some fixes to make it work better on lower-end hardware. We have also added a pop-up that warns the player when they are playing on hardware that is below min spec. The plan is to improve upon this more in the months ahead as well.



So how do we work with improving stability?
First of all, by accepting our EULA, you agree to allow us to send crash information to us. No worries, we don't collect any personal information, and it's only intended to allow us to improve the game’s stability.
When a crash happens on your side, we use Microsoft's MiniDumpWriteDump functionality to write a crash dump and also attach some additional data, such as what mission was being played, which CPU/GPU was used, which graphics driver version is installed, etc. This is then uploaded to our backend crash service, which symbolicates the crash report (turns it into a human-readable format, and by human, I mean programmer :)). It then passes the crash report on to Sentry, another service that bundles crashes together based on their call stack and allows us to filter crashes to identify any trends and see graphs over how many crashes come in.

What we receive on our end is what's called a "mini dump.” It's a very limited crash dump, where we pretty much only can see where in the code it crashes. We can’t see information on what was going on in the game when it crashed, so it can be very difficult to understand exactly what happened to cause the crash and also the steps and hardware required to reproduce the crash.



What are we doing to make the game more stable?
We are constantly trying to find solutions for crashes, but we often don't have a clear reproduction for them, so it can be very time-consuming to try and recreate the circumstances for the crash. Sometimes it's easy to speculate what the cause could be and apply a code fix without a repro-case, but not always.

Another thing we are doing this year is to build a hardware Test lab, where we can try the game in a much more varied amount of hardware setups and use this to reproduce issues that we get reported to us. Dxdiag reports are something that we rely on to get more detailed information about hardware. You can follow this guide[support.secondextinctiongame.com] to send us your Dxdiag if you’re experiencing crashes.

To summarize, we are constantly working on improving stability, and we still have some ways to go. Please keep reporting issues to our support team here[support.secondextinctiongame.com], and if you find reproduction steps for crashes, that information can be invaluable to us!

Tobias Johansson - Lead Programmer on Second Extinction