devtrackers.gg
Hey, Legends.
We recently tweeted that an update had gone out to address some sound and visual effect drops since the start of Season 16 and wanted to provide more details and context. While this won't resolve all audio and visual concerns raised by the community, we are dedicated to improving awareness on the battlefield. Thanks for your patience and reports.
Love details or a peek behind the scenes? Keep reading to learn more about our dev team’s investigation and eventual fix. TLDR? Skip to “Cause and Effect.”
---
Discovery & Investigation
We first noticed the problem soon after the release of Season16 - Revelry with various instances of dropped sound FX, usually alongside particles. For instance, grenades would sometimes not explode even though they damaged players. This had not occurred during our Season 16 playtesting, could not be reproduced internally after initial reports, and was very difficult to pin down using live gameplay videos as the root cause was not always shown in the player's POV. Early investigations into "disappearing nades" took us down unrelated paths as they weren't “disappearing” per se. It wasn't until the later reporting of missing gun SFX and VFX that we started to understand what the cause might be.
After a preliminary investigation, the primary suspect was found to be the system our servers use to dispatch “start”/”stop” commands for various effects (e.g. certain sounds, particle systems, physics impacts, bullet tracers, explosions). Because our servers simulate entities (such as players, weapons, abilities, loot, etc.), they will emit sound/visual effects. Every server frame compiles an effects list of a maximum of 128 entries - any additional effects above the limit were getting dropped. This list is sent to any players who need the effects for the specific server frame.
Narrowing Focus
From there, the theory was that something may be flooding this engine limitation, requesting thousands of effects every second! But was this a systemic issue or could it be a single entity acting up? Every season update comprises thousands of changes to assets, code, script, and levels. Which meant finding a needle in a haystack.
This is usually where metrics come in. Our game servers send telemetry back to us so we can monitor performance, crashes, and various things that aren’t typical or generally happen. In this case, our current telemetry data wasn't helpful since it did not indicate any flags or issues in the system. This indicated to us that this was likely a unique and new situation our systems were not previously met with.
This left us with a complex issue that we knew was impacting our community, but was hard to reproduce despite detailed reports, had minimal leads internally, and there were no metrics to prove definitively that this limit was being hit at all.
The next step was to investigate the various limitations of this system. Eventually, we were able to reproduce the problem in artificial situations. For instance, having a squad of 50 legends all firing the same weapon at the same time or spawning a bunch of invincible players that would auto-run while using their abilities/weapons as much as possible with instant recharges! This gave us proof that FX would get dropped, but only with completely unrealistic test cases. Various aspects of our server performance were investigated, but nothing definite was found.
During our investigation we were keeping a close eye on any reports coming in about the issue, and noted that it seemed to happen more at high-level play. With a subset of games to look at, we started to look at deploying a server update to add some much-needed metrics to bring in more data about our server effect networking and narrow our focus further to hopefully find that needle in the haystack. As the server update was finalizing, we found it.
Cause and Effect
A single line of code was identified to be the root cause of the issue. Season 16’s new weapon.
The Nemesis has a particle effect that ramps up when heating up, but when it’s uncharged we don’t need to waste resources playing it as the effects are concealed inside of the weapon. So, when the weapon gets updated we would simply stop this particle effect if the weapon had no charge.
Every time the server simulates an input from the weapon's owner, this particular line of code is executed. Players send in their inputs for every single frame that is run on their client, and it’s the server’s job to simulate all of these inputs. This means that every single player with an uncharged Nemesis would create a “stop particle” "effect" on the server every frame, and this line of code was being called even when the weapon was holstered.
This immediately explained how this would occur more often at high-level play with the issue being directly correlated to the framerate of each client that had a Nemesis. 14 clients with a Nemesis running at 180fps would be enough to cause FX to begin being dropped. This also explained how this wasn’t seen during internal testing. The builds used for testing might not have had enough holstered Nemesis in play, had a rarer correlation with missing FX, or didn’t have enough clients at that fps - something for us to keep in mind and improve on for future testing.
An aside on testing and opportunities to identify "rare" bugs: a minute of players playing Apex is the equivalent of 10 testers playing the game for a year!
Fix Deployed
This fix was rolled into the metrics server update and deployed last Tuesday, and since then we have been keeping a close eye (and ear) on socials and our new metrics. This may not address all FX concerns, but everything with this particular beast of a bug is looking good again!
For future updates, follow the Respawn Twitter account for the latest info or check out the Apex Tracker Trello for bugs or concerns we’re continuing to investigate.
External link →