devtrackers.gg
Hi everyone,
Below is the first of our quarterly game performance improvement previews. This was written by our Engineering team and explains the main performance improvements planned for Update 27. If you haven’t yet, we encourage you to also check out the Update 27 Combat Preview here.
--
Hi friends!
We're here to bring you a preview of some of the performance improvements planned for Update 27. While it is impossible to capture everything that the team has done, we aim to highlight some of the areas we are focusing on and what we expect to gain as a result.
Persistent AoE Improvements: Area of effect abilities that last for long durations have the potential to send a large number of messages down to the client over time. This is exacerbated in zones like Cyrodiil where many players are routinely casting and being hit by these types of abilities. To mitigate this, we have implemented new AoE tech on the server that will keep the same functionality, but significantly reduce the number of hit result messages the client has to process. Internal testing has been positive, showing a reduction in latency and FPS drops on the client. We are taking a cautious approach to rolling out this new tech, choosing to focus solely on Dragonknight Standard (and morphs) for U27. We will evaluate the improvements on the PTS and live servers and continue to update additional player abilities in future updates.
Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.
Item Set Abilities: The team has been hard at work auditing and streamlining each and every item set ability in the game, with a focus on looking for ways to improve performance. This has resulted in reduced calculations on the server and, similarly to the AoE improvements, a reduction in the number of messages sent down for the client to process.
Critical Memory Situations: We know how frustrating it is to teleport into a busy city and encounter a crash. While we have significantly reduced the number of out-of-memory related crashes in recent updates, it is still something that can happen to our players. To prevent these crashes, the console versions will now detect when the game is about to run out of memory and react by temporarily unloading vanity pets and personalities from players outside of your group. We were able to build upon the work completed in U26 that moved vanity pets to the client to help make this possible. Once memory is back within a safe threshold, these assets will begin to load back in as players move through the world.
Physics Implementation: Our Havok physics engine implementation, specifically dealing with Keep pieces in Cyrodiil, has had some issues over the years. We took some time to clean up the code surrounding this, reducing unnecessary complexity and making Keep pieces behave more like normal fixtures. Essentially, they will still look and behave as they do now. While working on this, we tracked down an elusive bug where some physics objects were not being cleaned up properly. This issue was not specific to any zone and would result in the client crashing when it occurred. The end result of the physics hardening should be fewer bugs and crashes.
Trial Performance: Our metrics have shown that Trial performance on the server has not been a consistent experience. We were able to track down the root cause of this–instance launchers running multiple Trials simultaneously would routinely perform worse overall. We've taken a pass at the algorithm which determined how Trial instances are distributed and now factor this into account. This will decrease the chances of our instance launchers being overloaded in high usage situations and impacting Trial performance as a result.
Database Improvements: We've made some backend changes to incorporate an in-memory database cache in our server processes, which will reduce the amount of contention on our production databases when under stress. Initially for U27, we are setting the foundation and focusing on a small piece of the Activity Finder tool which should result in smoother processing overall. Once we’ve evaluated this on the live servers, it will create opportunities in future updates to move over additional functionality, allowing us to perform more work simultaneously without performance degradation.
Intermittent Load Screens: It's a source of frustration for everyone when you're running around Tamriel and you're intermittently hit with short load screens. We've identified two areas where this behavior could occur. The first issue is a bug with streaming of scaled assets that are switching between LOD models and main models. This should go to the live servers with an upcoming U26 incremental. The second issue is regarding a bug that could cause the game to have much bigger physics streaming radius when a high view distance is selected, resulting in more loads than necessary. This fix will release with U27. We will be actively monitoring the live server metrics after U27 releases to determine if additional investigation and work is needed.
We know this won't solve every issue our players are experiencing, and there is still much work to be done. We will continue working diligently to provide you the best experience possible. Be on the lookout for some experimental Cyrodiil changes over the course of U27. Details for that will come in a future update. Thank you!
Below is the first of our quarterly game performance improvement previews. This was written by our Engineering team and explains the main performance improvements planned for Update 27. If you haven’t yet, we encourage you to also check out the Update 27 Combat Preview here.
--
Hi friends!
We're here to bring you a preview of some of the performance improvements planned for Update 27. While it is impossible to capture everything that the team has done, we aim to highlight some of the areas we are focusing on and what we expect to gain as a result.
Persistent AoE Improvements: Area of effect abilities that last for long durations have the potential to send a large number of messages down to the client over time. This is exacerbated in zones like Cyrodiil where many players are routinely casting and being hit by these types of abilities. To mitigate this, we have implemented new AoE tech on the server that will keep the same functionality, but significantly reduce the number of hit result messages the client has to process. Internal testing has been positive, showing a reduction in latency and FPS drops on the client. We are taking a cautious approach to rolling out this new tech, choosing to focus solely on Dragonknight Standard (and morphs) for U27. We will evaluate the improvements on the PTS and live servers and continue to update additional player abilities in future updates.
Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.
Item Set Abilities: The team has been hard at work auditing and streamlining each and every item set ability in the game, with a focus on looking for ways to improve performance. This has resulted in reduced calculations on the server and, similarly to the AoE improvements, a reduction in the number of messages sent down for the client to process.
Critical Memory Situations: We know how frustrating it is to teleport into a busy city and encounter a crash. While we have significantly reduced the number of out-of-memory related crashes in recent updates, it is still something that can happen to our players. To prevent these crashes, the console versions will now detect when the game is about to run out of memory and react by temporarily unloading vanity pets and personalities from players outside of your group. We were able to build upon the work completed in U26 that moved vanity pets to the client to help make this possible. Once memory is back within a safe threshold, these assets will begin to load back in as players move through the world.
Physics Implementation: Our Havok physics engine implementation, specifically dealing with Keep pieces in Cyrodiil, has had some issues over the years. We took some time to clean up the code surrounding this, reducing unnecessary complexity and making Keep pieces behave more like normal fixtures. Essentially, they will still look and behave as they do now. While working on this, we tracked down an elusive bug where some physics objects were not being cleaned up properly. This issue was not specific to any zone and would result in the client crashing when it occurred. The end result of the physics hardening should be fewer bugs and crashes.
Trial Performance: Our metrics have shown that Trial performance on the server has not been a consistent experience. We were able to track down the root cause of this–instance launchers running multiple Trials simultaneously would routinely perform worse overall. We've taken a pass at the algorithm which determined how Trial instances are distributed and now factor this into account. This will decrease the chances of our instance launchers being overloaded in high usage situations and impacting Trial performance as a result.
Database Improvements: We've made some backend changes to incorporate an in-memory database cache in our server processes, which will reduce the amount of contention on our production databases when under stress. Initially for U27, we are setting the foundation and focusing on a small piece of the Activity Finder tool which should result in smoother processing overall. Once we’ve evaluated this on the live servers, it will create opportunities in future updates to move over additional functionality, allowing us to perform more work simultaneously without performance degradation.
Intermittent Load Screens: It's a source of frustration for everyone when you're running around Tamriel and you're intermittently hit with short load screens. We've identified two areas where this behavior could occur. The first issue is a bug with streaming of scaled assets that are switching between LOD models and main models. This should go to the live servers with an upcoming U26 incremental. The second issue is regarding a bug that could cause the game to have much bigger physics streaming radius when a high view distance is selected, resulting in more loads than necessary. This fix will release with U27. We will be actively monitoring the live server metrics after U27 releases to determine if additional investigation and work is needed.
We know this won't solve every issue our players are experiencing, and there is still much work to be done. We will continue working diligently to provide you the best experience possible. Be on the lookout for some experimental Cyrodiil changes over the course of U27. Details for that will come in a future update. Thank you!