devtrackers.gg
Here are some brief updates on some ongoing concerns of a more technical nature.
Master server issues
The master server experience will not be terrible going forward. Much of our concern before release was that our dedicated game server component could handle any capacity we wanted to throw at it. This is the executable that actually serves gameplay that we run thousands of instances of globally. Instead, the server component turned out to be very efficient and scale very well, plus the fact that we accidentally over-provisioned servers helped even more with that. With the master server on the other hand we have been struggling from the start, the main problem being one of those silly programming things that are hard to see. Between weekend two and three we did a massive refactoring to simplify things and see things clearly. This caused some crashes initially because sometimes getting rid of leaks exposes issues that were previously hidden. We do have an automated stress-testing mechanism for the masterserver and also an alpha community that tests between weekends, but it’s hard to replicate the real thing accurately, because our masterserver is relatively feature-rich, so some of those issues sneaked in. Eventually we fixed these minor issues and figured out the major issue that has been plaguing us for three weeks, so things should be stable henceforth. We could have done all of this better in a variety of ways, we apologise for all the inconvenience.
Cheating
I want to reassure everyone that the people who are cheating are being correctly flagged as cheaters in our systems. This has been true so far for any video or report of anybody cheating that we've seen. We haven't been enforcing en mass yet for two reasons: we had other priorities that have kept us fully busy (like stability of the service and we are still trying to push new features/gameplay every weekend), but also, we didn't want to start enforcing right away until we determine which of the events that we get from EQU8 are a sure sign of cheating and which ones are wont to producing false positives and we should handle manually. For example, perhaps the events that deal with manipulation are pretty reliable, but those which deal with snooping can be triggered by harmless programs. We want to have a good grasp on this and have EQU8 potentially tweak some signals before we start over-banning. Either on the next weekend or for open beta you should start seeing prompt banning of cheaters. But be assured that the cases that have been posted in this subreddit are all being detected.
The massive ram leak
Some people were experiencing a massive RAM leak. This was related to an optimisation where we kept our menu interface in a suspended state while a game is in progress to make sure it doesn't interfere with the experience. Due to a specific race condition, some users were placed in a state where our UI engine (Gameface) would keep accumulating work and not releasing it. This is why going to the menu or bringing up the console would stop the leak. This crude optimisation has been removed and replaced with other controls, and this major memory leak appears to be gone, from our testing.
Performance and input smoothness issues
We track performance and smoothness very closely and we are aware of some issues that we will address as soon as things calm down:
In some circumstances our game doesn't respect the exclusive full screen setting.
The anticheat interfered with the ability of people to set the "disable fullscreen optimisations" settings, unless the setting was applied using the "apply to all users" screen, but this particular issue should be fixed now.
Windows' seemingly varying behaviour from version to version and from setup to setup with regards to whether to respect "disable full screen optimisations" or not complicates diagnosis of some of these issues, we are trying to get to the bottom of this and we're also trying to contact someone at Microsoft who can shed some light.
We are aware that more control over whether to allow tearing or not would be welcome and we will provide that.
Netcode and client-sideness
As many have noticed, many things are client-side right now that don't need to be. This was judged convenient for the stress test weekends as it will allow us to deal with server-side registration issues later and it would also make the servers run lighter (which was a concern at the time). The important thing for us at this point was to test that the general transport of packages is robust and that it doesn’t tend to congest. This is where things go usually wrong with games and not in the implementation of higher-level details like the degree and mode of reconciliation/extrapolation, even though this is usually the focus of community discussion. We will be introducing more server-side checks as we test them. Be assured that we will not introduce any change that makes hit registration any less reliable than it is now (assuming some ping requirements). In the future we will probably still have client-side prediction for hit-testing below a certain threshold of ping, with an informational approach on the server side where we are recording cheating incidences. For example: full-prediction up to 100 milliseconds, trust client in the context of the game, register infractions to act later. If for example you have 140ms of a ping you would have to engage in 40ms worth of estimating for placing your shot. The 100ms figure is an example, we haven’t decided yet. Going forward we want to expose in the advanced settings of custom games all of our netcode settings, (i.e: general prediction window, projectile prediction window, whether extrapolation is used and how much, and whether reconciliation is used and how much.) We would like to just put this out there and let the community experiment and the feedback that we then get from the community will inform what official settings we use for our servers.
Location-specific routing issues
- Some users using the Dubai datacenter are getting a dismal ping. This is due to an ongoing dispute between local providers that i3D has been trying to work around for a long time. The ping for users of the provider Etisalat is really high as a result because those packages are having to be routed through Europe. So Etisalat subscribers from UAE should avoid this datacenter for the time being. Otherwise, the connectivity in this location is generally good for the Gulf region.
- Similar issues affect some Russian providers with respect to the Moscow datacenter, and for the people affected the packages are being routed through Frankfurt, so people in the area with a latency to Moscow that seems abnormally high should play on the Warsaw datacenter for the time being. Update: i3D also tells me that the connectivity from Ukraine to the Moscow datacenter should improve pretty soon due to a new peering agreement in the country.
- Swedish customers of the Bredband2 provider will have seen their ping improve by 10-15ms to the Rotterdam location thanks to some routing fixes i3D enacted based on our user reports.
- A routing issue affecting the connectivity of some users with the Ashburn location was fixed during the past week, too.
I’m sure I’m forgetting something important, but know that we are going through every issue people have, no matter how minor or how few people are affected and we’ll eventually get to any given issue.
Update: Added the routing issues segment.
External link →