[2020] The War Has Ended on Emerald

Update on this...

The zone (Esamir) definitely crashed, which resulted in the sh*tshow epic display of camaraderie in the video posted. We are troubleshooting now and will post more details once we have a handle on the specific cause and required fix.

Regarding the Emerald and Cobalt/Miller performance issues you mentioned. There have been some ongoing ISP routing issues that our NetEng team has been tracking over the past couple weeks, resulting in a lot of EU players (most recently those based in Germany) suddenly having horrible connection quality to our EU servers. I've been in contact with a few players over the past couple days, along with our NetEng team making some manual routing changes. The latest round of changes we made was yesterday at about 12:55pm (Pacific). After we made the routing changes, the few players I was in contact with reported their connection quality returning to normal. But I'm sure there are others that might still be having problems.

To be clear, the issues above are not related to server performance. Its ISP routing between the player and our data center. Again, this is likely not the only issue, but in these cases, its was and it was pretty widespread. Our NetEng team is actively working with the telecoms that we've identified to see if they can make permeant changes on their end.

As for Emerald, we are continuing to troubleshoot. The zone crash (Esamir) that happened last night was an exception and does not happen frequently. However, we do see the reports of server connection quality taking a dive - especially around primetime. Each time we see these reports, we immediately check the various server health stats to see what might be happening (world crash, zone crash, low server FPS causing lag, etc). More often that not, there is no smoking gun on the server-side that indicates something catastrophic is/has happened. But obviously something is happening for players to suddenly start having their connection quality go to sh*t.

One thing to take into consideration with Emerald is due to its centralized physical location, we have a lot of EU/NA players on it (as well as players from locations physically further away.) So when we have a server issue or network event being reported, not only do we check the game servers and data center, we also need to take into consideration where each of these reports is coming from and determine whether it might be an individual ISP or routing issue that is impacting a specific group of players, as we saw with the routing issues of EU (Germany) connection quality to Cobalt/Miller I mentioned above.

If you have a sudden change in connection quality/server performance, you're welcome to post here or on twitter and tag me and the official Planetside2 account. But I would also ask that you submit a CS ticket as well. Doing so will help us more quickly identify any ISP issues that might be the cause, since doing so requires some back and forth, running traceroutes, etc. Which is what I/we had to do with the several players in Germany having connection issues of the past few days. But it fixed their problem.

In the meantime, we hear you and we will keep investigating.

Thanks,

Andy