devtrackers.gg
about 2 years
ago -
Rocket
-
Direct link
Intermittent issues with master server connections appear to be occurring. Team are working on it. No obvious cause, possibly similar to past outages.
Will post updates here and merge topics.
UPDATE: 8:16PM NZST Services being restarted. This might involve connectivity issues for up to 30 minutes. You can continue to keep playing, as it will reconnect and resync just fine when services restored. Apologies for the disruption.
UPDATE: 9:16PM NZST outage still ongoing. We have to reinstall the whole backend service so it’s a bit complicated. Will keep you posted. Again, apologies team are working as hard as they can to restore.
UPDATE: 10:18PM NZST confirm some users still having problems (myself including). Will keep you updated.
UPDATE: 11:50PM NZST we are still working the problem. Thought we had it, but it wasn't it. Everything is green across the board, so we are adding more logging to identify the problem. It's a near total (but not complete) outage. About 30% of normal utilization on the SQL, and players sometimes get through. So that makes it even more confounding. The next step here is to add more logging, deploy that service to the backend, so we can narrow down better where the problem is. Will keep you all updated.
UPDATE: 12:20AM NZST We are deploying a new update to our backend, just with more logging. We will also restart some of our backend services again, but with a very watchful eye. Some of the services might be down for the few who are connecting okay for periods of 20-30mins. For most they cannot connect at all still (unless offline).
UPDATE: 01:41AM NZST Still working at it. We've had some difficulty adding additional logging. We had to add logging changes, then deploy this to the backend - but that made things worse. We've now finally figured that out, so the additional logging should give us a clearer picture of the issue.
UPDATE: 02:51AM NZST Still working folks - apologies nothing further to report now sadly but will keep you posted. We can see some players are able to save their data and get details - so we will keep the service online for those users who are not having problems, while we debug further. For those unable to connect: apologies we are working as hard as we can.
UPDATE: 03:58AM NZST We've been adding a lot of logging to the game now as well, and testing local builds. This is because nothing unusual came up on our logging of the backend services - other than showing many of the services were doing nothing. We have escalated to Microsoft, who are looking into the issue as well. Currently we have even managed to reproduce it (not reliably) on our development environment. If what we saw is similar, it means that the connections are failing to even make it to our backend. While they have been doing this, I have been investigating other options if it becomes ongoing.
UPDATE: 07:29AM NZST So I'm the backup. While Microsoft & our engineers look for solutions (possible SSL cert issues), I am working on the backup plan to accelerate migration to decentralization.
UPDATE: 11:28AM NZST We have high confidence that we have found the issue, and are working on deploying a fix. this WILL require an emergency update to be deployed to all clients. Standby for more information about deployment of a new build. Once we have confirmed what the issue is we will give more detail. The solution is easy, but it will take an hour or two to deploy a new build on steam.
UPDATE: 12:05PM NZST Outage appears to be resolved. Please let us know how you get on. As it is an DNS issue, there may be some lingering problems.
CLOSED: 12:25PM NZST Outage has been confirmed resolved. Some users may need to wait or flush their DNS, while the records propagate.
17 November 2022 outage has been resolved. If still having issues, try Flushing your DNS, switch your DNS to google's, or wait for DNS record to propagate. It can take up to 24 hours at the extreme for DNS changes to make their way everywhere.
Will post updates here and merge topics.
UPDATE: 8:16PM NZST Services being restarted. This might involve connectivity issues for up to 30 minutes. You can continue to keep playing, as it will reconnect and resync just fine when services restored. Apologies for the disruption.
UPDATE: 9:16PM NZST outage still ongoing. We have to reinstall the whole backend service so it’s a bit complicated. Will keep you posted. Again, apologies team are working as hard as they can to restore.
UPDATE: 10:18PM NZST confirm some users still having problems (myself including). Will keep you updated.
UPDATE: 11:50PM NZST we are still working the problem. Thought we had it, but it wasn't it. Everything is green across the board, so we are adding more logging to identify the problem. It's a near total (but not complete) outage. About 30% of normal utilization on the SQL, and players sometimes get through. So that makes it even more confounding. The next step here is to add more logging, deploy that service to the backend, so we can narrow down better where the problem is. Will keep you all updated.
UPDATE: 12:20AM NZST We are deploying a new update to our backend, just with more logging. We will also restart some of our backend services again, but with a very watchful eye. Some of the services might be down for the few who are connecting okay for periods of 20-30mins. For most they cannot connect at all still (unless offline).
UPDATE: 01:41AM NZST Still working at it. We've had some difficulty adding additional logging. We had to add logging changes, then deploy this to the backend - but that made things worse. We've now finally figured that out, so the additional logging should give us a clearer picture of the issue.
UPDATE: 02:51AM NZST Still working folks - apologies nothing further to report now sadly but will keep you posted. We can see some players are able to save their data and get details - so we will keep the service online for those users who are not having problems, while we debug further. For those unable to connect: apologies we are working as hard as we can.
UPDATE: 03:58AM NZST We've been adding a lot of logging to the game now as well, and testing local builds. This is because nothing unusual came up on our logging of the backend services - other than showing many of the services were doing nothing. We have escalated to Microsoft, who are looking into the issue as well. Currently we have even managed to reproduce it (not reliably) on our development environment. If what we saw is similar, it means that the connections are failing to even make it to our backend. While they have been doing this, I have been investigating other options if it becomes ongoing.
UPDATE: 07:29AM NZST So I'm the backup. While Microsoft & our engineers look for solutions (possible SSL cert issues), I am working on the backup plan to accelerate migration to decentralization.
UPDATE: 11:28AM NZST We have high confidence that we have found the issue, and are working on deploying a fix. this WILL require an emergency update to be deployed to all clients. Standby for more information about deployment of a new build. Once we have confirmed what the issue is we will give more detail. The solution is easy, but it will take an hour or two to deploy a new build on steam.
UPDATE: 12:05PM NZST Outage appears to be resolved. Please let us know how you get on. As it is an DNS issue, there may be some lingering problems.
CLOSED: 12:25PM NZST Outage has been confirmed resolved. Some users may need to wait or flush their DNS, while the records propagate.