Today at 9am NZT we took down the realm for the deployment of the new account system. This migration was expected to take around 4 hours.
The first thing that went wrong was that the migration took longer to run than it did on our test hardware. This extended the downtime for an extra hour past the point that we had budgeted for.
After the realm was brought back up around 2PM NZT, we found that many players were getting disconnected frequently. This was caused by crashes in one of the backend master servers that caused online account session information to be lost.
We spent around 15 minutes trying to investigate the causes of these crashes but were unable to immediately come up with any solutions so we decided to roll back the patch.
Unfortunately in this case, what would normally take a very short amount of time to roll back took a very long time due to the extensive database migrations that had occurred during deployment. The databases are very la... Read more
The first thing that went wrong was that the migration took longer to run than it did on our test hardware. This extended the downtime for an extra hour past the point that we had budgeted for.
After the realm was brought back up around 2PM NZT, we found that many players were getting disconnected frequently. This was caused by crashes in one of the backend master servers that caused online account session information to be lost.
We spent around 15 minutes trying to investigate the causes of these crashes but were unable to immediately come up with any solutions so we decided to roll back the patch.
Unfortunately in this case, what would normally take a very short amount of time to roll back took a very long time due to the extensive database migrations that had occurred during deployment. The databases are very la... Read more