[2021] Extremely Slow Queue Processing

UPDATE/TL;DR: Queue currently fixed. There was an hour of it going super slowly. We will make sure this never happens again. See below updates for notes about current realm stability.

ORIGINAL POST: When the Ulstatimatum league started this morning, it was immediately apparent that the login queue was moving quite slowly. We are investigating this, and so far it appears that the reason is that this league's character migrations (which are a process that runs when a character logs in, to convert it to the new internal version) are much slower than normal.

Users are getting in, but it's going to take a while for the queue to clear and we're very sorry about that. We're acutely aware that a similar problem occurred last league launch and we thought we had resolved it.

Queue processing should speed up as more characters are converted, and we are trying to find other solutions that will help in the meantime.

Once again, we're very sorry about the delayed start to the league for most users. We will make sure that this never happens again.

We will update this thread as more information is known!

EDIT: We have a plan! This may result in people not having past league progress in Standard until we can catch up with that, but should massively speed up the queue for people logging in to Ultimatum (which is 99% of users right now). Will keep you updated.

EDIT2: Okay, so that plan sped up the queue by a lot. We're keeping an eye on stuff very closely .

EDIT3: We have been investigating some realm stability issues that trigger when there are a lot of users online. Our current plan to resolve this is to downgrade the database version we are using to the one that was stable for last league launch. We did stability testing on the live realm over the last week and also some pretty extreme load-testing with this new version before deploying it, but something is certainly up. Will update when we have more information.

EDIT4: We are now performing the change mentioned in Edit3.

EDIT5: Sigh, that made no difference. We have identified another server code change that is different in 3.14 and might cause problems in rare circumstances (which might actually be "all the time") and will revert that change to see if it fixes it. I want to emphasise that these changes have been load-tested before deployment, so we have no explanation for why they are failing under the load of real users.

EDIT6: Deploying the change mentioned in Edit5. The issue has occurred once since that point, so we will keep looking.

EDIT7: We're still looking for the cause of the server instability.

EDIT8: https://i.imgur.com/a9Qn6If.jpg

EDIT9: Okay we fixed it. That took 13 hours -_-