devtrackers.gg
Hey everyone,
First, I want to apologize to our EU community, since this took place right in the middle of their prime-time. Please know that we try to avoid this at all costs, unless the severity warrants it. We will continue to avoid this going forward.
Changes included in the 130pm (Pacific) hotfix:
- After the 6am hotfix this morning, all Bastion component progress prior to the servers coming down was lost, along with the associated resources. Expedition progress was also lost. More details below.
- A medium-frequency client crash was fixed.
Of the two changes, the Bastion component crafting and Expedition progress loss were the reason we decided to perform an emergency hotfix at this time.
So how the hell did we miss such an obvious bug?
The way our live infrastructure is configured, each region utilizes dedicated hardware for the game servers. However, we have several "central" services that essentially store and sync data that needs to be tracked between all regions. During our testing of the War Asset system prior to launch this week, we verified many different scenarios to confirm that progress would not be lost when servers were brought down.
However, for this morning's update, in order to keep downtime to a minimum, we did not restart one of the necessary central services. We felt that it was not required - we were wrong. As we all now know, that particular central service tracks the component crafting and Expedition progress. The emergency hotfix we just performed included a reset of that service and as far as we can see, all progress prior to this morning's 6am (Pacific) update has been restored.
We've added this to our run book for future publishes, so we can avoid this oversight in the future.
Again, we are sorry to everyone for the inconvenience of having their component crafting and Expedition progress temporarily reset earlier today, and then getting booted during their primetime for the emergency maintenance a short time go to correct it.
Thanks,
Andy
External link →