Really wishing servers hadn't burnt down right about now. Wasted so much time for this crap. Spotter was in position and DC'ed the second he got there.
Also now physically impossible to re-run the OP.
[Keeping the details light on the actual op for various reasons, sorry]
External link →The update gave us a lot of valuable new data and situations to learn from. And one of those is urgent needs for the game to have a larger and more robust server infrastructure. Massive queues, unable to connect, disconnects, regions going offline many times, voice chat problems, and ect...
This is all with just 2,000 - 3,000 players. Nearly everyone is expecting to see far more players then this when the game goes full 1.0 release. Even if it was just doubled the amount as we have gotten, say just 6,000 players. It will be a catastrophe, as there is no way the servers will be able to handle it. There is no way the playstyle of the game can handle it. Hell we're already struggling with just trying to get supplies to the frontlines. As well as the logi hubs are being bottlenecked simply because there are too many players in the back line trying to do logi.
Personally I think this is a major problem, and one the devs should focus on for the next few months.
You are right that there have been scaling issues, especially with queues and voice chat problems. To solve the queues, we just need to improve performance to support more players per region and have a larger world (more regions), and looking into multiple shards even.
In terms of the increased disconnects and offline regions we've had over the last 2-3 days, this has really been unexpected. To shed some light on this, we have been hosting our servers on OVH for over 3 years now and while their reliability has not been perfect it has been overall reasonable stable. This week, OVH had a major disaster with one of their data centers and they have been scrambling to recover from the situation. You can read about it here: https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B20NU. Games like Rust have lost days of player progress for some of their servers hosted with OVH as a result of this disaster.
While our servers were not in that datacenters, their recovery efforts seems to have caused unexpected latency and disconnects for us. It just started hitting us these last few days. We migrated the servers that were experiencing the worst issues over to others to mitigate the issues. However, there has still been continuing fallout but we hope things will settle for OVH soon. We have been on call day and night trying to put out fires this week and are frankly exhausted from the disruption this has caused.
We are truly sorry for the gameplay issues that this has caused for players, especially those that have invested in major operations that have been disrupted. If there are continuing issues, we are going to be looking to move our infrastructure to a new provider and we will also look into better ways for gameplay recovering when problems do occur in the future (because server problems are ultimately inevitable no matter how robust a provider you have).
Is there anything that we the players can do to also help with the situation. I know there isn't really anything we players can do. But an idea has been tossed around every once in awhile for a form of subscription support system. Where players can send a monthly payment to help support the game. Like giving extra funds to help improve the server infrastructure system. I know some players like myself would do so and wouldn't ask for any kind of reward or special "things" like other subscriptions has. Just seeing improvement on the backline systems of the game regarding this issue will be enough for us.
Or do you feel that implementing such a system just wouldn't have any real impact as not enough players would engage into a subscription support system?
Thanks for the idea, but the constraint for this specific issue we've been seeing these past few days isn't monetary. It has been a problem at scale with our server host and has impacted thousands of websites and services across the internet, not just Foxhole. We just have to wait for OVH to sort things out and then make sure we're doing everything on our end to mitigate problems as they occur. We hope these recent issues will be short lived.
Besides, we wouldn't be comfortable accepting additional payment for something that should work in the first place. It's our responsibility to have Foxhole work smoothly, not players'.
In terms of the longer term issue of scaling up to support more players more effectively (i.e less queues), we do intend to invest in optimizing our server engine further and finding other ways to increazse capacity. While the population will likely stabilize at a lower count than it is now in the coming months, there will always be big spikes for future big updates that we need to be ready for. It was short sighted for us to have not put further effort into this sooner.
As an additional note, the limiting factor of our capacity per region specifically, isn't the cost of servers. We already rent the highest end hardware that's available for this type of application. Foxhole runs on Unreal Engine 4, which is not optimized for supporting hundreds of players. To put things into perspective, Fortnite and PubG which also use this engine only support 100 players. We had to customize the engine to support 200 while allowing players to build thousands of structures in the world.
devtrackers.gg