Original Post — Direct link

I understand that there's issues with server capacity and that turning off practice tool helps with that, but also practice tool is really important to the actual purpose of pbe. Just having it be disabled for a week any time new content comes out makes it extremely difficult for us actually find out how things work in detail and check for bugs that would be hard to notice in the middle of a real game. I feel like there might be some kind of compromise that could be reached where like, for an hour once a day in the middle of the night, practice tool is enabled even if it means turning off normal games for that time period.

That would be a 4% reduction in the amount of real games that get played, or theoretically even less if its scheduled at the time when fewest people are playing. In exchange for that, a bunch of stuff that can only realistically be tested in practice mode goes from not being tested at all to being tested at least a little bit. To me that seems like a very worthwhile tradeoff.

I thought this new thing of disabling practice tool was gonna be a one-off thing for yone's pbe cycle, but it seems to be the new status quo for now content coming to pbe, and I think its a really bad idea to just leave it like this indefinitely. Whether its the solution I'm suggesting, or something else, I think this is a problem that needs to actually be solved at some point.

External link →
almost 4 years ago - /u/Porosite - Direct link

I can share some knowledge around this. For context I'm a release manager. We're the ones that manage the patches, including doing PBE maintenance on the weekdays to get the new builds out, and have a hand in handling the queues. I'm not the one running 10.21 on PBE right now, I'm currently managing 10.20 on Live.

I guess first, I'll be clear on where we get the most value from PBE. Our highest priority queue is SR Draft because it gives multitudes more data than any other queue. We have robust internal and external partner testing that works great for testing individual new items, champs, mechanic changes, etc. but given the complexity of the game, there is a ton of value from just having 10 random champs play in a random way. Stability of the game is of critical priority, and having thousands of players play with different combinations of content is way more efficient at finding those 1% game crashes than we our with dedicated testers. 1% will rarely happen with internal tests, but 1% on Live impacts thousands of players per day.

Players testing in practice tool is basically ad hoc testing. There is value in this. One of the more common ways I see players develop bug reports is playing an actual game, noticing a potential bug, and then investigating it solo in Practice Tool. I do want to emphasize that this type of testing does overlap with testing we already do however, so it is lower priority for us if we have to decide between letting players do this versus the above. Even in a world where we could enable Practice Tool during popular points in the patch cycle, we wouldn't disable 5v5 testing to free capacity for more Practice Tool games because every game of 5v5 is of much more value to wholistic stability.

Now that being said, Practice Tool is disabled specifically because if it wasn't, we'd hit active game capacity issues that would snowball into PBE itself becoming unstable, specifically with being able to start new games consistently. I did a more complete write up here if you want to know some details.

As for why single player Custom Games are enabled, I only caught fringe parts of the conversation but I believe they kept the custom game player cap at 1 because the player population is in a middle band between PBE having issues and it being safe to enable Practice Tool. Practice Tool for obvious reasons is a much more popular queue than Customs for single player testing. Just a guess on the why, but less players are using single Custom Games than Practice Tool even when PT is disabled because it's not easy to try things out. Given that it's not yet safe to reenable PT, we can at least test the waters with Customs and at least give you that to play around on.

Now why no local Customs or PT? Simple answer is because League as a game isn't contained solely within your computer. A large portion of the game is hosted server side. Why was it designed to be hosted server side? I can't say for certain, cuz I didn't design how the game works. I can definitely say security plays a factor though. Also purely a guess, but I'd also wager that when the game was being made a decade ago, they determined there wasn't enough value for an online 5v5 game with no local multiplayer feature to have an offline mode. Move forward to now, for the cost it would take to make league operate locally without a game server, I still don't think it's worth it. There are plenty of more valuable things that League can improve with those resources.

To the last topic of why is the mitigation of disabling Customs and PT so common nowadays when it hasn't been for the history of PBE... I haven't had a chance to really dig into the data, but there have been a number of shifts on the PBE in the last year that are definitely contributing. The core is that the number of PBE players is higher than it has been historically. With the launch of TFT last summer, the honor 3 restriction got removed to let TFT only players also play on PBE (because TFT needs testing too!). This makes PBE accounts much easier to make, enabling more players from Live to simply sign in to PBE when a new patch drops to try the new skins, the new champs, etc. for a few hours. Most will likely just get on, and hop into PT because they just want to check out the new content, but not have to deal with PBE queues (which is a whole different not easily fixed issue).

As time has gone on, more and more players make a PBE account, and get used to signing in to try stuff out. With (I think) Lillia, we hit that breaking point where for the first few days, the majority of the population was just on to try Lillia in PT, with additional server load coming from a high population playing Nexus Blitz. And then Yone hits PBE, and PT is disabled for almost a week. And on and on. PBE gets more popular for checking out new content, we encounter more Practice Tool issues.

The current state of things is that during normal patches and during the second week of hype patches on PBE, the server load and game capacity are well within healthy limits. But when we hit conditions where we have to disable PT, because of how inefficient practice tool is for both server cpu load and managing quantity of active games, the amount of servers we'd have to spin up to support that would be incredibly expensive and painful. I'm talking 5 or 10 times the cost as a low spitball. Not something easily budgeted for.

almost 4 years ago - /u/Porosite - Direct link

Originally posted by Porosite

I can share some knowledge around this. For context I'm a release manager. We're the ones that manage the patches, including doing PBE maintenance on the weekdays to get the new builds out, and have a hand in handling the queues. I'm not the one running 10.21 on PBE right now, I'm currently managing 10.20 on Live.

I guess first, I'll be clear on where we get the most value from PBE. Our highest priority queue is SR Draft because it gives multitudes more data than any other queue. We have robust internal and external partner testing that works great for testing individual new items, champs, mechanic changes, etc. but given the complexity of the game, there is a ton of value from just having 10 random champs play in a random way. Stability of the game is of critical priority, and having thousands of players play with different combinations of content is way more efficient at finding those 1% game crashes than we our with dedicated testers. 1% will rarely happen with internal tests, but 1% on Live impacts thousands of players per day.

Players testing in practice tool is basically ad hoc testing. There is value in this. One of the more common ways I see players develop bug reports is playing an actual game, noticing a potential bug, and then investigating it solo in Practice Tool. I do want to emphasize that this type of testing does overlap with testing we already do however, so it is lower priority for us if we have to decide between letting players do this versus the above. Even in a world where we could enable Practice Tool during popular points in the patch cycle, we wouldn't disable 5v5 testing to free capacity for more Practice Tool games because every game of 5v5 is of much more value to wholistic stability.

Now that being said, Practice Tool is disabled specifically because if it wasn't, we'd hit active game capacity issues that would snowball into PBE itself becoming unstable, specifically with being able to start new games consistently. I did a more complete write up here if you want to know some details.

As for why single player Custom Games are enabled, I only caught fringe parts of the conversation but I believe they kept the custom game player cap at 1 because the player population is in a middle band between PBE having issues and it being safe to enable Practice Tool. Practice Tool for obvious reasons is a much more popular queue than Customs for single player testing. Just a guess on the why, but less players are using single Custom Games than Practice Tool even when PT is disabled because it's not easy to try things out. Given that it's not yet safe to reenable PT, we can at least test the waters with Customs and at least give you that to play around on.

Now why no local Customs or PT? Simple answer is because League as a game isn't contained solely within your computer. A large portion of the game is hosted server side. Why was it designed to be hosted server side? I can't say for certain, cuz I didn't design how the game works. I can definitely say security plays a factor though. Also purely a guess, but I'd also wager that when the game was being made a decade ago, they determined there wasn't enough value for an online 5v5 game with no local multiplayer feature to have an offline mode. Move forward to now, for the cost it would take to make league operate locally without a game server, I still don't think it's worth it. There are plenty of more valuable things that League can improve with those resources.

To the last topic of why is the mitigation of disabling Customs and PT so common nowadays when it hasn't been for the history of PBE... I haven't had a chance to really dig into the data, but there have been a number of shifts on the PBE in the last year that are definitely contributing. The core is that the number of PBE players is higher than it has been historically. With the launch of TFT last summer, the honor 3 restriction got removed to let TFT only players also play on PBE (because TFT needs testing too!). This makes PBE accounts much easier to make, enabling more players from Live to simply sign in to PBE when a new patch drops to try the new skins, the new champs, etc. for a few hours. Most will likely just get on, and hop into PT because they just want to check out the new content, but not have to deal with PBE queues (which is a whole different not easily fixed issue).

As time has gone on, more and more players make a PBE account, and get used to signing in to try stuff out. With (I think) Lillia, we hit that breaking point where for the first few days, the majority of the population was just on to try Lillia in PT, with additional server load coming from a high population playing Nexus Blitz. And then Yone hits PBE, and PT is disabled for almost a week. And on and on. PBE gets more popular for checking out new content, we encounter more Practice Tool issues.

The current state of things is that during normal patches and during the second week of hype patches on PBE, the server load and game capacity are well within healthy limits. But when we hit conditions where we have to disable PT, because of how inefficient practice tool is for both server cpu load and managing quantity of active games, the amount of servers we'd have to spin up to support that would be incredibly expensive and painful. I'm talking 5 or 10 times the cost as a low spitball. Not something easily budgeted for.

You know I always think I'm just gonna write a few sentences to respond.

almost 4 years ago - /u/Porosite - Direct link

Originally posted by Bro_miscuous

Why can't we have a practice tool "queue" in PBE? So basically your game can only start when there's an empty slot (and say you have 100 slots). I wouldn't mind waiting 15 minutes for a slot if I could test the few interactions or bugs I want to.

That would be unique tech for pbe. There are no dedicated engineers for PBE, so we'd need to write some kind of proposal that asks for a budget to make that. We don't currently have a use for that kind of queue on Live though, so it's a very hard sell. With PBE being just a testing environment for Live, generally we try to keep parity between PBE and alive, and trying to make something custom just for PBE doesn't have a strong return on investment when we're talking about new unique features. The PBE Draft mode without bans is about as fancy as we've gotten as far as unique modes goes.