devtrackers.gg
https://twitter.com/RiotPhroxzon/status/1771644991120789627
A post I've been wanting to write for a long time is about developing software at scale at Riot (at least on League Gameplay). There's a lot of implications due to our size that adds a lot of unique challenges.
There are often suggestions like: "Why don't you do 'X idea that seems easy', but in reality is a lot more difficult than it seems. And sometimes these are great ideas that we should do!
Sometimes these aren't so great ideas though... eg. "Why don't you do this thing!" which solves that problem, but has some massive downside, or would in reality take 2 years to build at scale.
Tackling these challenges is fun, but simultaneously difficult. I wanted to dig into some of them, not as an excuse for why we can't do/are slow to do some things, but mainly because it's fun to talk about/to help aspiring game developers/transparency for how working at Riot (at least on League Gameplay) functions.
1. Size
Let's just say as a lowball estimate, there are a few million lines of code in our codebase, thousands of product guidelines, design principles, art style guides, quality standards, etc.
Let's say any individual could realistically be an expert (you know the most about this thing) in ~5% of your area (eg. Design/Engineering, etc.) and be a contributor (you know enough to be able to positively contribute) in ~30% of it.
Simultaneously, the code/knowledge base is constantly being worked on/improved, so your understanding of what that system was/is doing is changing over time
There are definitely things across all disciplines that aren't up to modern standards, but even the best written game foundation is so large that it's unrealistic to know all of it
Because League is a big game, there are also a lot of protections that need to be built in/designed around to protect us from being a target, to relentlessly optimize software performance that would otherwise be totally fine for something at a medium scale
There are problems that only get surfaced when things reach a certain size, which is a great problem to have, but means that the experts need to be even more specialized at resolving these problems
One inefficient table read or service query could rack up to significant performance/monetary hits when this is multiplied by millions of people triggering that operation every day.
2. Specialization
People have areas of expertise, whether you're a Systems, Mechanics (eg. Champions), UX, Progression Designer, Rendering, Profiling, Services, Front-End Engineer, etc.
In the same way you wouldn't ask Messi to play goalkeeper (even though he probably could), some things can be very difficult to get movement on without the right person with the right expertise
What on paper could look like 7 Designers, could realistically only be the output of 3 Designers if they're not working on the right project for their skill specialization
Implementing something the wrong way, or making the wrong design choices could be potentially devastating; we have processes in place to make sure experts see things before they go in, but even then, mistakes can happen
Definitionally, there are always far less experts than there are contributors. The more an expert is reviewing, the less time they're spending on actually implementing and doing stuff that only they can do
3. Teams and Organization Management
At Riot, we operate within team boundaries
Teams own certain things within their area of expertise and have Rioters with various specializations to work on said stuff, follow up on broken features/bugs, make new things.
Even if you have the right specializations, there's also the institutional knowledge that has to be reckoned with. eg. If we hired Faker to work on our Live pod, he would not be performant right away
Playing and designing the game are 2 entirely different beasts and there's 15 years of past design decisions that you have to reckon with to be baseline performant
When priorities change, it can sometimes mean that the specializations on those teams no longer make sense, meaning either the priorities can't change effectively, or search needs to be undergone to other teams to find individuals to service those gaps in specialization
If Rioters move around to help a team on a gap, it can also mean that the thing they were originally working on falls behind
Some tangible examples are when we moved Designers around to help get Arena's first release over the line, carrying delays in work that Summoner's Rift Team or Champions team were able to do
We evaluate the tradeoffs of what these moves would mean, alternate ways of solving the problems and decided this was worth it
When it comes to finding which team should work on a certain thing, that is also a major challenge in itself. You need to know the right teams, have good relationships with other leaders and people to ask and know where to look to find which teams own different things
As an example, if we noticed Fog of War was suddenly non-performant and tanking our server performance, it could be a number of different teams that could theoretically solve that problem. So it comes down to negotiations between teams for which Rioters have the best specialization set and how that would tradeoff against the opportunities that they're currently working on, which can get quite complicated and takes time, but has to be done, because it means we can get things out to players faster and at a higher overall quality
An Example
So to use an example of "make players play 5 normal games before playing ranked" that I've been hearing recently. A great idea and one we've been discussing internally.
To make this happen though, you'd want:
Ranked team to propose the design for the change + the edge cases, the rollout plan. The ranked team would also consider whether this solution is the best thing to do vs other potential solutions that have different value, dependencies and costs depending on our staffing and skill specializations available (eg. improve our seeding, so it wouldn't matter what queue you played -> does this get our seeding algorithms to the quality level we would want) and has no Client/UX work related dependencies
League Data team to calculate the correct number of games, how this interacts with seeding models, etc.
One of the League services teams that handle ranked services to figure out how to calculate, store and query the information in the most efficient way, also future proofing
Client team to understand how to query and pass the data at different areas of the client
UX team to understand how the flows work (eg. do you prevent inviting from lobby if ineligible, where is the warning that says you can't queue together/what does it look like [is it on the queue text itself, or in the lobby], do you show the warning when they're in lobby and grey out the queue button, etc.)
Client team to implement and handle all the edge cases of the UX design
Build and Release teams (a lot of this gets automated)
If all goes well, despite the large number of teams involved, these dependencies and negotiations of work can happen pretty smoothly and we might be able to get it out within a few months, but for example if any particular team are all hands on deck for a higher priority project (eg. if the UX team was fully wrapped up with working on something like Season Start), then it can mean that this particular solution is a non-starter.
What on paper seems like a pretty simple idea, when it's broken down into its constituent parts can become pretty daunting to approach
Hopefully this was insightful, this wasn't my usual type of content, so let me know if you'd like to hear more/less of how game development tends to work!
TLDR: https://twitter.com/RiotPhroxzon/status/1771645726361080130
External link →Making a game at scale is not as easy as you think, but we try our best to put the best things out for players :)