[2019] We Are League Game Engineers, AMA

No we do not have plans for west (or east) coast severs. We took the compromise of putting our servers in Chicago then doubling down on Riot Direct to reduce latency. Super cool blog on this if you havent read it: https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i

So, let's talk minion block. I get that it is supposed to exist, but right now it's incredibly frustrating and sometimes you just get stuck with no way out. Is there no way to lower the amount of body block in the game, because it's not really a fun mechanic?

The pathing system is one of the more challenging places to do engineering work. It's a complicated system, has the potential to be very expensive on both player machines and game servers, and has a direct impact on important feels. This makes it so we would need to be very confident that a change that we're considering would definitely be an improvement in order to put in the effort.

We can certainly all point to times where, as players, we've had a bad time with minion block. I know there have been some hacked together experiments like, for example, having different collision radii for allies vs. enemies. But none of those prototypes have "solved" the issues sufficiently for us to decide to then try to make them work in a non-hacky way, in order to ship them.

2 - How does Yuumi Q work with the cursor following?

We needed to make sure that it works well even with lag. Thus there's a delay between when the mouse is in a new position and when the missile starts chasing that position. Your game client sends up 3 recent mouse positions to the game server, and a timestamp for when they were sampled. Every 1/10th of a second or so it sends new positions.

The server then takes those 3 mouse positions and the time. It uses the most recent mouse sample that it has that is older than the lag-offset delay I mentioned above. So if it's been hearing quickly from the client, it will use a sample "from the right time". But if there's some lag, it will stick with whatever the most recent mouse sample that it's heard of.

The server then sends down to every game client the exact path that the missile follows, a little bit before the missile "actually moves", so everyone can draw it in the right place.

Unlike most other missiles, the game client isn't allowed to guess where Prowling Projectile will go. This is because all of my attempts to set up prediction guessed wrong most of the time. Therefore, we only draw the missile moving exactly where it was told to go.

3 - How did you get extra spell slots to work with Sylas ult? Just assigning each one before casting it?

For context, lots of "spells" that you would think of are actually made up of several spells, some of which are automatically cast by the game. For example, Lulu Whimsy has two different spells for whether you cast on an ally or enemy. Those extra spells are put into hidden "spell slots" that are kind of like the QWERDF slots, but players can't push a button to cast them.

Yeah, we've changed the best practice for "extra spell slots" for a number of reasons, and Sylas was a forcing function to clean up a bunch of cases that didn't handle it. Rather than just saying "Cast extra slot 4" in scripts, which requires you to go look somewhere else to figure out what spell that is, we expect scripts to say "Put LuluWhimsyEnemy in extra slot 4. Cast extra slot 4"

And Sylas was built to be very careful with his own use of extra slots to make sure that stolen ults get to use the extra slots freely.

What was the most difficult ability to make?

The hardest ones I've been around for have been Sylas ult, Neeko disguise passive, and Mordekaiser ult. All of them share one specific quality that makes them way harder to build than a typical spell. Each of them required special care to make sure that every other champion works with them.

For Sylas ult, we needed to get a much cleaner definition of "what is a spell?" It couldn't just be a miserable little pile of scripts.

For Neeko disguise, there were a ton of special cases for things like Aurelion Sol missiles, Graves auto attack, DJ Sona, etc. that required special handling to take the visuals and auto attacks without taking on the entire identity of the character.

For Mordekaiser, we specifically needed to think about what happens with every spell in the game when it's cast as you transition into/out of the death realm. We tried to come up with solutions that work correctly for most spells without any extra thought, but there's always exceptions.

13 - Did you ever fix the servers dying when the game reaches 25000 seconds of game time? It's not exactly something that's easy for me to check myself... Do you know what exactly is timing out that causes it?

Depending on the hardware we can run over 100 games on one physical machine. There is a single process called the local server manager that is responsible for all of those games. One of its duties is to kill off any game server it thinks is misbehaving. If a game runs at 100% cpu for too long or if it seems like its a ghost game running forever it will terminate the process. So if your game runs for too long you need to close out or the LSM will judge you.

When it first came out, there were some challenges with it deciding what height to draw at. Our game logic (almost) all takes place in 2D, but we then need to draw missiles at a height that makes them understandable, dodgeable, etc.

I added a new field to the "follow terrain height solver" that controls the maximum slope that a missile can follow. That prevented it from jumping up and down as it crossed over walls, down into the river, etc.

Correct, this mouse tracking was built specifically for Yuumi Q.

What you're describing sounds more to me like an issue somewhere between input latency and human perception. Any game engine needs to take into account delays between when a button is pressed and when it is reacted to. You can shift input samples around within a frame. Some games are drawing "last frame" while setting up "this frame", which leads to an extra frame of latency from your input. But also the human brain is bad at identifying exactly what's happening on extremely fast timelines, like the framerate of a video game. Sometimes the brain's prediction of "I'm going to do the thing" winds up dominating the perception of what's actually been done.

I'm guessing, Poluact, that you're experiencing a mix of both of these phenomena, making it so that if your mouse moved a lot in ~1/30 of a second or so, the game won't necessarily agree with you on exactly where it was when you pushed your key.

We used to house the PBE in Vegas but its currently in Chicago. You may experience a different ping as we are experimenting with hosting game servers in the cloud.

How did Riot respond initially to the bugs on Riven that became features, like doublecasting and the fast q?

With any champion and any bug, we have to make decisions like that about whether to fix something, or whether to even think of it as a bug anymore. I can't speak to exactly how we make those design decisions. But I have seen "bugs" specifically on Riven that happened because we fixed things that we considered systemic bugs, but we then needed to reimplement the behavior that Riven relied on. Specifically around animation cancelling.

Riven is aimed at a very specific combo/animation cancelling gameplay fantasy that means she stresses our animation and spell casting flow a lot. As a game engineer, I love having a champion like that in the game to make sure we're questioning our assumptions and building with an open mind.

When Riven e's into a wall from a certain distance, she will briefly clip into the wall before being pushed out, but this small window allows her to jump some massive walls, such as raptors to midlane. Why/how does this occur?

When doing a Move, there's a variety of options of how to decide where to land. For example Kled E1 can't go through walls at all, some spells go to the nearest pathable terrain along a straight line, and others can only go through a wall if you make it all the way through. It's up to the designer to decide which type of terrain interaction gives the feel they're looking for.

Riven for a while now has had a bug where her q will go backwards when you lock-on q while close to someone, is Riot aware of this?

I'm not sure, but I'll mention it to our QA to make sure we're tracking it. No guarantee we can/will fix it, but at least a human will check on it.

If you want to work as a game engineer, the most important things I would focus on are:

1. Collaboration/communication, particularly with non-technical creators. We constantly work with designers and artists. It's not important that they know exactly what the computer is doing. But they need to know how the computer's behavior will affect their ability to make something awesome.
2. Understanding of how game engines work, at a high level, across a lot of technology areas. For example, rendering, animation, scripting, networking, physics, sound, VFX, etc. I'm not saying that you need to know every detail about how all of these systems work. But having a reasonable understanding of the interactions between them can help you speak intelligently about how things are built.
3. Some technology area that you specialize in. Choose one of those areas from Point 2, and make sure that you understand deeply what's going on. Let's say you want to specialize in VFX. A great exercise is to ask yourself, "Could I build a VFX system from scratch?" I don't care necessarily whether you could rival Unreal's particle systems. But I do want to know that you understand all the pieces that go into a functional VFX system.

Without saying anything compromising, what about the games design or architecture do you think lends itself to being an environment that seems difficult to compromise?

The genre itself lends itself well to giving fewer tools for hackers to compromise. For example, we do not give game clients the location of a player when they are in fog of war. We do have our own proprietary solutions to other hacking options but all I can really talk about is that it's a never ending battle that requires a dedicated team.

Actually I'm not sure. I will say that this is the sort of bug we often spend a while digging into. Subtle timing issues and interactions between spells can be challenging to figure out.

I can say that Ahri Q uses an old pattern that we've been moving away from:

FaceDirection

GetPointByUnitFacing

This can misbehave if for any reason the champion refuses to face a given direction. For example, Yuumi is forced to face a given direction when casting R, and there was a bug where she would always flash in that direction, rather than where you tried to flash to. I rewrote Flash to not rely on setting the facing of the unit in order to choose a target position.

I'm not sure exactly how much truth there is there. It is pretty common (in League development and software development in general) to copy/paste sections of code or script that do something similar to what you want to do. So I wouldn't be surprised if Thresh Q was built by copying some elements of each of those spells.

Minions affecting player pathing at the high level as a game feature is a good thing and something we will probably keep in the long term:

-It adds context and different playing fields to champ ion combat
-It helps to make the world feel more physically real (things get really floaty fast when everyone just glides around from point to point)

That said - there are a few cases that lead to a shocking/unintuitive/frustrating caused by our current execution of pathing around minions
-Minions are forming a "wall" that doesn't really read visually as a full block
-Minions shifted while I was in progress, so my initial move would have brought me through but now it fails. This is especially bad if things keep shifting causing my character to get trapped and repeatedly change paths.

In conclusion - fundamentally minions will probably affect pathing in the long term but we could do work in the future to make it more predictable, consistent, and match your expectations as a player.

What engine is League based on? 100% homebrew?

League is built in the League engine, which is a custom engine built in C++.

Do you have level editors?

In the past few years we've built up a level editor, specifically aimed at making custom game modes easier to do. Back when we did the Summoner's Rift Update, there was a complicated process of black magic and Maya exporting that needed to be done to update the map. But now, with Maps Mountain, we can do relatively quick iterations on map things.

Would it be possible to open League for modding by players like Warcraft 3 or Dota 2 does?

Support for modding is a huge decision for a studio to undertake. It requires a lot of extra effort and support to make something that's stable enough for external creators to work. It's pretty easy, if you know what you're doing, to make League crash by doing bad stuff with data. And that's ok because our content creators will work with their engineering/tech art support to make sure that the things that they want to do work, are stable, don't crash, etc. But if we were truly supporting modding, we would need to take the time to sand down all those rough edges and make a stable playground to work in. Not to mention documentation, adding support for features that we speculate modders might want to use, rather than just the things we know we're going to use, change/version management, anti-cheat implications, server hardware support, etc.

That type of work, particularly on an engine that wasn't built with modding in mind from day 1, can take enormous amounts of time.

Ahhh. Subtle timing-based stuff like this definitely happens in a lot of places. For example Shen, Gragas, J4 having their CC interact with flash being cast in the middle of their dashes could be called a bug, but also leads to awesome plays.

So basically I'd say I'm not surprised that there are weird timing interactions with Riven's dashes.

It's all super exploratory right now and what we find will effect our decisions. Eg. If the latency or performance hitching is worse then we may be ok with all of our alternate modes and custom games hosted there but not ranked games.

The biggest advantage is having the convenience of quickly scaling up/down as we need. Downsides are not having the direct control over all aspects of our players' experience which is something we care about deeply.

That would be awesome but would have implications on match making quality and times. Not something we are talking about at this early stage though

It's pretty common to put "debug prints" in scripts to figure out what's going on. For example "I wonder whether we're actually executing this part of the script, so I'll make it say something if we do."

But sometimes you forget to remove those debug prints, once you're done fixing the bug or adding the feature.

We've made some changes in recent times to make it less likely that those debug prints actually do anything on Live builds, by making most versions of printing to chat only happen on internal builds. That said, things like Draven passive do want to have scripted printing to chat, so we can't turn off printing to chat entirely. We just need to make sure that creators easily get the behavior they want.

IIRC, the "taunty" debug print was in a "raw lua script" as opposed to a "blockbuilder script", which means it was in a less common type of script, and thus its behavior was less well understood.

On Summoner's Rift, (0, 0, 0) is a little below the ground just off the map to the southwest. Anytime I see a bug with things heading down and to the left, the first thing I check is whether there's some vector that might have accidentally been set to the origin.

What exactly was going on under the hood in the "Xerath uses Q at a very specific spot and it hits the entire map" bug?

That's one of several bugs we've run into with skillshots going "up". We have a lot of scripting blocks that let you make sure you're only working in XZ, ignoring the Y component. But there are also things that operate in XYZ. If you do something like asking "how far away along this line is an enemy" and that line is pointing straight up, the answer will be "not very far away", since both units are standing on the ground.

That's a slight oversimplification of the exact math problem, but it's the basic idea.

We've hardened a bunch of code paths against using vectors that only have Y components to avoid bugs like this.

(Making sure this post gets Riot flair. Shaqfuuuuu sits next to me, but apparently something happened to his reddit flair)

My strongest advice is to learn your fundamentals from how CPUs behave to what the compiler and OS is doing with your code. You'll be able to apply that knowledge through your whole career even though technology and game making norms change. Secondly, something I greatly prize is people who have their own personal projects. It shows a passion for the work and a willingness to see things through.

League has the most diverse of technical backgrounds of any game engineering team I've worked on. We have game industry veterans with 25+ yrs exp to non games people in their first role. The common element would be the desire to work collaboratively with other to make really great experiences (also being good at the codings).

What led to the discovery of the Shurima Shuffle, was it intentionally made to exist or was it an accident? Were you guys considering ever fixing it if it wasnt?

I'm not sure exactly, but I know that these sorts of subtle timing interactions happen all the time.

How'd you code the tech for Azir and his soldiers?

There's a script event call PreLaunchAttack that lets you do reactions to when an auto-attack is about to be fired. When that happens, Azir gets to check whether there are any soldiers nearby, and if so, switches to casting a spell that tells them to cast a spell, instead of doing his auto-attack. Some other examples that use that system include Graves, Rakan (to have different auto attacks at different ranges), and Kalista.

Why did Children of The Grave bug out so much? Did you guys ever consider rescripting it before the gameplay team decided to VGU him?

The biggest thing I'd say was that it was built a long time ago when we had a different relationship between design and engineering. Back then, designers were largely left on their own to prototype and figure out how to implement things, and engineers would help towards the end of the project to do whatever we absolutely needed to to get the champion out the door. Honestly that spell had a lot in common with Neeko disguise or Sylas ult in that it had complicated interactions with every single other champion. But as a team, we didn't build it with a systematic approach to make future bugs less likely.

Now we have a lot of conversations early in the process of building a champion to make sure we all agree that we're building it in the best way we can.

You humor has been noted. Appropriate measure have been taken /s

What we did was change the Clash design so it better suited how league works

Nasus would weigh in on this but there's minions up top that won't farm themselves

We do run a few third party libraries on other threads and do small amounts of work there already. The most work done on different threads is the particle simulation - that's pretty good, but we can still do a bit better there. As for porting game logic to more threads, that's quite tricky but definitely something we're thinking about for the future. As for multithreading the renderer, as we're still Dx9, that's not really an option.

The other thing we need to consider is min and mid spec support. There is no point making the game require 16 threads as we'd make the game run too slow on the lower spec machines. As 8 core machines become more popular, we'll be able to utilise them more, but for now we need to build for our player base.

Not my area of expertise but I understand the internet packet journey from Europe to our Chicago servers is like a couple of lonely hobbits trying to make it to Mordor with Gollum trying to ruin your day.

This idea popped in my head recently after observing LCS pro Froggen do some in-house scrims the other day on his stream.

It was supposed to be a practice environment but it felt like... Your average 5-man league game with a lot of back and forth communication - This is the picture pros see in their games. It's hard to explain but the fact that observers see what happens on the entire map, clouds observers judgement of one or the other action.

In order to minimize that, a clear indication would be nice to have (since we're all about clarity ever since jungle timers). It would allow people to understand what's going on and learn from it from their own games. It would have various uses in learning or commentating.

I also watch a lot of Froggen on stream - one of my favorite pro players :)

We have talked about this in the past and it is something I'd like us to explore. I know one of our rendering experts did some experimenting a while ago but to my knowledge it's not been something any team has gone further with. I think its worth revisiting this or some variant on better pro game visibility information.

Thankyou for the suggestion!