[2022] True Skill 2 Is a Potential Problem

Super interesting take, and a great read!

I just want to be 100% clear, VALORANT does not use true skill 2 - we made our matchmaker and MMR system.(which by the way is technically two different things/systems).

I want to start out that your post seems like it's coming from a high level player, which technically is has the worst impact when it comes to matchmaking issues. I don't doubt the issues you faced, and in fact experienced the same match-making woes when the industry was trying to figure out how to do this(I was in early esports). I also want to talk about "the employee" you referenced, and what you are eluding too. Lastly, I feel like there is a "I feel this happened" rather then data or arguments to really support it. I understand you don't have access to that data, and at the end of the day feedback of "This feels bad" is super important and is the main source of player to dev feedback; I just wanted to point this out because while your thoughts on how the system work is close but not completely correct.

(I'm trying to word this as straight forward as possible, while also not sounding like an ass. I don't expect people to fully understand how our matchmaker works, or how true skill 2 works. At the same time I think it helps to be honest where there are beliefs that the match maker is doing things it actually isn't, because that perpetuates false beliefs in the community. My goal isn't to say "You're Wrong", because this discussion is important. My goal is to say "This is how it's working, this is why we feel it isn't exactly as you say, and I want to figure out why you think it feels this way to prevent this negative experience in the future".)

Okay so! The experience you talked about in Halo is actually true of any game that has high ranks(even Valorant). It's not necessarily only about the total player population, it's about how far apart the top rank players are from each other. For example; the #1 Valorant player could have 100 MMR more than the #2 player, and be expected to win 70% more of the time(all numbers made up). Now think about the gap between the #1 player and the #100 player. What if none of the top 99 players are queueing at the same time as the #1 player? Should you not find that player a match? What if that player is queuing at 2am and you have a pretty good idea that no top 100 player will be queuing for hours potentially? Does that player just not get to play? It doesn't matter how many players there are in the community, it's the skill gap between players that makes it harder to make matches(usually seen at the highest ranks of any game). Then, after the gap between the high rank players is a problem, that's where the total player count becomes an issue. Because now when that top player queues up, if you want them to be able to play a match, you need to find them a game using the players you do have.

I'm guessing, based on your experience, that you were probably a pretty high ranked Halo Player. I actually looked up some of the worst matchmaking culprits in halo 5, it looks like gold to onyx was possible if it was like 3am after the game had a smaller community. That's the trade off you have for games where there are extremely skilled players with larger skill gaps between them, on top of a smaller community of players. If you were one of the top players I'm guessing, more often then not, the match maker had a hard time finding other players at your skill level. While it may feel unfair that you feel like it's Lebron vs. a highschool team, the only solution would be not to let lebron play at all. Unfortunately, especially in smaller communities, you have to decide when a match must be made(we are combating this with 5 stacks right now, which is why you've seen so many updates about them and queue time). Imagine if you never found a match for that high rank player, and they just quit the game or give up; You may be okay with 15-30min queue times but they are not(usually you can use data to figure out when players leave the queue to find out how long players put up with being in queue). Now that is one less high rank player able to be used for match making, that could potentially be gone forever. Slowly you start forcing players out of your game, and making match queue times worse, because the high rank players bleed out due to long queue times(they are hit the hardest in queue times so if they can't find a match they will just leave). At the end of the day PvP games need people to PvP against, it's why PvP games are also a hard market for smaller indie studios to break into; you need players to populate your community to have PvP.

This issue isn't exclusive to true skill 2, any matchmaker will have someone behind it deciding if a match between a high rank/mid rank player can happen. I think, what makes it difficult in talking about ranked modes in games, is that they all suffer from similar issues which you don't experience unless you play in those communities. This isn't to say we shouldn't be trying to improve rank, but your post is an example of this. I think, if I'm understanding your post right, you are saying that Valorant is starting to feel like Halo 5; and you may have assumed that we use true skill 2. Well we don't use true skill 2, but you believe we have similar issues to what you experienced in Halo. That's a great example of how player population, and skill gaps, especially at the high ranks is something that match makers can't solve and there is no great catch-all solution(if there was it would rapidly make its way through the game industry). Again, this isn't an excuse to throw our hands up and give up, but it's just to point out that the game industry as a whole has not been able to solve it and is trying constantly. We can't make more players high skill, or make more people play the game, so we have to choose between making a match with what we have or not letting those outlier players play at all.

I really want to emphasize, True Skill 2 doesn't uniquely require a large player base to work, ALL match makers perform better with higher population and lower skill gaps. True Skill 2 is actually one of the best at handling smaller communities, because it is one of the most accurate matchmakers around(where other matchmakers actually create worse matches, if you don't believe me go play a call of duty before Black Ops 2 or any old FPS title).

Now the person you are talking about joining Riot is I'm assuming Josh Menke. Josh is amazing, super intelligent and has some amazing talks you can google about match makers. While Josh was highly involved with True Skill 2, I want to call out that True Skill 2 was made by an amazing group of people at Microsoft. They have a team, with PHD's way smarter then me, who have been working on match making and MMR systems for over a decade. This group, in my eyes, is the pinnacle of knowledge in the game industry for match makers. Unfortunately, it takes a lot of time, complex math, and data gathering to truly dive into the match making world. Many games do not have a team of PHD's, let alone even 1 PHD, or even mathematicians to fully implement their match maker. It's pretty rare to be "a match maker specialist" in the game industry(I mean I've never even seen a college course on it, much less a degree). So that's why I hold True Skill 2, and the team at Microsoft who works on it, so highly. The amount of learnings, and depth, of that system is just impressive to me. So to recap, Josh definitely is important and one of the pioneers of match makers in the industry, but he's not the only one that holds the key to the kingdom. Also Josh does not work on Valorant, but I do talk to him a bunch because he has so much experience and knowledge(in fact I reached out to him to talk about this post and ask him questions!).

The other thing to truly understand True Skill 2, and why it's a good match maker, is to understand how accurate it is. A match maker is only as good as it is effective. True Skill 2, while I'm unsure of the exact effectiveness, I know is one of the most accurate I've seen. It's extremely impressive to have an accurate match maker, and while I am not going to lie and pull data out of thin air, I only remember when researching True Skill 2 it's very impressive. Again, that's just my word but I'm sure you can look around google to find out more about it. Match makers aren't doing a ton different, they all operate in very similar ways, some are just better at the math and therefore have better results. Sorry, this is a vague paragraph, but I just want to call out that most match makers are the same(more or less) and True Skill 2 is just one of the most accurate I've seen.

Lastly, you brought up that True Skill 2 makes matches off of trying to create a 50:50 match. This is actually true of almost all match makers, they all try to make a 50:50 match and that's the goal when implementing a match maker in your game. Also, while you can have a Lebron vs. High School team, there are actually settings to prevent this(again most match makers should have this). How it usually works is there is a skill gap that tries to be enforced, but usually the skill gap will widen the longer a player is in queue(again this is to solve players never finding a match). This isn't unique to True Skill 2, the longer you are in queue the wider rank disparity you may see in your match. So match makers actually try to prevent the thing you were experiencing, but this kind of ties back to higher rank players will experience this the most.

The other thing to think about is that players are often on their own "journey" in terms of their ranked experience. It doesn't matter if it's your rank badge, or a rank system that shows your MMR, a player that plays 10 games is further from where they will end up then a player that has played 100+. This is more of a reason why players see various ranks, or skills of players in their games, but leads to a ton of various feedback points because 1 player could be on their way up while another down. This, along with people having good or bad days, smurfs, the human element, etc. can really change your ranked experience day to day. Again, this is something that doesn't have a catch-all solution, but it can lead to feelings or beliefs in how the match maker is working. I've had pro players complain to me someone doesn't belong in Radiant, then when I investigate their match history the one time they played with that pro they just had a bad game(and proceeded to carry their next 5 games). This is honestly one of the hardest parts about parsing ranked feedback, but again that doesn't mean we should ignore because there is a reason people feel X way when they give feedback.

Oh I also wanted to point out distribution, because it's a hot topic. Distribution is tricky because it skews more negative in most systems, this is because of players that play their placements then never play ranked again. If, after placements, we are often saying "Hey we think you are Silver/Gold but we aren't sure so you are going to start bronze/low silver." those players need to play games to for sure show us they belong in gold and to climb out of bronze. If they end up not playing, they just sit in bronze and inflate the population. Because you never can place above diamond, also because more players are lower rank then Diamond+, the lower ranks are the ones where these accounts sit. This isn't the majority by any means but it's something.

The other thing is that we base our ranks on MMR and player population %'s. I love the idea that players can say "Hey I'm in plat I'm in the top x% of players". We also make it so our ranks are all the same MMR apart, which is why it's a bell curve at all(I mean we could change it tomorrow and make it so every rank has 5% of players). What I mean by this is that if you are silver 2 and play against a silver 1, it would be the same skill difference as a Gold 2 playing against a Gold 1. This obviously gets broken in Immortal+. We've actually tried to make ranks less negatively skewed, and try to make the center point high silver/low gold, but it's tricky and a slow process. I believe every episode we've tweaked ranked distribution upwards, and I'm sure it won't be our last time doing so. Not only do we have to worry about accounts that no longer play ranked, we have to pay attention to how the game as a whole is growing and how the community is improving and if it's effecting MMR. Like I said, I do think it's a little low but, but it's close. We want the higher ranks to still feel prestigious, and if we lean too positive they can lose meaning or worth. Right now it feels good to be gold+, but a majority of the playerbase became Gold/Plat, the only ranks the community feel like ment anything would be Immortal+. Honestly, there is no "right" answer because a lot of this is arguing player feeling. You have to balance high rank players feeling of exclusivity with the community's ranked expectations. We get complaints that there are too many people in Immortal, while getting complaints that too many people are silver, if we pulled people down and pushed others up the ranks would be akward.

Unless you are in the top 100 players, maybe top 500, in your region you will never "always be the best" on your team. That's not how match making works(well technically it's possible, but super improbable LOL so it's realistically never going to happen). Grouping up with someone, especially someone far off from your skill(like a 5 stack) can really effect your experience as well. Our match maker will never put you in a match where you have no chance of winning(or it's so heavily skewed like 90:10 where it's unrealistic to win). If you get put into a match, you always have a chance to win; this includes the range of your opponents. If you see a bronze in a gold match, it's either a 5 stack which breaks our ranked restriction rules, or it's a smurf that had their MMR sky rocket upward. Again this is because match makers actually have barriers against what you are talking about, unless you are in a 15+ minute queue(which usually low ranks have so many people this doesn't happen). I truly feel bad for players that feel like they have to always carry, or they are hardstuck, or their teammates weigh them down, because it's something that is not an easy fix. Ranked is simply a ladder, if you win you go up and if you lose you go down, the only thing holding someone down is you and playing enough games to get to where you belong(which is usually 20-50). We can't give free rank just because of 1 good game, or a good streak. I do think there are things we can do to improve the system, and some of those things take a lot of time and effort to ensure we aren't breaking things. I hope this helped open up some of your thoughts on the match maker, and if I have time I'll respond to any questions(I've just been super busy!). Sorry for the super long response! I was working on it throughout my day so it took a bit in between meetings. There are probably some typos because I was just cruising and didn't have time to glance over it a ton. Have a good one, and good luck finishing out the season!

There is no "golden child" queue. In fact, no player is held higher over another and the entire match is based on each individuals MMR. Our Match maker grabs groups of players of similar skill, then tries to make two teams for those players. It does not choose one player, says "this is supposed to be the best player" then purposefully tries to choose lower players or singles anyone out. If you get into a match it's because all the players around you are the same MMR. The only time this isn't the case is in 5 stacks which can break grouping restriction rules. Top 100 players can also be an edge case, but there is a restriction in the match maker which limits the distance of MMR players can be put in a match with.

no problem! Sorry it was short, I went indepth in my post in this thread.

It's hard, matchmaking will always be super complicated and mostly a black box with lots of beliefs in how it works. I would be worried if we stop having posts complaining about ranked and matchmaking, because I've never seen a game not have complaints. Hopefully future updates will help alleviate some of the issues people feel with their ranked experience.

Interesting take, adding ranks is definitely something I've seen people mention. Where would you put the rank and what would you call it? :P

Thank you!

It's hard right, the matchmaker is a black box and you just have to take my word for it. I don't expect to make everyone believe me, or for people not to have their theories. If anything the feedback, and issues people face, all make us want to improve the system or explore new spaces! Healthy skepticism is always good.

I don't know enough of the space, but I do know your familiarity with an Agent and how many times you've played them can affect your winrate. It's so small tho, and you are in the same closed environment as everyone else, that it's not a big thing we need to worry about for matchmaking (yet). I have wanted to explore this space, there may be a world someday where we have to take into account role or Agent to get a better idea of skill but we aren't there yet.

Great feedback!

I think we could probably do a better job of rewarding Immortal 1-3, or exploring the leaderboard space more. No plans right now, but I have it on the list of improvements I'd like to explore at some point. (Maybe after our tournament system, which our team is STILL pushing along and working on! Have to always have a list of things we can do, so we don't run out of stuff to improve!)

Hidden MMR is in every PVP game, for the most part. You will only find mobile games using "rank" as the main way to match make people. If you think about it, using rank to match-make is just making rank your "MMR".

Matchmaking rating(MMR) is just a way to rate a player's skill and put them in a match with others around the same skill. So it doesn't matter what you use, at some point something is going to act as MMR and put you in a match. The only games, that exist today, that don't have some sort of skill-based matchmaking are server browsers, mobile games, or indie titles(usually due to smaller communities).

So hidden MMR is the only thing that determines matchmaking, and this is true of Valorant. I think there is a heavy belief that MMR is somehow holding people back, or doing weird things to manipulate a player's experience in VALORANT(or any game really). But, honestly, MMR is just a ladder with complex math to determine how fast you move up and down it.

So , in any ladder system, the part you are trying to figure out is how far to move a player when they win or lose. If you are Silver, and win against a Gold player, how far up do you move and how far down does the Gold player move. In the original ELO system made for chess, this movement was very slow. So if you were a grandmaster in chess it could take you hundreds of games to actually get to grandmaster MMR/ELO; which would have you crush 100's of players in the process. Now we've just gotten better, using more stats, to be able to move you to where you probably belong faster. This results in less matches that are one-sided, and you get more competitive matches(which data has shown leads to more satisfaction/retention in gameplay).

So while MMR may sound like it's doing a ton, in reality when you win you move up the ladder and pass players and when you lose you go down and players climb above you. If the data ask wasn't ridiculous, we could print out every player in a giant list from top to bottom, because no player ever shares the same MMR(because again it's a ladder). This circles back to my point, it doesn't matter if we use a number or rank to match make players, because at the end of the day both of those systems would be determining who you play against. MMR is just built to be able to make more fair matches instead of "just grab anyone at X rank". Mobile games, or games that have a lot more random, can get away with matchmaking off rank; but again it depends on the game and how hard it is to make fair matches for that game. In Valorant, if we get MMR wrong, it can lead to extremely unfair matches; if you had a matchmaker for a card game like "war" it's so random and players don't have an effect on the outcome so you don't need a complex matchmaking system.

Our goal is to put you in a fair match, with players around your skill, but we know we can never account for things outside of the game. Maybe you aren't on your main this match, maybe you are bad at the map you got, maybe a teammate is sick and playing poorly, maybe the enemy just happens to timing you for 5 rounds and it tilts your mental, etc. There are so many factors that can lead to a match that on paper is fair, but because of the human element turns into a stomp or seems unwinnable. That being said, if our matchmaker was truly not working it would be 100% obvious. I do see a lot of complaints about coin-flip type matches, but if our matchmaker was not good it would be much more extreme. Every match would end 13-1, 13-0, etc.

That being said, matchmakers are not perfect and they are just a way for us to try and figure out who is better than who and give a rank to players. Because it's just a ladder, if you are truly better then others you will win and climb above them. That's why, at its core, it's up to you just to win and climb up the ladder past those around you.

I hope that helps explain a bit more, in short yes every VALORANT match will use MMR because if it didn't it would be completely random. Without MMR you would have Radiants against Irons, and instead of having any chance of winning you would just be hoping you got the Radiant/Immortal on your team and not the Iron/bronze.

(:<