[2020] Riot Blaustoise: "Yuumi ranks 13 in NA for "frustrating to play against"... In China, she's 119"

EDIT -- Check my comment history for full details on the survey

grabbing the top comment to say:

data is from October 2019. It's why in another thread I remarked that Korea is more likely to find champions frustrating if they have high pro-play presence (they disproportionately found Pantheon, Renekton, and Leblanc frustrating relative to other regions.

We typically run this survey around end-of-year, so we'll update data for 2020.

Champions not included:

- Samira, Yone, Lillia, Sett, Aphelios, Senna, Voli + Fiddle updates (only old versions)

The source is from our Mass Champion Survey (many devs have referenced it before, myself included. We've talked about it in official blog posts too). It's a survey we run in some of our largest regions that asks players their opinions about champions regarding everything from being frustrating to play against, their visual design, to their VO and lore.

The survey is distributed via the client (it gives a link to a Qualtrics survey). It's a pretty massive sample size, typically 30-40 thousand players per region. There's actually a high likelihood that players within this thread have taken it before.

As for cherry-picking, it was an interesting regional disparity that I wanted to highlight. I highlighted some other disparities too, but Yuumi in NA vs CN was one of the biggest once I found. I also pointed out that a bunch of champions are pretty close in rank for "frustrating to play against" across regions surveyed to demonstrate that it's not just all champions that NA and CN view differently.

Hope that helps.

From the most recent survey:

NA, 30,000 players each evaluate 3 champions randomly selected.

Total of 90,000 champion evaluations from 30,000 players

For the ~145 champions on this survey, that comes out to over 600 players per champion per region.

It's difficult because given the number of champions we have (now 150), and if we want to slice the data by region (let's say we do 5 regions), the survey gets pretty massive. Around ~150,000 total players surveyed. Hence "Mass" Champion survey

Question asked:

Next, consider these specific aspects of [CHAMPION]'s gameplay in League of Legends and rate your agreement with the following statements:  

 [CHAMPION] is frustrating to play against.

Answers: 5 point likert (Strongly disagree to Strongly agree)

---

Who was asked:

Active League of Legends players throughout the month of October, 2019, in 5 major regions: North America, Brazil, EUW, Korea, and China.

---

Sample Size:~150k for the total survey, ~30k per region. Respondents randomly see 3 champions to evaluate, 90,000 total evaluations per region, ~600 per champion per region. So "NA Yuumi" data is n~600, "CN Yuumi" data is n~600.

---

Answers received:

As stated above, it was a likert score. Scores are then ranked per demographic or split that you want (so Pyke had the highest or rank 1 likert score in this question for NA, Yuumi had the 13th highest)

From our Mass Champion survey (giant survey we run in a bunch of major regions to assess champions on a number of different things).

It's a nice addition to playrate and winrate data (those being behavioral data, Mass Champion being sentiment data)

If you check my comments I responded more in depth about the details of the survey to some other people in the thread.

Agreed actually. I think the biggest critique me saying "players find X champion less frustrating in this region" is that playrate likely plays a huge role.

You won't be as frustrated with a champion if you never see it, and you're likely going to be more frustrated by a champion if you constantly have to deal with it every other game.

Hey friends,

Goal of this was just to highlight some interesting disparities in player sentiment around champion frustration. I wasn't trying to suggest that because China doesn't think Yuumi is frustrating, that NA finding her more frustrating is invalid. There are other champs that China, or Korea, or Brazil find frustrating that NA doesn't as much.

For example, Leona (??) scored way higher for frustration in China than anywhere else. Idk maybe they meme Leona over there? But I do find it really interesting if memes or community perception could drive these differences to this degree (I think playrate and some other things factor in as well).

If y'all are curious about the data source, it's our Mass Champion survey. I responded to a comment in the thread and gave more details on this specific question's wording, sample size, and some other stuff.

I don't just make up these numbers and I'm not trying to cherry-pick to make arguments. If anything I just want to highlight that it's a global game, and we often find really big and interesting regional differences that I figure are cool to discuss.

also made a comment in the main thread. wasn't my intent to be antagonistic

as someone who spams Zilean to get to diamond at the end of the season, I agree

idk why people keep missing the real takeaway. good catch u/m0bilize

(1) he was only higher in Korea relative to other regions

(2) This was October 2019, around worlds. He was overtuned and 100% banned. I attribute hist higher frustration rating there to Korea being more sensitive to pro play.

(3) frustration isn't the only thing that speaks to the sucesses/misses of a rework.

A bunch of other people are writing in RE the statistics... I'll just say consider the business case.

I'm already surveying ~150,000 players. The full list of questions per each champion, plus a demographic block, plus we run some other blocks + questions in this survey means it's already long per person.

Now I can either survey more people, or I can make more people evaluate each champion. Either way I'm putting strain on players to be evaluating more champions or I'm expecting more people to take this survey (remember, we're running surveys on esports, on skins, on other cosmetic content, on seasonal events, etc. etc.).

Also we're not using this data like winrate data. I don't have the confidence to say that if a champion ranks 13th in frustrating with a likert score of 4.25 that the champion at rank 14 with a likert of 4.23 is less frustrating. But compared to the champion at rank 35 with a likert of 3.9? Now I'm more confident. The champion at 125 with a likert of 2.2? probably less frustrating.

We often split up the data from the mass champion survey into quintiles or quartiles. So if you're a top 30 champion for visual appeal, we just consider you as being pretty appealing regardless of if you're 26 or 23. The kinds of decisions we use this to inform are not as granular or as specific as nerfing champions RE winrate/banrate.