[2019] Riot's surveys often have leading, misleading, biased, or incomplete response options.

Hi there!

I'm Riot aeneia. I designed, wrote, and launched this survey.

I can't address everything you've brought up here, but I did want to mea culpa something, and hopefully provide a little education:

• That first question, about how recently you purchased in-game content. That was my bad. I forgot to include the option for "It's been more than a year." I done f*cked up there, and I'm sorry to make anyone feel like their historical purchases weren't important :(.
• As for the rest of the survey, I'd encourage you to look up Choice Based Modeling and Max Diffs. They're pretty brilliant methodologies that help us to prioritize different concerns. (And, yes, we know that something being knocked to the bottom of the list doesn't make it a non-issue. It just means it's less of an issue than the other things above it.)
• There's more to the repeated questions than just filtering out troll responses ;). Like I said, do some research on Max Diffs!

Thanks for that! We're continuing to revise this survey as results come in, so it's still very much in the editable phase. Knowing that this is confusing or feels bad is super valuable. Thanks again :)

Thanks for replying! I didn't think you were objecting, but I think I misinterpreted your comment: "By repeating questions and options you can better ensure consistent responses to gain more accurate results. This combats people who just click through a survey without reading it." as being limited to just addressing click-through ("troll") responses, not being essential to the design of the survey. My apologies there for misunderstanding.

As for your point on objectivity, I'd love to see any academic papers you can point me to so that I can better understand your point. While objectively equal and subjectively different items would be IDEAL for a super clean Max Diff, I've never read anything to suggest that its methodology is inappropriate for something like this.

I don't want to dive too deeply into Reddit debates, so I'll probably sign off here. I appreciate your input -- this survey is iterative, so we're continuing to improve it (design-wise and player-experience-wise) over the next few months. I'm sure this comes across hollow, but I really do appreciate these kinds of posts that help us to make our surveys better.

*Fistbump*

*fistbump*

Another reason other than honesty check can be to avoid miscategorizing people with smurfs / multiple accounts. If someone gets a survey on their smurf that is registered with a different email, it could be misleading as to their actual behavior as a player in league.

Say they have never purchased a skin on their smurf but have a ton of skins on their main, their behavioral profile is very different from someone who truly has never bought a skin :)

Hey there! Troll responses are responses that are invalid, and do not reflect real-world behavior. For example, if a player says, "Yes, I've purchased an Event pass," but they HAVEN'T actually purchased an Event Pass, it indicates that they aren't answering the survey honestly. That would be a "troll" response.

If a player says, "Yes I've purchased an Event Pass, and I f*cking hate them. They're terrible. Riot you suck." when they HAVE actually purchased an Event Pass, that is NOT a "troll" response, and would therefore be treated as a valid player opinion.

I hope this clears things up :)

Hey there! Asking players to rank-order a set of 4 questions 12 times (or even like 6) is a HUGE cognitive ask! It probably would take them half an hour to have filled out the survey. We try to keep our surveys shorter, when possible, to respect player time. Also, Max Diffs work either by selecting the "most" option, or the "most" and "least" option. Sawtooth software doesn't currently offer a "rank the full set" option, although that would theoretically reduce the amount of screens we have to show. We use the "most" and "least" because it gives us the most data possible.

As for the vague estimation of preferences, we know that we (as Rioters) have a super different experience of the game than players. This is especially true of PIE, the team who works on things like Skins, Loot, Event Passes, and other in-game products. We want to make sure we don't bias the results by assuming our understanding is representative of player experience.

Thanks for the question :)

For this survey in particular, we're doing an "alpha" release from now until December in just a couple of regions. The finalized version of the survey will be ready to go at the start of 2020. Yes, we won't be able to track things longitudinally for these first three months. However, we'll be able to get some preliminary information (like, "What is seriously on fire right now for our players?) to help make more immediate decisions.

That way, once the survey is fully ready and good to go, we'll be confident that we're asking the right questions in the right way.

This is intentional, yes! If we find that this isn't an issue at all for players (and it's just wasted space on the survey), that question may very well get replaced with something like "Riot has too many alternate currencies." I won't know until I see the data!

That said, a massive overhaul of all our alternate currencies makes me want to cry lol...RIP.

@thugginuggs in for the nami support

Thank you so much :). We really appreciate you taking them! There are a TON of people who just click out of the prompt, and I can't blame them. Not everyone has time to go through a survey. So it means a lot when people do take the time :)

My pleasure! Thanks for engaging and wanting to know more :) fistbump

From Riot, no. From me? Maybe. Probably. Be warned, though. I might skew that data based on my own preferences XD.