[2021] [AMA] We're part of the Riot Data Center and Riot Direct Networking/Software/System Team!

Holy cow!!! A talking Hamster!

Twilight Fades!

Clashy Boi too Stronk

Morde, hands down

huehuehue

Nerf Not Irelia!

Its really hard to tell with just that bit of knowledge. If ping suddenly changes, it could mean that the path you take to the game server has changed, or it could mean that something in between is overloaded. If you are using wifi, it could also mean that you are having interference. Theres probably two dozen other reasons, but thats where I would start.

1. Yes, we have several teams dedicated to various types of security, including network and app
2. For each publicly available service, you increase your attack surface. Riot has so many in fact that its no longer possible for an individual to even comprehend an entire ruleset, let alone how many rulesets interact with all the others. On a small network I would expect any network engineer to be able to answer "Is port X open to the public any where" but on large networks we have to rely on our tools and automation to tell us that. Not sure if that makes us better or worse.
3. Cloud Providers offer a lot of things, but there is a break point where the cost of a bare metal server is cheaper. The best thing is a judicious balance of many cloud providers and bare metal, trading back and forth for cost/speed as each use case needs. I would say there is never THE right answer when it comes to infrastructure, neither cloud nor bare metal will solve every problem.

Depends a lot on whats causing the fluctuations, if you are using wifi...get wired. If its a shared network more bandwidth or maybe even Qos if your router supports it. If the problem is outside your house, theres not a lot you can do other than complain to your ISP or change ISPs. Take all this with the grain of salt that I dont know much about your situation.

Lot of stuff in here, unfortunately we dont really talk specifics when it comes to numbers and versions, I also dont know much about the back office (IT) side of the house so I wont speak for them.

1. Lots of servers, at least 100 Garens
2. We have about a 60/40 mix of windows and linux servers, depending on usecase
3. We use open source front end / middleware on top of whitebox servers with a lot of disk
4. With few exceptions we treat our servers as cattle instead of pets. Meaning if any one server were to suddenly die, its a statistic rather then a sadness. So most of our servers dont require backups. Things that deal with persistence (Databases) replicate their data to multiple other servers, and then one of them writes that data to...well...its black magic to me, but theres a team that manages where that goes and what happens to it after.
5. Riot tries to get every engineer to be a T, a deep tech passion, with a lot of broad knowledge. Each team at riot is responsible for a thing, and they take care of every aspect of that thing. So sorta both specialized per team, but not per engineer.
6. Question for IT folk, sorry i cant help
7. M365?
8. Asset management is still a super tough problem. Different teams use different software depending on their needs. I *think* our DC team uses Device42, but beyond that, I honest not sure.

Sorry for vagueness or non-answers. I did my best

Probably to get promoted and not be able to code anymore

Is he a Doc?

Theres a lot of options with wifi, a dedicated point to point bridge on a different channel could help. Consider changing between 2.4 and 5ghz, there might be something specific interfering with which ever one you are using. You may try changing channels as well. It might be a crowded airspace. Might consider getting directional antenna, could help a lot there too. I dunno, don't give up there, work the problem, experiment with settings and positions for your wifi.

Its the client stats reporter end point. That feeds data that allow us to detect problems with ISPs and feeds services like lagreport

Good luck!

Glad I could help!

1. Game servers are treated like every other server, just with high network other resource utilization. They differ mainly in that the work loads last between 15 minutes and 4 hours and cant be interrupted with out impacting players. So we try to not do that ever.
   1. We do have orchestrations like docker / k8s /dcos.
2. Some of use make heavy use of ansible and jinja, others are into Napalm, and we also have a lot of in house built tools to do specific one off tasks.

Like no email no call nothing? If thats the case, I'd say something is probably wrong and you should resubmit.

Does that mean that the Delorean has the top let down and the sunshine is shining?

Damn 100 Garens is a lot. I guess its quiet noisy in your datacenters.

Lots of whirring, its like a URF one for all with both sides picking Garen

Really interesting concept. I think that is one of the main differences when everything is scaled up like crazy. However you still need to worry about corrupt data spreading over multiple systems - right?

Not really, I mean theres obviously a risk, but its pretty low given that most data is pushed too servers, and the only thing that comes back goes through well formed APIs. I dont know that we've had an incident like you describe.

Sounds interesting. Can you tell something about the management hirarchy you guys are running? Classig team with teamleads or something different?

Teams are mostly self organizing, but my part of the world we have a Dev Manager hat that takes care of trying to keep us working on a sustainable pace and helps us meet our internal deadlines, a Product Lead hat which takes care of working out the vision for the thing we are working on and works with other teams to create and set expectations, a Tech Lead hat that is accountable for the quality of the work we do, an Engineering Manager that is responsible for our happiness / taking days off / and career progression, and finally a Team Captain hat that ensures all the other hats are doing their roles and generally is the back up for all the other roles. Each hat could be worn by any engineer, and will shift as needed / wanted. Engineering Managers span teams, and all engineers work for an Engineering Manager (EM), and then the loans out engineers to teams. All the roles, except Engineering Managers and Dev Managers are also Engineers that do engineering work every day. I have no idea if that makes sense, and it certainly sounds weird, but its grown on me over the years to separate the person that tells me what to do, from the person that cares about how I am. Removing that conflict of interest has had staggering improvements on my quality of life.

Office 365

Guess its obvious that its not that then :P

Thanks a lot for that information. Never heard of that software but sounds interesting on the first look. Definitely will have a look at it in more detail tomorrow.

I couldn't recommend it, but I also couldn't recommend any other either.

I dont know of any, but I'm not in that team/office so I cant really speak for them

I would say, respond back and ask for feed back and recommendations on what you could improve or change.

A game stays to the same server it started on no matter what. Even then I don't have much more than personal application "for fun" experience with k8s (as i'm an infrastructure fellow that works on stuff beneath that layer), so I'm not sure what has been tested or not, nor the results.