Yep, there is all the money investment riot promised at the start of the year lmao
It takes a while to translate investment to making a thing, and it takes longer if you are investing in the tech that makes the thing faster, which is what we did. Too many devs playing Satisfactory :)
Still not the development velocity we want but it will only get better
Seems like this season will have the biggest overhaul ever in league.
We've been investing extra heavily in tools, workflows and map features this last year to make these kind of changes easier. Still lots of improvements to go but large high quality changes are becoming more feasible
hi spawndog what projects did you work on today?
Hey friend, I am often involved in a lot of different things as I'm the tech director for league. Among other things I've been trying to help with server capacity planning for the south east Asia launch of team fight tactics on mobile
Most folks write off Vel'Koz as "Funny geometry squid" but his E is such an awkward ability to use that it's amazing when you do manage to get the self peel off and not die instantly on him.
His development name was "Professor eye beam". I like "Funny geometry squid" maybe we missed a chance there
Perfect 4D chess : bait to get the Ori shutdown gold onto Zed as Jihn knew he wasn't going to carry. No problem here /jk - sweet thresh mechanics are always the best to watch
How do you not know the root cause? Do you guys not have any observability within your architecture?
Our telemetry/observability costs are eye watering. An issue is too little signal to noise which we are working on.
We are a little closer to a root cause. A default value in the MySQL update for locking behavior changed and triggered the issue. We're rolling that fix out now
Would you be able to say if this might have been causing issues with Legends of Runeterra this past week or so?
From the incident report 7 days ago for Runeterra: "Due to the occurrence of a Database Deadlock, The game kicked players out of Games and The Client."
That does look suspiciously similar. We can see if there is a connection. Nice catch!
Do none of these issues appear outside of euw?
We have not rolled out the change everywhere but its occured in multiple places we did such as Taiwan and Oceania
MySQL still or was that only for account data? Did Riot issue a bug report that I can view?
MySQL 8 update is the suspected catalyst, the current version is hitting end of life this year which forced our hand. As we do not know the root cause it could be a bad code pattern that has existed for years on our side.
Sorry, I haven't seen a bug report outside of ticker status.
This is the same issue as occurred 12 hours ago which we have not found the root cause of. A full platform restart returns to service which we will do once existing games have ended.
After an update to the database software in a few regions it has a small chance to perma lock a table. If you cannot read a game ID you cannot start the game
May i ask why the game server needs to retrieve his id from the database ?
couldnt he generate it and try to send it to database? so when game start you don't need to get information from db, and if she drops the write it, the server can continue starting the game or prevent it to start depending of the type of match ?
i may oversimplify and extrapolate a lot from the comment
tldr: does the game server have to read from db ?
Good question. tl;dr I believe its due to needing persistent knowledge of what's happening in case of failures and needing to guarantee a unique ID everything agrees on.
I assume from your gender pronouns you are Spanish, French, maybe Italian? I lived in Montreal for a while :)
Starting a game relies on a bunch of services. Parties (friends). Team Builder (matchmaking + other things), Content Access (what you own), Game service manager (delivers your game), Local service manager (launches a game on a physical machine) and more. When something fails (which it will) you need to fall back on known state which means putting it somewhere safe
There is a team actively working on this and likely solved by now. This is similar to a recent OC incident where creating new games are unable to progress.
We suspect but have not confirmed, its related to a recent database update which under specific circumstances causes table rows to lock and cannot be read. aka "We want to start your game, what is the ID? ... No"
What's the ham incident?
Froggen memes of him (not really) DDoS'ing servers but the ham part was when we brought URF back during a promotional eSports event
My personal list would be:
Caps
Perkz
Jankos
Rekkles
Froggen
Froggen is probably the most controversial one, but I have him here because he's the only Western player ever to have a big claim of being the absolute best player in the world at a certain point.
Watching Froggen stream got me into watching eSports. Still my favorite player despite the ham incident - I chose to blame that mess we had to fix on skumbag Krepos bad influence
Any estimate as to completion time?
Tentatively the issue is resolved. We still need to dig to find and fix the root cause, until we do we wont have confidence in it relapsing.
Launching games is failing in OCE. There is a team triaging it. We don't have an underlying cause yet
Bots ai getting better is really good, and i wish there was a harder mode like how bots used to be, they were really fun in the early seasons.
The intention is to build better bot foundational technology that can help new players understand how to play the game better as well as much more sophisticated cool things we have plans for.
We believe we have resolved this. Let us know if you are still having an inability to connect to games. Player support is the best avenue as we can track large ticket numbers.
What happened? As a part of upgrading our NA game server fleet we encountered some networking issues this morning (/23 blackholing some networking traffic from ATT if people are curious). The problem is mitigated right now but we'll need some time to get normal connectivity (latency) working again for ATT customers.
We apologize for the inconvenience.
Hi, u/spawndog , I know you probably won’t see this nor reply but it’s going on two weeks since this occurred. I am completely locked out of playing league/TFT. I have submitted tickets and heard nothing.
Im on at least my 20th reinstall, and I have toggled every setting I can think in every combination. The game is just broken. Client, for once, is seemingly working fine, it’s when the game launches to the load in screen. It hasn’t reached the loading “splash screen” in two weeks…
You are my last attempt as I’m tired of looking for answers, when there is zero out there and no solution. Any insight would go a long way I’m reaching my never turn back point.
Sorry you're still having issues. We fixed a few bugs related to DX and some very custom archaic code that did not show up in the crash reporting. Our crash rates are now back to baseline but it sounds like something we've missed. I'll DM you
Anecdotal story but one of League Gameplays most beloved and veteran engineers retired yesterday. He is a Kat main so the team played (custom 5v5) where everyone played Katarina. There was some interesting Katarina variants.
devtrackers.gg