How do you not know the root cause? Do you guys not have any observability within your architecture?
Our telemetry/observability costs are eye watering. An issue is too little signal to noise which we are working on.
We are a little closer to a root cause. A default value in the MySQL update for locking behavior changed and triggered the issue. We're rolling that fix out now
Would you be able to say if this might have been causing issues with Legends of Runeterra this past week or so?
From the incident report 7 days ago for Runeterra: "Due to the occurrence of a Database Deadlock, The game kicked players out of Games and The Client."
That does look suspiciously similar. We can see if there is a connection. Nice catch!
Do none of these issues appear outside of euw?
We have not rolled out the change everywhere but its occured in multiple places we did such as Taiwan and Oceania
MySQL still or was that only for account data? Did Riot issue a bug report that I can view?
MySQL 8 update is the suspected catalyst, the current version is hitting end of life this year which forced our hand. As we do not know the root cause it could be a bad code pattern that has existed for years on our side.
Sorry, I haven't seen a bug report outside of ticker status.
This is the same issue as occurred 12 hours ago which we have not found the root cause of. A full platform restart returns to service which we will do once existing games have ended.
After an update to the database software in a few regions it has a small chance to perma lock a table. If you cannot read a game ID you cannot start the game
May i ask why the game server needs to retrieve his id from the database ?
couldnt he generate it and try to send it to database? so when game start you don't need to get information from db, and if she drops the write it, the server can continue starting the game or prevent it to start depending of the type of match ?
i may oversimplify and extrapolate a lot from the comment
tldr: does the game server have to read from db ?
Good question. tl;dr I believe its due to needing persistent knowledge of what's happening in case of failures and needing to guarantee a unique ID everything agrees on.
I assume from your gender pronouns you are Spanish, French, maybe Italian? I lived in Montreal for a while :)
Starting a game relies on a bunch of services. Parties (friends). Team Builder (matchmaking + other things), Content Access (what you own), Game service manager (delivers your game), Local service manager (launches a game on a physical machine) and more. When something fails (which it will) you need to fall back on known state which means putting it somewhere safe
There is a team actively working on this and likely solved by now. This is similar to a recent OC incident where creating new games are unable to progress.
We suspect but have not confirmed, its related to a recent database update which under specific circumstances causes table rows to lock and cannot be read. aka "We want to start your game, what is the ID? ... No"
What's the ham incident?
Froggen memes of him (not really) DDoS'ing servers but the ham part was when we brought URF back during a promotional eSports event
My personal list would be:
Caps
Perkz
Jankos
Rekkles
Froggen
Froggen is probably the most controversial one, but I have him here because he's the only Western player ever to have a big claim of being the absolute best player in the world at a certain point.
Watching Froggen stream got me into watching eSports. Still my favorite player despite the ham incident - I chose to blame that mess we had to fix on skumbag Krepos bad influence
Any estimate as to completion time?
Tentatively the issue is resolved. We still need to dig to find and fix the root cause, until we do we wont have confidence in it relapsing.
Launching games is failing in OCE. There is a team triaging it. We don't have an underlying cause yet
Bots ai getting better is really good, and i wish there was a harder mode like how bots used to be, they were really fun in the early seasons.
The intention is to build better bot foundational technology that can help new players understand how to play the game better as well as much more sophisticated cool things we have plans for.
We believe we have resolved this. Let us know if you are still having an inability to connect to games. Player support is the best avenue as we can track large ticket numbers.
What happened? As a part of upgrading our NA game server fleet we encountered some networking issues this morning (/23 blackholing some networking traffic from ATT if people are curious). The problem is mitigated right now but we'll need some time to get normal connectivity (latency) working again for ATT customers.
We apologize for the inconvenience.
Hi, u/spawndog , I know you probably won’t see this nor reply but it’s going on two weeks since this occurred. I am completely locked out of playing league/TFT. I have submitted tickets and heard nothing.
Im on at least my 20th reinstall, and I have toggled every setting I can think in every combination. The game is just broken. Client, for once, is seemingly working fine, it’s when the game launches to the load in screen. It hasn’t reached the loading “splash screen” in two weeks…
You are my last attempt as I’m tired of looking for answers, when there is zero out there and no solution. Any insight would go a long way I’m reaching my never turn back point.
Sorry you're still having issues. We fixed a few bugs related to DX and some very custom archaic code that did not show up in the crash reporting. Our crash rates are now back to baseline but it sounds like something we've missed. I'll DM you
Anecdotal story but one of League Gameplays most beloved and veteran engineers retired yesterday. He is a Kat main so the team played (custom 5v5) where everyone played Katarina. There was some interesting Katarina variants.
We recently upgraded the game client to be 64-bit in the 13.7 patch. This change was tested for compatibility as well as soaked on PBE, but since it was a significant change, it can still cause some unexpected problems. We are currently investigating new issues that have arisen since the upgrade.
We have noticed that some third-party apps like Discord, Blitz, and others may not be working properly causing instability. If you're experiencing issues, we suggest checking these apps and making sure that your drivers are up to date.
Don’t know if you’re aware, but played a couple games today and some of the Fighter item tooltips are somewhat off. Things like Eclipse/Ravenous saying they have 0% omnivamp instead of the new stats.
Appreciate the heads up mate. We can't ship localization in a hotfix so tooltip accuracy misses out in favor of us making the game fair and balanced quickly. Something for us to think about solutions though
why did my version number on the bottom right not change (when hovered)? jw
Its still 13.1 but with a bunch of hotfix changes. We can live patch data but the executable change requires a full redeploy which our capability is temporarily degraded due to the recent security breach.
everywhere or only NA?
Everywhere but China. They typically skip this patch for their new year
Still nothing!
Its live. Patch notes incoming
devtrackers.gg