7 months ago - /u/WotC_Joey - Direct link

Hi all -

We published an article ( https://magic.wizards.com/en/news/mtg-arena/mtg-arena-anniversary-week-issues ) to have a candid update with you - our community - about the issues we have been experiencing since last week. Further down, we dive into some of the behind-the-scenes with Ben Smith on the details.

u/WotC_Jay will be hopping in and out of the comments to respond to questions and feedback - please be kind!

We appreciate the patience and understanding that everyone has had, and we will continue to do our best at having as much openness as we can with everyone.

External link →
7 months ago - /u/WotC_Joey - Direct link

Hi all -

We published an article ( https://magic.wizards.com/en/news/mtg-arena/mtg-arena-anniversary-week-issues ) to have a candid update with you - our community - about the issues we have been experiencing since last week. Further down, we dive into some of the behind-the-scenes with Ben Smith on the details.

u/WotC_Jay will be hopping in and out of the comments to respond to questions and feedback - please be kind!

We appreciate the patience and understanding that everyone has had, and we will continue to do our best at having as much openness as we can with everyone.

External link →
7 months ago - /u/WotC_Jay - Direct link

(Admins: WotC_Joey is legit; can we get flair for him?)

7 months ago - /u/WotC_Jay - Direct link

Originally posted by tom277

Thanks for the interesting article! Has there been any consideration for giving out rewards through codes like we used to get if the inbox system does not come back online?

Inbox will be back soon; we're mostly just working on resolving how safe/how soon we can get that (like the article mentions, riskier changes need to go in releases further off for more testing, safer, more vetted changes can go in with shorter timelines).

If the outage was going to last too long, we could look at codes. But codes would only hit ~30% of the players, while 85%-95% will claim rewards out of inbox. So we have a heavy preference for using the inbox to distribute things.

7 months ago - /u/WotC_Jay - Direct link

Originally posted by JMooooooooo

The original message about Anniversary Week rewards implied that you had to log in on specific days to be eligible for that day rewards. I would guess a lot of anxiety around those issues still not getting fixed comes from players worries that once issues get fixed, it will turn out they didn't do whatever was now required for Anniversary Week rewards, and clearing up how that eligibility will be handled would go long way in calming players down.

When the inbox gets fixed, we're going to err on the generous side here. I don't have full details worked out, but we're not going to be doing things to artificially limit the recipients here.

7 months ago - /u/WotC_Jay - Direct link

Originally posted by dwindleelflock

Those articles/updates are very much appreciated. Even something as small as letting us know about the issue and that you are working towards a fix is what makes a difference, in my opinion. Describing what went wrong and your process is very interesting too.

As an aside I want to ask about a specific server issue that some of us in Europe faced. I personally have been getting "network error - unable to get collection" and "waiting for the server" ever since a couple of weeks before WOE release. When I messaged support and going through the usual troubleshooting the only solution that worked was using a USA based VPN. This issue was, for the most part, present until the 18th of this month (when it was completely fixed) so I was not really able to play for this timeframe. What I want to ask is: was this a known issue and is it related to the issue described in OP? Is this an ISP issue as some suggested in the comments of the reddit thread above, or is this a server issue?

And again thank for taking the time to write those articles. They really mean a lot and help us understand your situation.

I was not personally aware of that issue (though others on the team might have been). That does sound like an ISP-side issue. We didn't change anything on the 18th, so I suspect your ISP fixed something then

7 months ago - /u/WotC_Jay - Direct link

Originally posted by ProbablyWanze

question for u/wotc_Jay: Had the issue of the draft tokens being sent multiple times on multiple days something to do with the performance issues of the premier drafts?

Thanks for hte article btw, these are always appreciated.

Based on current investigations, they came from the same cause - that database performance issue mentioned in the article. That gummed up the player drafts, and it also created issues with the inbox messages getting distributed properly (i.e. once only) to players

7 months ago - /u/WotC_Jay - Direct link

Originally posted by ProbablyWanze

thanks. i have 3 accounts and during last week noticed that they get the same mail sometimes hours apart. obviously, it wasnt working as intended but it got me to wonder how long it usually takes, if the system is running as intended, to send one mail to all accounts?

There are a few different ways to send them, but when things are working properly it can go out almost instantly to everyone. (We don't need to send to people individually, we can set it up so that everyone references a central mail item.)

7 months ago - /u/WotC_Jay - Direct link

Originally posted by thetrooper007

Are we allowed to use the hordes of draft tokens we were sent in error, or is there a risk of that being considered bug abuse?

The tokens are yours. They came from our error, and there was (basically) no abuse involved. Bank error in your favor.

7 months ago - /u/WotC_Jay - Direct link

Originally posted by mtg_is_a_drug

Immagine starting a server maintenance at 8 pm european time

Sorry for the timing. Given a team that's been working almost around the clock for the last week, we wanted to do this during normal working hours (Seattle time).

7 months ago - /u/WotC_Jay - Direct link

Originally posted by advanced_funposter

Will the infinite gold bug that comes up every so often during rotation day be fixed once and for all some time? It makes for a bad player experience to go against a bunch of people conceding after turn 1 when you try and finish your quests on that day, and they get 10s of thousands of free gold.

It's certainly our goal to fix it. We thought we had (for LTR). Then it broke again in a different way.

7 months ago - /u/WotC_Joey - Direct link

Originally posted by Holy_Beergut

u/Wotc_Joey I'm curious as to what we would receive for the remaining anniversary presents, would you be willing to share it here to satisfy my curiousity?

Maybe put it in spoiler tags like so, so at least redditors that don't want to be spoiled can avoid looking.

As far as the contents of the presents - your curiosity will have to continue. :)

However, as Jay mentions elsewhere, I know we are planning to err on the generous side of getting everyone their gifts - to qualm any concerns there.

7 months ago - /u/WotC_Jay - Direct link

Originally posted by Legal_TrowAway_6789

Thank you for the additional information and clarity. We really do appreciate it.

However, not to be too critical, I have a question. The just of the issue seems to be that an update to the database of some sort caused performance issues, and since premier draft is (was) database heavy it suffered great degradation.

This is understandable as issues do slip through QA and mass load testing is never quite as accurate as the real deal. However why was the attempt to fix a P0 issue, in which the root cause analysis points to database performance, a complete rewrite of the drafting system, instead of a rollback of the database degradation update and an investigation into that? This article alone expressed the importance of taking time and doing QA on new features/updates, but yet it seems these changes were pretty major and rushed out in a panic as opposed to taking their proper time through the QA process. Was it not possible to revert the previous changes? And is there intent to fix the underlying database issues that are at the root of all this? Is there fear that these new rushed and major changes are going to be the catalyst for a bunch of additional undiscovered issues?

Rolling back the DB was an option that we investigated, but it ended up being much higher-risk than the changes to draft. This DB upgrade was also something that we had to take at some point (per the analogy with phone OS), so rolling back could only be a temporary solution

7 months ago - /u/WotC_Jay - Direct link

Originally posted by MrPopoGod

I'm actually quite curious if you ever pinned down why you were having database performance issues on writes. Was it lock contention, a bad index that was being thrashed by these writes, or something else?

Still working through that the managed provider we're using. Current theory is that this is due to an internal change in how they process operations that is, in most cases, an optimization, but in our use case (large, largely binary documents) turns out to be a pessimization. We've managed to find some other settings we can tweak to reduce some of the impact, but there's still more ground to make up

7 months ago - /u/WotC_Jay - Direct link

Originally posted by Hjemmelsen

Sad for those of us who didn't get any at all, but nice of you to admit this is on you.

Once mailbox is safe to use again, we do plan on getting back to the players who were inappropriately missed. (That was another consequence of these problems.)

7 months ago - /u/WotC_Jay - Direct link

Originally posted by LemonFennec

u/WotC_Jay what advantage is there to the current system of in-game inbox messaging promotional rewards, versus the previous system of promo code input via the store page or WotC website?

Distribute rewards via promo codes: ~30% of players who log in during reward period claim the reward
Distribute rewards via inbox: 85%-95% of players who log in during the reward period claim the reward

The portion of the player population that sees twitter, our website, or other ways to communicate the code is fairly small. Inbox works much better (when it's, you know, working)

7 months ago - /u/WotC_Jay - Direct link

Originally posted by ProbablyWanze

u/WotC_Jay: Which patch, bug, exploit or other issue had you sweating the most since you have been in your current position?

By far the worst to me was when we had a problem during the STX launch and a large number of player inventory transactions that we tried to make failed. They got dropped on the floor and just didn't happen.

That could mean non-harmful things, like you tried to exchange gold for a pack, nothing happened, you tried it again, and then it worked. Or it could mean you spent $100 to buy gems and just never got the gems. Which we consider just about the worst thing that could happen.

I ended up with a raw JSON dump of 32,960 transactions that failed, and I had to go through them all to determine what the player was trying to do, what actually happened, and what we needed to do to make it right. We got it all sorted out within (I think) 24 hours, but that was very un-fun.

7 months ago - /u/WotC_Jay - Direct link

Originally posted by ProbablyWanze

Or it could mean you spent $100 to buy gems and just never got the gems. Which we consider just about the worst thing that could happen.

yeah, as soon as players losing money is involved and accounts get auto-banned from the internet service provider, if payments are reversed, its a nightmare, not to compare with a few late rewards and a bit of downtime of draft.

I ended up with a raw JSON dump of 32,960 transactions that failed, and I had to go through them all to determine what the player was trying to do, what actually happened, and what we needed to do to make it right. We got it all sorted out within (I think) 24 hours, but that was very un-fun.

this number reminded me of a completely unrelated anecdote of my life that i like to tell every once in a while. Well, at least it also has to do with lots of money.

15 or so years ago i was working on a sailing ship, that sailed general goods to undersupplied small atolls in the south pacific from Hawaii.

Fanning Island in the Republic of Kiribati wa one of them and he had around 70k dollars worth of goods to deliver.

the problem was that there was only 32,960 worth of cash on the whole Island and 1 bank. I rememberthat number because i was involved in trying to figure out, how we split up our orders to be done with unloading in 2 and 1/2 days.

Online Banking wasnt a thing back then, at least not yet in those parts. So before we arrived, they urged all the islanders (around 2000) to bring all their cash to the bank.

Then on the first day, our first customers would withdraw around 30-32k worth of cash and we would unload their cargo and get paid over the day. And in the evening, we would put all that cash back to the bank, they would wire us the amount to our Australian bank account and rinse and repeat the next day.

In the evening, that 30k of cash had to be counted by us and since i was the cook on board and didnt have too much to do over the day except snorkeling for some lobsters for dinner and provide the crew with cold drinks, i volunteered for the counting so the other crew, who had been grafting away all day in scorching heat could enjoy a nice barbeque on shore and relax.

There also was the prospect of posting for some nice selfies with a huge batch of money on a sailing ship anchored in the lagoon of a far flung tropical island, it was my first trip down there, so it was all a bit surreal.

Another problem was that all the money was in 3 different currencies. US, NZL and AU Dollars. While AU$ were the official currency, there was lots of US and NZL currency on the island from the occasional cruise ships that came around and their passengers tipping the islanders with those currencies.

But that also meant that around 25 of those 30k were 1 dollar notes (and 2k in coins), which made counting so much more unfun than you would think.

Me and the Captain needed around 3-4h to count it ourself and then we had to sit next to the single bank employee to count it again for another 7h.

That's hilarious! Working in digital sure has it's Fun Times, but I'm also thankful every time I hear about the funny constraints that operating in real-world goods imposes

7 months ago - /u/Ridikule - Direct link

Originally posted by liquidben

With server-side writes to the database off, how are draft picks tracked? Is that only done client-side and then the remaining cards are communicated to the next participant's client? If a client loses network connectivity in the middle of a draft, does a server-side operation handle the autopicking, or does it potentially restart the draft round for the other players?

The state of the draft is tracked in memory (ram) on the server. Since any given draft usually lives for about 20 minutes, the data is ultimately ephemeral. If a client loses connection in the middle of a draft, auto-picks work as they did before.

7 months ago - /u/WotC_Joey - Direct link

Originally posted by DevilSauron

Hi, I’ve installed the game only a few days ago and since then, I have been able to connect only about half of the times I try it. I get Network errors often and usually have to wait for +- an hour before trying again. Is this a normal experience at this time, or does it point to other problems at my end (my internet connection is totally fine)?

Hi and welcome to MTG Arena! As the article mentions, we have unfortunately been experiencing some technical issues over the past week or so, so some/all of the errors you're seeing might be from that - wouldn't be able to say.

I would recommend you follow us on our MTG Arena Twitter for the most up-to-date happenings! If you're enjoying the game and want to be more involved, we are also on all the socials (Instagram, Facebook, Tik Tok, Threads), and you can enjoy lots of back-and-forth with the community in our Discord. Heck - we even post deck techs on YouTube! (Is it obvious yet that I work in marketing?)

Welcome to the community and hope you're having fun!!