Magic The Gathering: Arena

Magic The Gathering: Arena Dev Tracker




28 Sep

Comment

Originally posted by ProbablyWanze

Or it could mean you spent $100 to buy gems and just never got the gems. Which we consider just about the worst thing that could happen.

yeah, as soon as players losing money is involved and accounts get auto-banned from the internet service provider, if payments are reversed, its a nightmare, not to compare with a few late rewards and a bit of downtime of draft.

I ended up with a raw JSON dump of 32,960 transactions that failed, and I had to go through them all to determine what the player was trying to do, what actually happened, and what we needed to do to make it right. We got it all sorted out within (I think) 24 hours, but that was very un-fun.

this number reminded me of a completely unrelated anecdote of my life that i like to tell every once in a while. Well, at least it also has to do with lots of money.

15 or so years ago i was working on a sailing ship, that sailed gene...

Read more

That's hilarious! Working in digital sure has it's Fun Times, but I'm also thankful every time I hear about the funny constraints that operating in real-world goods imposes

Post
    on News - Thread - Direct

Sep 28, 20:01 UTC
Completed - The scheduled maintenance has been completed.

Sep 28, 17:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 27, 23:42 UTC
Scheduled - At 10am PDT @MTG_Arena will begin downtime processes for scheduled maintenance.

⏲️ Full deployment estimated at ~3 hours.
🛠️ Downtime is expected.
💻 Restart required.

Comment

Originally posted by ProbablyWanze

u/WotC_Jay: Which patch, bug, exploit or other issue had you sweating the most since you have been in your current position?

By far the worst to me was when we had a problem during the STX launch and a large number of player inventory transactions that we tried to make failed. They got dropped on the floor and just didn't happen.

That could mean non-harmful things, like you tried to exchange gold for a pack, nothing happened, you tried it again, and then it worked. Or it could mean you spent $100 to buy gems and just never got the gems. Which we consider just about the worst thing that could happen.

I ended up with a raw JSON dump of 32,960 transactions that failed, and I had to go through them all to determine what the player was trying to do, what actually happened, and what we needed to do to make it right. We got it all sorted out within (I think) 24 hours, but that was very un-fun.

Comment

Originally posted by LemonFennec

u/WotC_Jay what advantage is there to the current system of in-game inbox messaging promotional rewards, versus the previous system of promo code input via the store page or WotC website?

Distribute rewards via promo codes: ~30% of players who log in during reward period claim the reward
Distribute rewards via inbox: 85%-95% of players who log in during the reward period claim the reward

The portion of the player population that sees twitter, our website, or other ways to communicate the code is fairly small. Inbox works much better (when it's, you know, working)

Comment

Originally posted by Hjemmelsen

Sad for those of us who didn't get any at all, but nice of you to admit this is on you.

Once mailbox is safe to use again, we do plan on getting back to the players who were inappropriately missed. (That was another consequence of these problems.)

Comment

Originally posted by MrPopoGod

I'm actually quite curious if you ever pinned down why you were having database performance issues on writes. Was it lock contention, a bad index that was being thrashed by these writes, or something else?

Still working through that the managed provider we're using. Current theory is that this is due to an internal change in how they process operations that is, in most cases, an optimization, but in our use case (large, largely binary documents) turns out to be a pessimization. We've managed to find some other settings we can tweak to reduce some of the impact, but there's still more ground to make up

Comment

Originally posted by Legal_TrowAway_6789

Thank you for the additional information and clarity. We really do appreciate it.

However, not to be too critical, I have a question. The just of the issue seems to be that an update to the database of some sort caused performance issues, and since premier draft is (was) database heavy it suffered great degradation.

This is understandable as issues do slip through QA and mass load testing is never quite as accurate as the real deal. However why was the attempt to fix a P0 issue, in which the root cause analysis points to database performance, a complete rewrite of the drafting system, instead of a rollback of the database degradation update and an investigation into that? This article alone expressed the importance of taking time and doing QA on new features/updates, but yet it seems these changes were pretty major and rushed out in a panic as opposed to taking their proper time through the QA process. Was it not possible to revert the previous changes? And is there...

Read more

Rolling back the DB was an option that we investigated, but it ended up being much higher-risk than the changes to draft. This DB upgrade was also something that we had to take at some point (per the analogy with phone OS), so rolling back could only be a temporary solution

Comment

Originally posted by Holy_Beergut

u/Wotc_Joey I'm curious as to what we would receive for the remaining anniversary presents, would you be willing to share it here to satisfy my curiousity?

Maybe put it in spoiler tags like so, so at least redditors that don't want to be spoiled can avoid looking.

As far as the contents of the presents - your curiosity will have to continue. :)

However, as Jay mentions elsewhere, I know we are planning to err on the generous side of getting everyone their gifts - to qualm any concerns there.

Comment

Originally posted by advanced_funposter

Will the infinite gold bug that comes up every so often during rotation day be fixed once and for all some time? It makes for a bad player experience to go against a bunch of people conceding after turn 1 when you try and finish your quests on that day, and they get 10s of thousands of free gold.

It's certainly our goal to fix it. We thought we had (for LTR). Then it broke again in a different way.

Comment

Originally posted by mtg_is_a_drug

Immagine starting a server maintenance at 8 pm european time

Sorry for the timing. Given a team that's been working almost around the clock for the last week, we wanted to do this during normal working hours (Seattle time).

Comment

Originally posted by thetrooper007

Are we allowed to use the hordes of draft tokens we were sent in error, or is there a risk of that being considered bug abuse?

The tokens are yours. They came from our error, and there was (basically) no abuse involved. Bank error in your favor.

Comment

Originally posted by ProbablyWanze

thanks. i have 3 accounts and during last week noticed that they get the same mail sometimes hours apart. obviously, it wasnt working as intended but it got me to wonder how long it usually takes, if the system is running as intended, to send one mail to all accounts?

There are a few different ways to send them, but when things are working properly it can go out almost instantly to everyone. (We don't need to send to people individually, we can set it up so that everyone references a central mail item.)

Comment

Originally posted by ProbablyWanze

question for u/wotc_Jay: Had the issue of the draft tokens being sent multiple times on multiple days something to do with the performance issues of the premier drafts?

Thanks for hte article btw, these are always appreciated.

Based on current investigations, they came from the same cause - that database performance issue mentioned in the article. That gummed up the player drafts, and it also created issues with the inbox messages getting distributed properly (i.e. once only) to players

Comment

Originally posted by dwindleelflock

Those articles/updates are very much appreciated. Even something as small as letting us know about the issue and that you are working towards a fix is what makes a difference, in my opinion. Describing what went wrong and your process is very interesting too.

As an aside I want to ask about a specific server issue that some of us in Europe faced. I personally have been getting "network error - unable to get collection" and "waiting for the server" ever since a couple of weeks before WOE release. When I messaged support and going through the usual troubleshooting the only solution that worked was using a USA based VPN. This issue was, for the most part, present until the 18th of this month (when it was completely fixed) so I was not really able to play for this timeframe. What I want to ask is: was this a known issue and is it related to the issue described...

Read more

I was not personally aware of that issue (though others on the team might have been). That does sound like an ISP-side issue. We didn't change anything on the 18th, so I suspect your ISP fixed something then

Comment

Originally posted by JMooooooooo

The original message about Anniversary Week rewards implied that you had to log in on specific days to be eligible for that day rewards. I would guess a lot of anxiety around those issues still not getting fixed comes from players worries that once issues get fixed, it will turn out they didn't do whatever was now required for Anniversary Week rewards, and clearing up how that eligibility will be handled would go long way in calming players down.

When the inbox gets fixed, we're going to err on the generous side here. I don't have full details worked out, but we're not going to be doing things to artificially limit the recipients here.

Comment

Originally posted by tom277

Thanks for the interesting article! Has there been any consideration for giving out rewards through codes like we used to get if the inbox system does not come back online?

Inbox will be back soon; we're mostly just working on resolving how safe/how soon we can get that (like the article mentions, riskier changes need to go in releases further off for more testing, safer, more vetted changes can go in with shorter timelines).

If the outage was going to last too long, we could look at codes. But codes would only hit ~30% of the players, while 85%-95% will claim rewards out of inbox. So we have a heavy preference for using the inbox to distribute things.

Comment

(Admins: WotC_Joey is legit; can we get flair for him?)

Post

Hi all -

We published an article ( https://magic.wizards.com/en/news/mtg-arena/mtg-arena-anniversary-week-issues ) to have a candid update with you - our community - about the issues we have been experiencing since last week. Further down, we dive into some of the behind-the-scenes with Ben Smith on the details.

u/WotC_Jay will be hopping in and out of the comments to respond to questions and feedback - please be kind!

W...

Read more External link →

25 Sep

Post
    on News - Thread - Direct

Sep 25, 15:00 UTC
Resolved - This incident has been resolved.

Sep 21, 21:24 UTC
Monitoring - We have recently deployed a hotfix to address issues related to draft stability on @MTG_Arena. We have reenabled player drafts at this time. We are continuing to monitor the situation, and will provide additional updates if necessary.

Sep 21, 16:47 UTC
Identified - As we work to resolve draft instability, players may experience some disruption to other parts of @MTG_Arena today. We apologize for the disruptions, and are focused on getting things healthy as quickly as we can.

Sep 21, 00:23 UTC
Update - The investigation into player draft issues is ongoing, and while we are working as quickly as we can, we expect player drafts will remain unavaila...

Read more

22 Sep

Comment

Intent was that anyone who logged in during the Troubled Times from Tuesday through yesterday got one mail. By some accident (not yet fully understood), some players got two mails. Bank error in your favor; enjoy!