Magic The Gathering: Arena Dev Tracker

Tokenization is a component of our parsing process, one very early in the process. It's true that replacing one token with another similar one is often not worth considering to be a big difference. But what about one sentence structure with another? For example "If you would draw a card, draw two cards instead" vs. "If you would draw a card, instead draw two cards" should behave the same despite being worded differently (and both are in fact valid wordings). If we already handled a phrase like "If CARDNAME would deal damage, it deals twice that much damage instead" as well as "If CARDNAME would deal damage, instead it deals twice that much damage", then we've already handled that syntactical difference. Let's say we wrote tests for the latter two cards; we'd find in ad-hoc testing that once we got either version of the draw replacement working (and wrote tests verifying it), the other one would work too. Given that it worked "out of the box", how much effort should we spend testing...

Two main ways:

• The parser fails to generate code. This is great! It's recognizing that something is outside its current boundaries. We usually have a good idea of what we need to do from the error messaging.

• The parser generates wrong code. Less great. Human QA needs to play the card to see that it's doing the wrong thing. The most common type of problem here is with "anaphora resolution" - figuring out what ambiguous phrases like "it" or "that creature" mean. Why, I just estimated the complexity of a few LTR bugs with that issue moments ago... #wotc_staff

Pretty broad question. In one sense, we're somewhat similar: we both make code happen starting from English strings from new cards to make a good MTG play experience. But our engineering is completely different, from code generation to the actual engine design. #wotc_staff

Perhaps for an LTR implementation tale! #wotc_staff

I'd love to tell more stories about "challenging developments that went smoothly". I think there's a couple challenges to that:

• Less of a narrative! With a bug, there's an immediate hook of "how did that happen", then a cool investigation, a eureka of the issue, and often an embarrassingly simple fix (this bug was fixed just by deleting a line of code!). I think that makes for a pretty clear flow. Most implementation stories don't have such a narrative structure to them, which makes them harder to write about.

• Scope of background. Even this post had coworkers dozing off with the groundwork I presented to describe the bug. New features are often even less cleanly described.

• When? What? It can be hard after-the-fact to decide what would make an interesting story to talk about, or when to talk about it.

Still, the reaction to Ian's post and this has us pretty interested in doing more. Heck, I've always wanted to! I'm...

I believe that had been a UI bug, where the client was improperly batching the Muldrotha permissions in its presentation of your actions. #wotc_staff

Can you clarify what your suggestion is? "Printed on" is a pretty ambiguous concept:

• What about copy effects? If card A becomes a copy of Gutter Grime and triggers to make an Ooze, the reference to "Gutter Grime" on the Ooze means "Card A".

• That still holds true even if Card A stops being a copy of Gutter Grime.

• Through horrible shenanigans you're able to make The Book of Vile Darkness create a Vecna token that has Gutter Grime's triggered ability. In that case the "Gutter Grime" phrase on the Ooze it creates refers to the Vecna that made the Ooze token. Was that ability "printed on" Vecna?

I think perhaps what you're trying to say is "Gutter Grime" in the conferred ability refers to "the card that conferred this ability to this Ooze". But that's the whole point of this post - identifying when a self-reference is like that is nontrivial. Our original logic, due to the cards we had covered on Arena, was myo...

We test cards that involved a developer's effort to get to work in the first place. Human QA does a pass over a set to identify what didn't automatically work from the first time we generate code for a new card set. Anything that doesn't work at that point is, well, my day job! And work we do there gets verified against regression by an automated test.

When we're closer to release, QA does another full pass to hopefully identify regressions, again focusing on the new cards due to the huge explosion of possible interactions.

I identified in the OP the relevant cards in the story: Heliod's Punishment has plenty of tests that lean on "self-reference in conferred abilities". Unfortunately Heliod's Punishment's behavior doesn't involve the ProposeEffectCostResource rule, which was the center of this bug: its conferred ability's only cost is the tap-symbol. #wotc_staff

The problem with taking a recording of a game and saying "make sure it plays like that again" is in determining what "like that" means. We do plenty of changes to the game that don't change the gameplay outcome but do, for example, change the information in requests and responses to the client, change what information is available in the game, change autotap strategies, etc. The advantage with our "scripted game" tests is that we're able to decide precisely what is important to verify with automated assertions, and what aspects of the game's proceedings are allowed to vary over the development of Arena as a project. #wotc_staff

The notion is it doesn't matter which Falco you use - the action behaves the exact same way for either. For Serra Paragon, it does matter which you use - that one can't be used again this turn (and maybe you'd prefer to use the one with fewer +1/+1 counters on it just in case your opponent has removal!) #wotc_staff

What's a "new feature" for us? This has always been a pretty interesting question to me, for a code-generating system. When a vanilla creature comes out, do you recommend we make a regression test for it? What should the content of that test be? What about a french vanilla creature? What if we have tests for "Draw two cards" already but now a card comes out with "Draw five cards" - is that a new feature?

Our line is "involved developer changes to the parser or engine". This does miss bugs, but in my opinion it is rare. And the greater focus on "new work" allows us to put much more attention in testing the boundary scenarios for the riskiest new behaviors.

Slightly before I joined WotC 6 years ago, our regression test framework was much more inconvenient and brittle, but pretty much from day 1 of engine development there has been some form or another of testing.

As for our strategy for testing, our normal standard is a scripted game with assertions a...

Well, the dream has always been for the card parser to be a massive productivity boost for backfilling MTG's card catalog. There are a few reasons why it isn't just a snap-of-the-finger though:

• The Pareto Principle applies: the parser is excellent at handling normal MTG card text, but a sizeable number of MTG cards do things that really no other card does. For example, [[Void Winnower]]'s prohibition on casting even-mana value spells would play some havoc with casting X-cost spells. Perhaps we could just dump the large proportion of cards that work "for free" in engine...

• ... but in-engine isn't the only concern. There's also the client experience to consider. The engine's been worked on for longer and supports some interactions that the client has never needed to implement before. Plus there's our standards of presentation: we want new content we release to meet our standards of clarity to players and to work with our auxiliary systems like autotap...

That's an excellent question! The unfortunate truth is that Gutter Grime's implementation came in pretty late, apparently after Blazing Torch was retested for release. That's pretty abnormal, and mega-unfortunate. #wotc_staff

Machine learning is not used in our parser. The generation of code is intended to be deterministic, which is a feature machine learning is not a good fit for. Our natural language processing techniques are more old-school stuff like generating syntax trees from grammatical productions and encoding semantic meaning with first-order-logic expressions. #wotc_staff

I can't really, as it's outside my area of the code. I'm focused pretty much entirely on the "playing a game of MTG" side of things. I do know that getting that system working again is a priority for us. #wotc_staff

When we made the Gutter Grime change, our test for Toralf's Hammer failed. That led to us actually carving out an exception to the "delete the ability constraint" for "unattach" costs, which led to that test passing again. We really should have taken that as a warning sign that other similar cards may have issues, certainly. But the upshot is Toralf's Hammer never ended up buggy in release. #wotc_staff

That is absolutely intentional. You should also be able to tell which Serra Paragons have been "used up" (and may want to change which you prefer based on the state of the different Paragons). #wotc_staff

You know, I think that actually is an undesired behavior. Our rules for providing an alternative cost to an existing action, as Runeforge Champion does, do not have the logic of "is the source object meaningful" like our rules for providing novel action permissions do. I think I'll make a ticket for that, thanks for bringing it to my attention! #wotc_staff

Boons have a few things going for them:

• We (Studio X card design and us Arena devs together) first got the idea after puzzling over Y22-MID's [[Tenacious Pup]]. We wanted an embedded trigger inside a triggered ability, and for that to be the outer trigger's only effect. But something like "When CARDNAME enters the battlefield, when you cast your next spell" was dissatisfying to read. For the Pup, that's why the 1 life is tacked in there. But we didn't want trinkets like that to be the long term solution. So advantage #1 is that it's a more pleasant read.

• Boons are currently digital-only, as they sort of have a memory issue for paper. More digital design space is fruitful for us.

• As they are digital-only, we have more flexibility in adjusting the game rules around them if design needs that compared to paper mechanics.

One disadvantage is that it's pretty awkward to make downside boons, given the connotation of the w...

In practice, at least 75% of the cards' generated code do not need individual attention. The advantage of the language of MTG cards is that it's very precise. This is what allows our project to not be a fool's errand in the first place. #wotc_staff