[2023] A Bug to Pierce Flesh and Spirit Alike - The Story Behind the Citizen’s Crowbar and Ninja’s Kunai Bug

I had first thought the issue was with anaphora resolution (having the parser correctly identify what the self-reference meant). That is, the bug would have been "the parser thought some other phrase was what that self-reference meant". It turns out it did that correctly, we just squashed the meaning out of the result! #wotc_staff

In practice, at least 75% of the cards' generated code do not need individual attention. The advantage of the language of MTG cards is that it's very precise. This is what allows our project to not be a fool's errand in the first place. #wotc_staff

Boons have a few things going for them:

• We (Studio X card design and us Arena devs together) first got the idea after puzzling over Y22-MID's [[Tenacious Pup]]. We wanted an embedded trigger inside a triggered ability, and for that to be the outer trigger's only effect. But something like "When CARDNAME enters the battlefield, when you cast your next spell" was dissatisfying to read. For the Pup, that's why the 1 life is tacked in there. But we didn't want trinkets like that to be the long term solution. So advantage #1 is that it's a more pleasant read.

• Boons are currently digital-only, as they sort of have a memory issue for paper. More digital design space is fruitful for us.

• As they are digital-only, we have more flexibility in adjusting the game rules around them if design needs that compared to paper mechanics.

One disadvantage is that it's pretty awkward to make downside boons, given the connotation of the word! #wotc_staff

You know, I think that actually is an undesired behavior. Our rules for providing an alternative cost to an existing action, as Runeforge Champion does, do not have the logic of "is the source object meaningful" like our rules for providing novel action permissions do. I think I'll make a ticket for that, thanks for bringing it to my attention! #wotc_staff

That is absolutely intentional. You should also be able to tell which Serra Paragons have been "used up" (and may want to change which you prefer based on the state of the different Paragons). #wotc_staff

When we made the Gutter Grime change, our test for Toralf's Hammer failed. That led to us actually carving out an exception to the "delete the ability constraint" for "unattach" costs, which led to that test passing again. We really should have taken that as a warning sign that other similar cards may have issues, certainly. But the upshot is Toralf's Hammer never ended up buggy in release. #wotc_staff

I can't really, as it's outside my area of the code. I'm focused pretty much entirely on the "playing a game of MTG" side of things. I do know that getting that system working again is a priority for us. #wotc_staff

Machine learning is not used in our parser. The generation of code is intended to be deterministic, which is a feature machine learning is not a good fit for. Our natural language processing techniques are more old-school stuff like generating syntax trees from grammatical productions and encoding semantic meaning with first-order-logic expressions. #wotc_staff

That's an excellent question! The unfortunate truth is that Gutter Grime's implementation came in pretty late, apparently after Blazing Torch was retested for release. That's pretty abnormal, and mega-unfortunate. #wotc_staff

Well, the dream has always been for the card parser to be a massive productivity boost for backfilling MTG's card catalog. There are a few reasons why it isn't just a snap-of-the-finger though:

• The Pareto Principle applies: the parser is excellent at handling normal MTG card text, but a sizeable number of MTG cards do things that really no other card does. For example, [[Void Winnower]]'s prohibition on casting even-mana value spells would play some havoc with casting X-cost spells. Perhaps we could just dump the large proportion of cards that work "for free" in engine...

• ... but in-engine isn't the only concern. There's also the client experience to consider. The engine's been worked on for longer and supports some interactions that the client has never needed to implement before. Plus there's our standards of presentation: we want new content we release to meet our standards of clarity to players and to work with our auxiliary systems like autotap, automatic trigger ordering, etc.

• And still importantly, it needs to make business sense. Even though the work is cheaper than a new set, it's not free to produce and would similarly not be free to distribute. We want to productize the back catalog and make fun experiences like the remaster sets have been, rather than just dumping thousands of cards that you would acquire... how, exactly?

#wotc_staff

What's a "new feature" for us? This has always been a pretty interesting question to me, for a code-generating system. When a vanilla creature comes out, do you recommend we make a regression test for it? What should the content of that test be? What about a french vanilla creature? What if we have tests for "Draw two cards" already but now a card comes out with "Draw five cards" - is that a new feature?

Our line is "involved developer changes to the parser or engine". This does miss bugs, but in my opinion it is rare. And the greater focus on "new work" allows us to put much more attention in testing the boundary scenarios for the riskiest new behaviors.

Slightly before I joined WotC 6 years ago, our regression test framework was much more inconvenient and brittle, but pretty much from day 1 of engine development there has been some form or another of testing.

As for our strategy for testing, our normal standard is a scripted game with assertions about the game state (or sometimes the internal engine state for stuff like memory leak fixes). They certainly involve a lot of MTG knowledge about "what are tricky interactions to verify". Sometimes, with more algorithmically complex features like autotap (or, perhaps surprisingly, the ZNR party mechanic), we develop unit tests that directly test internal components of the engine. #wotc_staff

The notion is it doesn't matter which Falco you use - the action behaves the exact same way for either. For Serra Paragon, it does matter which you use - that one can't be used again this turn (and maybe you'd prefer to use the one with fewer +1/+1 counters on it just in case your opponent has removal!) #wotc_staff

The problem with taking a recording of a game and saying "make sure it plays like that again" is in determining what "like that" means. We do plenty of changes to the game that don't change the gameplay outcome but do, for example, change the information in requests and responses to the client, change what information is available in the game, change autotap strategies, etc. The advantage with our "scripted game" tests is that we're able to decide precisely what is important to verify with automated assertions, and what aspects of the game's proceedings are allowed to vary over the development of Arena as a project. #wotc_staff

We test cards that involved a developer's effort to get to work in the first place. Human QA does a pass over a set to identify what didn't automatically work from the first time we generate code for a new card set. Anything that doesn't work at that point is, well, my day job! And work we do there gets verified against regression by an automated test.

When we're closer to release, QA does another full pass to hopefully identify regressions, again focusing on the new cards due to the huge explosion of possible interactions.

I identified in the OP the relevant cards in the story: Heliod's Punishment has plenty of tests that lean on "self-reference in conferred abilities". Unfortunately Heliod's Punishment's behavior doesn't involve the ProposeEffectCostResource rule, which was the center of this bug: its conferred ability's only cost is the tap-symbol. #wotc_staff

Can you clarify what your suggestion is? "Printed on" is a pretty ambiguous concept:

• What about copy effects? If card A becomes a copy of Gutter Grime and triggers to make an Ooze, the reference to "Gutter Grime" on the Ooze means "Card A".

• That still holds true even if Card A stops being a copy of Gutter Grime.

• Through horrible shenanigans you're able to make The Book of Vile Darkness create a Vecna token that has Gutter Grime's triggered ability. In that case the "Gutter Grime" phrase on the Ooze it creates refers to the Vecna that made the Ooze token. Was that ability "printed on" Vecna?

I think perhaps what you're trying to say is "Gutter Grime" in the conferred ability refers to "the card that conferred this ability to this Ooze". But that's the whole point of this post - identifying when a self-reference is like that is nontrivial. Our original logic, due to the cards we had covered on Arena, was myopically focused on attachment cards. Gutter Grime challenged that assumption, and us changing our code to account for that led to this oversight. #wotc_staff

I believe that had been a UI bug, where the client was improperly batching the Muldrotha permissions in its presentation of your actions. #wotc_staff

I'd love to tell more stories about "challenging developments that went smoothly". I think there's a couple challenges to that:

• Less of a narrative! With a bug, there's an immediate hook of "how did that happen", then a cool investigation, a eureka of the issue, and often an embarrassingly simple fix (this bug was fixed just by deleting a line of code!). I think that makes for a pretty clear flow. Most implementation stories don't have such a narrative structure to them, which makes them harder to write about.

• Scope of background. Even this post had coworkers dozing off with the groundwork I presented to describe the bug. New features are often even less cleanly described.

• When? What? It can be hard after-the-fact to decide what would make an interesting story to talk about, or when to talk about it.

Still, the reaction to Ian's post and this has us pretty interested in doing more. Heck, I've always wanted to! I'm sure we'll overcome the above challenges haha. #wotc_staff

Perhaps for an LTR implementation tale! #wotc_staff

Pretty broad question. In one sense, we're somewhat similar: we both make code happen starting from English strings from new cards to make a good MTG play experience. But our engineering is completely different, from code generation to the actual engine design. #wotc_staff

Two main ways:

• The parser fails to generate code. This is great! It's recognizing that something is outside its current boundaries. We usually have a good idea of what we need to do from the error messaging.

• The parser generates wrong code. Less great. Human QA needs to play the card to see that it's doing the wrong thing. The most common type of problem here is with "anaphora resolution" - figuring out what ambiguous phrases like "it" or "that creature" mean. Why, I just estimated the complexity of a few LTR bugs with that issue moments ago... #wotc_staff

Tokenization is a component of our parsing process, one very early in the process. It's true that replacing one token with another similar one is often not worth considering to be a big difference. But what about one sentence structure with another? For example "If you would draw a card, draw two cards instead" vs. "If you would draw a card, instead draw two cards" should behave the same despite being worded differently (and both are in fact valid wordings). If we already handled a phrase like "If CARDNAME would deal damage, it deals twice that much damage instead" as well as "If CARDNAME would deal damage, instead it deals twice that much damage", then we've already handled that syntactical difference. Let's say we wrote tests for the latter two cards; we'd find in ad-hoc testing that once we got either version of the draw replacement working (and wrote tests verifying it), the other one would work too. Given that it worked "out of the box", how much effort should we spend testing the new wording? As much as for the wording we had to do work for? It's a tough call, but it's not economical to test every card equally.

what sort of stuff doesn't that catch?

I suppose one type of bug we sometimes miss for cards we don't work on is that we don't do as deep edge-case hunting for the human passes (after all, they have hundreds of cards to get through). When we do work on a card, multiple people brainstorm ways it could go wrong and we check those. With the file pass, there are fewer eyes on it. Internal playtesting does help find issues though.

Are those assertions based on what the parser is saying the card should be doing?

Our tests are pretty much entirely hand-written, so it's a dev's judgment about what's worth paying attention to. We have to update tests sometimes as behaviors change, but that's also true for paper cards. In terms of Alchemy changes, [[Static Discharge]] has had its test updated a lot due to its wording changes, and I'm sure we've not seen the end of that haha. #wotc_staff

I guess my point I'm trying to make is that we do have such a concept (it is a bit hard to discuss given that "ability" means three separate but tangled concepts in MTG). There absolutely is a relationship between an ability-on-card and the card that possesses it, and that's normally how self-references are interpreted. A large part of the complication here (and complication -> misunderstanding -> bugs) is that self-references mean different things in different contexts in MTG text. Your summary of the correct answer is pretty accurate, and it reflects our current logic, but getting there took some iteration and seeing more examples of abilities that contradicted our understanding. #wotc_staff

Another factor here is QA time. Just because the parser thinks it understands a card, it doesn't necessarily mean that it's right (there are some great stories here that we'll tell sometime). We need to either write automated tests to validate behavior (which takes time) or have QA manually test the card in a variety of scenarios (also takes time).

We want to expand Arena's card pool just like players want. At the most obvious level, releasing new cards directly makes us money. But, more than that, we all work here because we love Magic, and we love Arena, and we want it to continue to grow. But we need to balance that with the people we have who can do the work required. As Ben noted, we're also hiring.

We are unable to reproduce this bug, on live or in our dev environments. Are you sure the reproduction steps here are accurate? #wotc_staff

We almost never make kludges for cards - our normal workflow is to build solutions that would handle variation: what similar designs would have the same issue? Can we preemptively handle those? For example, for [[Muldrotha, the Grave Tide]], our solution made it so we could handle a similarly worded card that allowed you to play different colors of cards, or different land types, etc., even though Muldrotha is one-of-a-kind. #wotc_staff

Ack, sorry I missed this one! There are plenty of reasons to regenerate card code:

• The biggest reason is consistency. We do not want to be in a world where two nearly identically worded abilities work totally differently because one was "locked in" years ago and behaves totally out of date. Worse yet would be something like two different printings of the same card behaving differently!

• Frequently, updates we do to new cards want to apply to old cards too, particularly for rules-tangential behavior (client communication, autotap, trigger ordering, etc.). If we've identified behavior in an ability we want to treat differently for those tasks, it's good to apply it to all abilities with that behavior.

• Rules changes aren't super infrequent in MTG. Having our code be regenerated makes it so that we don't have to manually identify which cards are affected by the rules change - as long as we correctly handle the new requirements generating code for the text, those changes will come "for free" to the old cards. Of course, they still need to be tested! As an example, as we've recently anounced, the new Battle card type is now choosable for "any target". Instead of needing to find every card with that phrase (as well as stuff like "any other target" etc.), we can just change the lexical definition of those phrases to include battles, and all the cards with them will automatically now accept battles as legal targets. #wotc_staff

It's a reference to the flavor text of the Kunai! Thanks to /u/WotC_Megan for the idea. #wotc_staff

Oh, one other great reason is that the whole point of our regression tests is to identify situations where we end up generating wrong code. If we make a change for a future card that would break an old card (in a way that we've tested), we really want the old card's code to change too: our test will fail and we'll notice that the change we're making is the wrong change to make. If the old card's code is frozen, then our regression tests aren't really doing much for us! #wotc_staff

A syntax error due to being unfamiliar with the phrase "After that turn". #wotc_staff

Have you checked it this week? The fix was released with Shadows over Innistrad: Remastered. It took so long because the development team wasn't aware of it until somewhat recently; we're working on improving the user bug report pipeline. #wotc_staff