Originally posted by ThoseThingsAreWeird
Therefore, we don't create such tests for every new card on MTG Arena
This surprises me a little bit, but it probably has a reasonable answer.
We create regression tests for each new feature, but we've done that from the start. So yeah that adds an extra bit of time onto creating each feature, but we've got a certain level of confidence that we're not breaking stuff in the future (assuming we right the tests correctly, which we always do every time ever...). In the grand scheme of things it's a lot of time, but for each release it's a relatively small amount of time.
Was there a period of time when you weren't creating regression tests? Or is it that your approach to regression tests wasn't covering every Rule? Presumably covering every Rule, would mean you cover every card with an ability? Or actually, that'd need to have regressions on every Rule interacting with every other Rule... Ok yeah I see where this is going...
...
Read more
What's a "new feature" for us? This has always been a pretty interesting question to me, for a code-generating system. When a vanilla creature comes out, do you recommend we make a regression test for it? What should the content of that test be? What about a french vanilla creature? What if we have tests for "Draw two cards" already but now a card comes out with "Draw five cards" - is that a new feature?
Our line is "involved developer changes to the parser or engine". This does miss bugs, but in my opinion it is rare. And the greater focus on "new work" allows us to put much more attention in testing the boundary scenarios for the riskiest new behaviors.
Slightly before I joined WotC 6 years ago, our regression test framework was much more inconvenient and brittle, but pretty much from day 1 of engine development there has been some form or another of testing.
As for our strategy for testing, our normal standard is a scripted game with assertions a...
Read more