Original Post — Direct link

tl;dr - DIM's stats, especially for perks/mods, are more accurate to what's in game now, and we used math to do it.

Hi folks, this is Ben, one of the people who builds DIM (Destiny Item Manager). I wanted to talk about some changes I made to DIM recently, and how they affect what you see in the stat bars and perk/mod tooltips. We're doing a bunch to prepare for Shadowkeep and Armor 2.0, and having accurate stats is an important part of that.

A few weeks ago I shipped a change to how we calculate stats for DIM that brings it more in line with how Destiny actually works (though not perfectly, as we'll discuss). This means that our stat bars and perk/mod values are more true to the effects you'll actually see in game. This can lead to some unintuitive results, for example a tier 10 masterwork mod adding +9 to a stat (instead of +10).

DIM works by using the Bungie.net API, which is the same API that all other Destiny apps and websites use, including the Companion App. Bungie generously makes this API available to everyone to build on, and within it is a wealth of information about how the items and systems of Destiny work. DIM reads your live API data about your characters and their inventory, and combines that information with "the manifest" which is a mostly static database (also from Bungie) with information about all the different parts of the game.

For each item, there are a number of stats (Impact, Resistance, Magazine Size, etc.) which have different effects on how the item works. Each stat has a value called the "investment value" which is the raw value of the stat, usually from 0-100. Then, this investment value gets transformed by per-item scaling functions that map them into the value you see in the game UI. It's actually more complicated than that - there are scripts that can run in the game that modify these values further!

The Bungie.net API provides apps with "calculated stats" which precalculate the effects of these per-item scaling functions. This gives a pretty accurate view of most stats, but even they cannot make use of the scripts that are used in game, so what Bungie.net (and DIM) reports as the stat values cannot ever be 100% accurate. Magazine Size is one of the stats that's notorious for being impossible to calculate accurately from just this data.

DIM used these calculated stats up until recently, with a few exceptions. First, we show a handful of "hidden stats", like Aim Assistance, Zoom, and Recoil Direction. These are never surfaced in the game UI, and the Bungie.net API doesn't provide them as calculated stats. Instead, we showed the raw investment value for those stats. Second, we also showed the value that perks and mods would apply to your stats. In game this is only vague text like "increases range", but we wanted to show the actual numbers. However, we only had access to the investment values, not the calculated values, so that's what we showed.

The recent change in DIM was to switch to calculating all stats on our own, using the per-item scaling functions to transform the investment stats into calculated stats. This let us show the proper value even for hidden stats, and we can also show the actual amount perks and mods contribute. You may have noticed that before this change, perks like "Extended Mag" would show "+20 Magazine" when it really couldn't add +20 - the real value is only a few extra bullets in the mag.

So let's explain the math. I'll use one of the puzzles I mentioned before - how can a +10 masterwork only add +9 to a stat? My New City submachine gun is masterworked for Reload Speed, and before my recent change hovering over the mod would say it adds +10 Reload Speed. But now it shows +9 Reload Speed. Why?

First, we look up the inventory item definition for New City in the manifest. You can see it here. Under stats there is a statGroupHash that points to the table of stat scaling functions for this weapon. You can see the stat group definition here. This has a table of scaledStats which defines a piecewise linear function for each stat that maps its 1-100 investment value to its real in-game value.

Going back to the inventory item definition's, we can see in investmentStats that the base investment value for the Reload Speed stat is 48. But there are also perks and mods, which are modeled as sockets, that add to that value. The tier 10 masterwork adds +10, and the active Tactical Mag perk adds another +10. This gives us an investment value of 68.

Now we look up the Reload Speed scaling function from scaledStats table in the stat group. It looks like this:

value  |  0 | 100 
weight | 10 | 100 

This is a simple table with only one segment - some stats have multiple segments. What this is saying is that when the investment stat is at 0, the game stat is actually 10, and when the investment stat is at 100, the game value is also 100. Inbetween, we linearly interpolate the weight. This means that every point of investment stat actually only adds 0.9 points in game.

To calculate this out, we figure out the start and end of the segment our value lies in (which is easy in this case, since we only have one segment). Then we apply this function:

t = (value - start.value) / (end.value - start.value); 
gameValue = round(start.weight + t * (end.weight - start.weight)); 

Substituting in the real values, we get:

(68 - 0) / 100 = .68 
10 + .68 * (100 - 10) = 71.2 (rounds to 71) 

So the final Reload Speed for my New City is going to be 71. But where did we get +9 for the masterwork mod? In DIM, we figure out how much an individual perk or mod would add by subtracting the investment value of the perk or mod from the total investment value of the stat (after adding all perks and mods) and then recalculate the stat as if that perk or mod wasn't active. Then we subtract the two numbers to find out how much the mod added. In this case, without the +10 masterwork the investment value for Reload Speed would have been 58 (we still have Tactical Mag), and running it through the formulas above produces a calculated stat value of 62. 71 - 62 is 9, so our tier 10 masterwork actually only adds 9 to Reload Speed.

If you stuck with me through this long post, hopefully you have some understanding for how apps like DIM work, how we figure out stats, and why DIM now shows different values than it (and many other tools) used to.

External link →
over 4 years ago - /u/EdgarVerona - Direct link

Originally posted by vid_boi

Yup, I've been in conversation with /u/EdgarVerona about this! Still trying to nail down exactly how it works but he confirmed in this issue that banker's rounding should be used.

Yes! Yeah, it's a really interesting and unfortunate story!

At some point I'll have to see if I can dig up the dissertation that I wrote when we first discovered the Magazine Size issue back in Destiny 1, but the TL;DR is that this is one stat in particular where the stat is so heavily modified by non-stat systems (the sandbox, the activity you're in, scripts that are run on the server etc... when you have certain perks applied, etc) that Bungie.Net doesn't have an authoritative way to calculate the stat, because we can't get at those. We think that potentially even the rounding algorithm being applied is different than it is for other stats, but I haven't had the time to investigate that. But what we can say is that in some situations you can even watch your magazine size change significantly even when doing something like going into an activity vs. being in orbit.

Back in D1 I almost pulled it out of the API, because I didn't want to return a stat if it was never going to be accurate - but API users at the time spoke overwhelmingly in favor of keeping it: they felt that having a stat they could "compare with" even if it wasn't accurate was better than having no stat at all. So here it has remained, the saddest of all the stats.

EDIT: Ah, here you go, this is what I wrote in the documentation about stats:

https://bungie-net.github.io/multi/schema_Destiny-Definitions-DestinyStatDefinition.html#schema_Destiny-Definitions-DestinyStatDefinition

over 4 years ago - /u/EdgarVerona - Direct link

Originally posted by Donny_Do_Nothing

Phew, yeah we never did get the chance to loop back around to putting it in our app again! But hopefully you all have found the data useful.

over 4 years ago - /u/EdgarVerona - Direct link

Originally posted by Donny_Do_Nothing

Hey, the legend himself!

I have a boss who likes to say, "Complaint is commitment."

I wouldn't have anything to say about mag size if I weren't in there looking at all the other awesome things already.

Cheers!

I'm really glad you all enjoy it!

over 4 years ago - /u/EdgarVerona - Direct link

Originally posted by blue_13

Thanks for the explanation and for what you and the team do! You guys are awesome!

Thanks, I appreciate it! I'll say that DIM's quality and breadth of features is really fantastic and always amazes me when I see it. I've never written an API for public use before coming to Bungie, and seeing someone take raw data and just absolutely run with it to make something incredible has been a fun experience! Folks are taking this API in directions that I never would have imagined, and I'm really impressed by it!