[2020] DirectX 12 Debugging

Seasons Greetings, Tenno!

Warframe calls for aid!

First of all I want to shout-out to those of you already testing our beta DirectX 12 support! In the first week since launch we’ve turned around fixes for almost all of the crash-bugs thanks to your help!

At DE we have a leaderboard that helps us prioritize the most frequent and serious bugs. Right now, despite about only 1% of players using the beta driver, the top of the leaderboard is currently a DirectX 12 crash:

The trouble is, unlike crashes in our code, these bug reports give us very little information because the code that’s crashing is actually in the graphics driver. All we know is that we’ve probably done something wrong and the driver crashed.

The bugs come in two main flavors: a common one that happens randomly, and a rare one that happens before you can even login. Although we want to fix both we’re most interested in the rare one because if it happens every time we can set your game into a special diagnostic mode that might tell us what we’re doing wrong.

We’ve also integrated NVIDIA Nsight Aftermath into the game to try to catch these problems in the act -- in theory we should be able to look at the GPU crash dump in a tool at our end and figure out what went wrong. All we need now is someone with the magic touch!

To help all you need is an NVidia GeForce 1050 or better (it doesn’t handle crashes for AMD drivers). If you qualify and accept this challenge:

1) Close Warframe: any edits to your config will be overwritten when the game shuts down so make sure it isn’t running.

2) Find your config: open up File Explorer and paste %LOCALAPPDATA%\Warframe in the address bar at the top. In that folder you should find a file called EE.cfg that you can edit in Notepad. At the bottom there should be a Windows_Config section where you can add Graphics.DebugDriver=3 in there like this:

3) Boot the Warframe Launcher and click on the gear icon in the upper-right corner. Make sure DirectX 12 is selected like this:

4) Play the game until it crashes.

If you catch a driver crash:

1) Please run through the crash reporter and set aside the WAR-Number.
2) After it submits, go back to the %LOCALAPPDATA%\Warframe folder you had open before and there should be a Crashes folder inside there now.
3) If Aftermath caught a driver-crash there should be a file called GPU.bin there; we have no idea how big these files are -- they might be huge -- so right-click on it and under “Send to” select “Compressed (Zipped) Folder”
4) Go to http://support.warframe.com/ and open a ticket and title it ‘DX12 CRASH FOR DEVS’; attach the compressed GPU file and include the WAR-Number if possible.

Final notes:

- Graphics.DebugDriver=3 only works for DirectX 12
- When enabled it may have a small impact on framerate (again, only in DirectX 12)

Thanks in advance! Once we can ensure DirectX 12 is as stable as DirectX 11 is we can move on to making it even faster and smoother!

1 hour ago, Jarriaga said:

Hi @[DE]Glen

Would this work to help isolate the cause of this behavior I reported

No.

8 hours ago, Sahysa said:

"Too bad that nsight aftermath requires pascal (I have a 980) otherwise I would've helped xD"

Doubly so for me because I have the the crash on DX12 startup. 

 

If only you'd kept the WAR#

6 hours ago, Debbysheen said:

noice! i've tried DX12 some days ago and the only thing was that it is too choppy, maybe my pc wasn't strong enough, today i got a 3070, im gonna test it out this weekend and see what i can find.

This is WIP -- it smooths out the longer you play. We can optimize that but right now we're focusing on making it stable.

9 hours ago, Arcerinex said:

I've got a 1060 so I can give it a shot... Haven't had any crashes thus far. Any ways I could possibly try to make it crash?

• Make a lot spawn in Cambion Drift?
• Toss as many vortex grenades down as I can to create an artificial sun?
• Organize a full Mirage team spamming Simulor?

I usually play it safe.

We've tried a lot of things to stress test the game to make it crash. Combat stress-test levels, levels with 182 random loadouts, etc -- I don't think the bug is something like that since we've tried quite a lot already.

So far we've had some excellent help!

@R4ndom reproduced three graphics crashes and submitted the GPU dumps -- it's not clear what was wrong but he's going to try some more things for us!

@Srovy discovered a nasty VRAM corruption bug that could occur if you changed certain graphics settings in DirectX 12 -- a fix for that will be coming in the next hotfix.

@CrispLV managed to reproduce some strange rendering artifacts on landscape missions -- huge swathes of the land turning jet-black. Nobody here can reproduce and we suspect it might be specific to GTX 10x0 series cards in a certain config. If you see this while testing we're interested in your specs and settings!

 

13 hours ago, Sahysa said:

WAR-2977222 (fresh)

This is very interesting thank you! This is one of our rare startup crashes!

It does smell driver related -- I know you're on very new drivers but there's a few things going wrong here:

- we're asking the driver to create a basic resource and it's saying "No such interface supported"

- your driver is also claiming to not support depth-bounds test and we know your card supports it

When's the last time you let Windows Update do its thing?

2 minutes ago, Aadi880 said:

I'm not sure if this helps, but I am having repeated crashes on specifically fissure missions. Particularly, when a fissure spawns (whether on-screen or off-screen). The crash does not happen on every fissure though, only on fissures that happen at least 2 minutes into the mission. I am am on Intel HD Graphics though.

No crashes in Dx11 (not at first at least)

WAR-2980080
WAR-2981088 (latest)

We have a fix for that second one coming in the next patch.

Thanks to @R4ndom we have discovered a DirectX-12 bug in the NZXT CAM overlay that can cause crashes; we are contacting them in the hopes of getting them to fix it but for now I'd recommend disabling the overlay:

2 hours ago, Sunny_Moon said:

So, i got an issue: Warframe crashes in Dx 12 mode when entering the Login screen but my changes to the EE.cfg file are always reverted when i start the launcher therefore i can't provide a .bin file (the EE file gets deleted on launcher start and a new file is generated), the latest WAR number is WAR-2979714. The warframe process is definetly not active before i edit the file. Any Suggestions?

That crash is related to the discord overlay -- normally it works but maybe you're having problems with other overlays too (I see you have the steam overlay enabled too). Normally both of these cooperate but maybe you need to update them?

On 2020-12-05 at 3:05 AM, Sahysa said:

There are no pending updates.  2020-11 Cumulative Update for Windows 10 Version 1607 for x64-based Systems (KB4586830) was installed on 11/29.

I purged my drivers (DDU in safe mode) and installed these again with the clean install box checked.  Bios version is latest since 2018 (Skylake)

I attempted to run DX12 again with Steam and Statistics overlays disabled and it still crashes.  WAR-2977222

Not sure what more I can do. I don't have a live boot version of windows handy for this hardware.  You said this is rare: its not just my system? Is it?

Ok, we've narrowed this down to a problem on systems that are still running "vintage" Windows -- the other explanation might be a simple limitation of your GPU. In either case the next hotfix will have a workaround for this.

The hotfix that was just released should have fixes for all known DX12 driver crashes. We'll be watching to see if there are any more and then starting to work on the shader-upload stalls.

Spoke too soon; we _just_ found 2 more small crashes. We'll probably get another stage of Dx12 changes out next week.

Edited 1 hour ago by [DE]Glen

5 hours ago, Jarriaga said:

Glen, the latest patch has lessened my issue somewhat. I can no longer reproduce this problem in DX12 + Enhanced Engine + VSYNC if VSYNC is set to AUTO instead of ON. Previously, on/auto resulted in the same behavior.

It's still nowhere near as smooth as VSYNC on, but at least I can see high GPU utilization (As expected) and framerate.

Ok great - TBH I think there's still some dynres latency on Dx12 that doesn't belong but we'll get there.