I had this exact issue a few weeks ago when I upgraded to a vega 56.
I stupidly forgot to check that I had the right cables, as the card was 3x displayport and 1x hdmi, so I could only connect one monitor to it. The cables were going to take a week so I turned on integrated graphics and hooked up the other two monitors so I could use them.
This made NXT have a fit, with the exact error you're getting, and if I turned on safe mode it was like my character took LSD.
Try going into device manager and disabling integrated graphics if you have it enabled.
There is a known issue when the client is run on a system with 2+ independent GPUs driving at least 1 monitor each
It should be ok to have integrated graphics enabled as long as they aren't driving a monitor