[2022] Dropping from 60fps down to 6fps

Started playing 2 days ago, got to level 30 and did a survival with a friend, got out and opened my stash, game started dropping fps, Went from 59/60 fps down to 30 for a little bit then went lower to 15 until it hit 6 fps, sometimes I would even go as low as 4. Haven't had any problems and it only started the issue after I successfully did the survival. Now the problem wont go away and will launch game at 60 and slowly go back down even if I dont do anything. Tried to verify game files, updated drivers, uninstall and reinstall, clear cache, and the issue is still happening.
CPU: AMD Ryzen 7 2700X Eight-Core
GPU: NVIDA GeForce RTX 2070
Ram: 16GB
OS: Windows 10