Random Stuttering Issues

I've been having an issue since some time after the horizon=odyssey update. I will randomly have massive frame time spikes. It seems to be worse in VR but is present in flat screen too. In VR I have a frame time meter for GPU and CPU, and it shows they are purely CPU frame time spikes.

The weird thing is that it doesn't seem to be environment dependent. I can be in the exact same area, with the same things going on, and one moment it will be running fine, the next it will be running like sh*t. It also seems to get better the longer I've been in a particular session.

I've tried reinstalling the game, messing with every setting, enrolling and unenrolling in steam-vr beta, killing all non-essential programs, still an issue.

Hardware: i5-12600k, MSI Gaming Z trio RTX 3080, Windows 11

Here's a screenshot that shows one of the frametime spikes, top left:https://imgur.com/a/qPY6uE3

You can see part of the time, the frametimes are smooth, but theres a spike in roughly the middle of the graph.

Sometimes these spikes will be relatively short in duration (like the one above), sometimes they're 30 plus seconds. They also vary in intensity, usually its just a bit stuttery but sometimes it is briefly unplayable.

Anyone else experiencing this issue?