Many things don't feel good but are not fixable in practical terms. Sometimes things can be changed, but they are not worth the investment. Redoing resource management would be nice, but would take months of time and then may only get little to no gain.
There is a fixed cost, since it has to scan all loaded resources, so doing it say every minute would be more annoying to only save .1 secs. Games typically hide this stuff with loading screens between levels/stages/whatever, which we don't have. Maybe someday Unity will have an incremental way of doing that.
Being Unity (or really any engine), there are often uncontrollable things we have to live with. Code can only be improved so much before you hit a performance wall.
I have 32GB, so maybe, but I've never heard of Unity doing anything different with 64GB.
F1 key opens the console.