Game slow downs/freezes in coop

I returned to Anno after over a year and I am having terrible slow downs - once in a while the game will almost come to a complete stop i.e. extremely low fps to freeze.

Before people start repeating what I was able to read on the forums:

• I played the game in 2022 and before with no such issues on the exact same hardware. Game used to work fine with much bigger populations, many many more buildings in my previous playthroughs.
• The other person I play with has the complete same hardware too.
• My drivers are up to date (5700 XT).
• I have 32 GB of RAM.

My guess is, one of the driver or game updates broke something but wonder if there is a fix?