I haven't found a way to fix it without restarting the game. It's on a list of high priority bugs to fix.