If this is an unmodded game, this is almost certainly a bug. Please make a new thread in Bug Reports and attach an affected save game for QA to look into.
I'm sorry but I cannot help with modded games here. If you get this in an unmodded game please report it as I suggested; if only in a modded game please post in User Mods.