I've had this happen a couple times before. I believe it was because the minimap was still trying to autocenter (although not visually) when you start dragging again and have to wait a couple more seconds. This was a long time ago so I could be talking out my butt.