We've a fix for this in the next patch! Sorry for any disruption this has caused :(
The bug in question is that penalties are incorrectly applying in Custom Games if you dodge agent select specifically. No in-game penalties are applied otherwise in Custom Games.
The default punishment duration for this is ~3minutes, so you shouldn't be seeing an hour+ ban unless you're repeatedly (many times) AFK'ing through Custom Game agent selects OR the punishment from the Custom Game dodge happened to get stacked on-top of having recently been AFK'ing non-Custom Game matchmaking agent selects and/or in-game rounds.