Why not make it so the queue disappears for you as soon as you decline it but stays till the end for the rest of the people who haven't pressed any button? This way the decline button does something while still filtering out the AFKs.
I advocated for this when we were originally designing the system, and implementation concerns were serious enough that we just went with the simpler behavior of keeping everyone in AFK check till everyone accepts/declines/times out. The hazard is that our queue times are so low in popular queues that your queue could pop in a new match before the old AFK check expires which would necessitate a lot of logic to handle that correctly and higher risk of bugs that could be hard to find and fix. That part of the game flow was never designed to pull individuals in or out of a match.