I think the irritation comes from having to switch it every run, and 1 moment of "I forgot to switch" can easily lead to showing up in death's office. The outfit will be left in storage as long as needing the brace is a thing as not getting PK'd is a higher priority than the benefits the outfit would provide. Letting it work from inv (even if you have to add a cost, either one-time or charges) would be a nice QoL in line with this, or maybe a different option, like being able to add it to the puzzle box, or the infinity outfit gloves themselves.
Yeah there's definitely a bunch of possible reward spaces or tweaks that could be made here.