It sounds like you learned something along the way.
We did indeed! Most 32 ohm headphones are incredibly sensitive and wiring the output transformers to 32 ohms leads to a very high noise floor that is completely undesirable with these HPs.
But to me the logic seems backwards.
How the OTs are wired is a balance of available power, output impedance, and noise floor. If you drive 32 ohm headphones with the S3X amp wired for 4 ohms, you are increasing the damping, decreasing the noise, and decreasing the available power. For most 32 ohm headphones, power is not so much of an issue.