There was a big thread a while back, but we seem not to have turned it up yet. The idea is to install some kind of L-pad between the SEX speaker output and the headphones. Even an 8 ohm adjustable L-pad from a speaker crossover would work, since the SEX is perfectly capable of driving an 8 ohm load. This would attenuate the output, while providing a phone output whose impedance is significantly less than 8 ohms, i.e. pretty good damping for 32 ohm cans.
Try this: short out the 120 ohm resistance so the headphones are connected directly to the output transformer, turn the volume control all the way down, and listen. If you don't hear any noise then the SEX amp itself is quiet enough to directly drive your headphones, and you can attenuate the input to get a suitable volume control range. If you do hear noise that is obtrusive (obviously nothing is completely silent) then you need to attenuate the SEX output before it gets to the phones.
Per the IHF standard, the 120 ohms resistor provides that attenuation for low impedance phones. The problem is many modern phones are designed to work with iPods (low voltage, low impedance) without regard to whether they work well with the IHF standard - manufacturers are ignoring the standard so it's becoming obsolete. You can use an 8 ohm loudspeaker L-pad (the kind used to adjust tweeter level in a two-way system, for instance) or a wirewound pot, of say 10 to 50 ohms, to find out how much attenuation is needed to get the phones quiet. That will reduce the system gain, but also the system headroom, so you don't want to attenuate more than necessary here. Once you've found the required level of attenuation, you can keep the L-pad or substitute another with the same attenuation but made of fixed resistors, and possibly a different input impedance.
After that experiment, if the amp is still too sensitive then you'll want to install some more attenuation at the amp's input. That includes the case where the straight output is quiet enough, but the volume control setting is too low for a useful range. The series resistor of 33K to 470K that PB (Caucasian Blackplate) described is appropriate for this function.
We've been looking at the idea of very small, inexpensive, passive kits - just a Sweetest Whispers in a box, or a baffle step corrector, etc. I'll see if I can come up with something suitable for these headphone sensitivity-optimization experiments. Headphones cover wider ranges of sensitivity and impedance every day it seems, and it's getting quite frustrating!