Headphone amp: when/why is it necessary or beneficial?

[troplin]

I've preordered one of the new S.E.X. 3.0 kit and I'll use it with speakers.
But since this is also a headphone amp I wonder what kind of headphones will benefit most from an amp and why.

I've googled a bit and mostly I've read that you need an amp with high impedance headphones.
That makes sense but I suspect that cannot be everything. At least the sensitivity will also be relevant.
I mean, just as an (arbitrary) example, let's take the Sennheiser HD 650 (because it seems to be popular here):
- Impedance: 300 Ohm
- Sensitivity: 103 dB/mW

An iPhone 6 (again arbitrary example) can apparently deliver about 3 mW, into a 300 ohm load, that's almost 108 dB which seems plenty of head room.

So, why would I use a dedicated amplifier with that headphone? Or is my math flawed?
I totally understand that you need a good amplifier with speakers. There the choice is good vs bad amplifier. By using a good amp you will avoid using a bad amp.
But with headphones the choice is a good vs no amp. You cannot just skip the soundcard. So why is *adding* another device beneficial?
Still people do it. Is it just some added coloring/distortion?

[Paul Birkeland]

There are several different answers to this question that depend a bit on the headphones you're starting with.

The easiest answer would be for those who own something like a Hifiman HE-6 or an AKG K-1000.  These are very power hungry headphones and not particularly high impedance either.  In this case, the amount of power that can be sourced from something like a phone is severely insufficient.

When you mention the HD-650's and the 103dB/mW sensitivity, it is actually 103dB@1V RMS, which is about 3mW at 300 Ohms.  When you try to get this out of an iPhone 6 for example, that's all that's there, once you exceed 3mW, you enter hard clipping.  This will be especially audible when listening to music with decent low frequency content at anything other than a moderate volume.

Tube headphone amps don't exhibit that behavior, and in the case of the SEX and Crack, you have so much headroom that this is no longer concerning.
 
Luckily for you, you can easily make the comparison on your own once your SEX is built