Yes, only a handful of milliwats -- that's an awfully high SNR, which most amps simply don't have. I've plugged my Shure SE-425s into the s.e.x. and it was not listenable -- sure, music came through but the noise level was intolerable. Try it.
Even a good passive voltage divider now makes what is basically a straight resistive load into a complex serial/parallel network, and that has to change the frequency response curve.
You know for sure that the iFi device is fully and completely passive? I ask because it seems to me to be quite a trick to cut out the hiss, yet leave the frequency response alone, or increase the dynamics, or tighten up the bass.
I've had a number of both tube and solid state amps with headphone outs, and not one was acceptable with IEMs. So, like you would do, I built a couple of different voltage divider attenuators, and every single one of them left the bass washed out and attenuated, and severely rolled off the highs.
You really can't compare IEMs to room speakers, or even more standard cans... first there is the nearly direct coupling to the ear canal (plus a huge drop in ambient noise, and also because the moving mass of IEM drivers is so infinitessimally small even compared to standard cans, and this makes everything you use to drive IEMs far more critical when it comes to background noise and frequency response.
Yes, probably most amps that can handle an 8 to 120 ohm load (typical being about 16) ohms, will indeed play the IEMs, but the background noise will be intolerable to most. Believe me, I've done this more times than I can count, and the iFi device (whatever it is) is the first and only "attenuator" that takes away the unwanted stuff and leaves the rest intact (or possibly improved.
Believe me, if it were truly as simple as a couple of resistors, my search would have been over a long time ago. As I said, I've done this more than a dozen times and with both tube and SS amps. Hell, even many of the portable headphone amps are just too noisy to use with IEMs.
Somewhere somebody did a comparison of 10 under $200 portable (mostly) headphone amps and many of those did not make the cut with IEMs, and it would have been a very simple matter to add an "IEM" switch and some resistors, but I've never seen that either -- and you have to know that a good number of people that buy shirt-pocket size amps are going to want to listen to IEMs with them.
Bottom line is that I've done this many times and each time that the hiss was eliminated, so was the enjoyment of the music and always with the FR severely compromised -- sometimes to the point that you'd never recognize the sound as being that of the untouched IEMs.
When the listening is that up close and personal, and background noise is 20-40 dB below ambient, there isn't much room for noise and non-linearities.
-- Jim