You can go higher if you want to.
I used some 800 Ohm philips 80's headphones with mine once, worked out fine.
Would not advise on using lower impedance headphones.
You could go to 120-100 ohm with some output tubes and a larger last powersupply + output cap setup.