You have to consider that the plate current in your output stage will travel through the hum pot, and hum pots are generally not bypassed by the cathode bypass capacitor. If you use too high of a value, you'll have a bit of unbypassed cathode resistor, which can cause some abnormalities during operation.
On the other end of the spectrum, if you had a 10V filament and used a 20 Ohm hum pot, it would burn up half an amp of extra heater current when 100 Ohms would be a tenth of an amp.
