It's always a tradeoff. Most observers have found that a high impedance load improves the sound of most sources, including those that "can drive a low impedance load without breaking a sweat." But a high impedance pot is also more susceptible to picking up electrostatic fields, including hum and radio frequency interference. There are a few who claim that a low impedance level control, i.e. 600 ohms, sounds better than the same design implemented at higher impedance (say, above 10K ohms) but I have not seen anything that can claim to be a true test of this one. All these effects are quite subtle - again, within the bounds of what I have experienced myself.
