What would the other RCA jack be for? :^)
To change the impedance of the output on a Paramount, you have to re-wire the output transformer secondary. I'll attach a diagram for using a 2-pole, 3-throw switch to achieve 4-8-16 ohm switching; if you only need two choices you can use a DPDT switch.
The effect of using a 4 ohms speaker when the output transformer is wired for 8 ohms would as you say decrease the damping factor - looser bass and a shift in the spectral balance to favor frequencies where the impedance is higher. The Paramount is fairly conservative in transformer design, so you would probably see more output power, but at increased distortion, mostly second harmonic. The increased second harmonic can sound warmer and more immediate on small-group music, or just muddier on spectrally dense music such as big symphonies. If you are running on the edge of clipping, the increased headroom might give a more relaxed and dynamic presentation.
Not much help I suppose; it's a tradeoff that depends a lot on what music you listen to, how loud, and your specific speakers' performance.
