It's unlikely that you're gong to cause damage unless you are listening to something like the HD650 and you have the other headphones still plugged in, as this will produce a very loud listening level out of the HD650.
The balanced/unbalanced switch moves grounds around but otherwise leaves the circuit alone.