The ground connection from the switch boards would normally connect the sleeve of the TRS headphone socket, which in turn is grounded to the center post of the terminal strip closest to it. Have a read through the manual and you will find it.
PJ did state before that grounding the secondary side isn't strictly required for the circuit to function in a S.E.X. 2.1, but its just a good practice / safety thing to do so.