... Is this just a crazy idea???
Yes, of course it's crazy. What's your point? :^)
I did that a long time ago, before we started using parallel feed. PB is right, the switches arced like crazy and I threw them out after the experiments were done - they were scorched and pitted. Really, I should have switched the amp off and let the caps discharge before switching output transformers each time.
First caution - you need to give each transformer at least 50-100 hours of music, fairly loud, before you can make meaningful comparisons. I did it with the local classical music station and hefty 8-ohm load resistors in place of speakers, but still it takes weeks to get them all burned in. I did some listening, shared it with a crowd at a club meeting, and we may have done a bit more, later. We had a 2A3 amp with the TFA-204 (series feed), a DS-025, some popular-at-the-time Japanese high-end output transformer, and at least one other I can't remember - might have been an FS-030? Or a Hammond 125E? Anyhow, an interesting experiment. Then I took it all apart.
It was worth doing as an educational experiment - I learned quite a bit.