Tube rolling 12AU7 - no differences in frequency response

[wbag]

I have a Crack 1.1 with speedball. Sounds great. I wanted to try different tubes, so I got a few different 12AU7s - whatever stock tube it came with (most of the writing is gone, how can I identify?), vintage RCA clear top, vintage RCA black top, vintage GE long plate, and new production JJ Electronic. People have strong opinions on the sound of different tubes - black tops are warmer, clear tops are brighter, new production tubes sound worse in general than vintage tubes, etc.

I tried all five in the amp, thought maybe I heard a small difference, but so subtle that I wasn't sure. So I did some measurements. I connected a Sennheiser HD6XX to the amp, stuck a calibrated UMIK-1 microphone in one ear cup, put a pillow on top, used my PC with a DAC as the input and ran frequency sweeps from 20-20,000 Hz using REW.  I took a few sweeps with each tube, then swapped the tube without touching anything else. They did have slightly different gains, so I level matched each curve to about +_ 0.1 dB.

The results were...interesting. All five curves were exactly the same. An image of the graphs is attached. Even when zoomed in, you can see they are all within +- 0.1 dB. There is maybe an additional spread of +_ 0.1 dB around 15kHz but that's it. I don't know about you, but I cannot hear a difference of 0.2 dB. I am positive that my mic and REW are setup correctly. I can easily see differences of volume, and if EQ is applied.

I know there is more to sound than frequency response. But differences in tubes are usually described as "warm, bright, clear, rolled off, etc" words that refer to frequency response. So that leaves me wondering:

• Is there a flaw in my test method?
• Does the 6080 tube affect the sound, but the 12AU7 tube doesn't?
• Are the differences in tubes something other than frequency response - like different distortion at different frequencies?
• There are no audible differences in different tubes and this is all just placebo? (good news for those that don't want to pay the high prices of vintage tubes)

Thoughts?

[Paul Birkeland]

Is there a flaw in my test method?

The default answer to this question is pretty much always yes! 

Does the 6080 tube affect the sound, but the 12AU7 tube doesn't?

The 6080 being used as a cathode follower means it will do its best to replicate what appears on the control grid at the cathode.  The sonic influence of the 6080 is far less substantial than the 12AU7.

Are the differences in tubes something other than frequency response - like different distortion at different frequencies?

Yes, this ends up being the root of the flawed methodology.  While frequency response may be the same, distortion vs. frequency is likely to be a bit different, and distortion vs. output level even more so.  You'd also probably want to look at the levels of each harmonic to really better understand how that might be influencing what you hear.