Here are my thoughts about bypassing capacitors.
Many capacitors, especially large value capacitors, have some kind of poor performance at high frequencies. This is especially true of traditional electrolytic capacitors which can become resistive or even inductive as low as 10kHz. In addition to not handling the high treble correctly, this can seriously degrade a circuit's immunity to RF noise, for example. A high-quality capacitor with a low-loss dielectric that maintains performance to much higher frequencies can "take over" the capacitor function by maintaining a more capacitive, lower impedance at these higher frequencies - effectively shorting out the badly behaving cap.
The technically-identifiable issue is that the combination now includes an inductor (the large cap which becomes inductive at high frequencies) plus the bypass capacitor, and they can create a resonance in the impedance function which, in severe cases could easily do more harm than good for the sound. I believe this is the reason some people do not like the sound of bypasses, while others do like it. A simple solution is to put a small resistor in series with the large capacitor so that the resonance is well damped, though almost nobody actually does that.
My own inclination is to use capacitors which have acceptable high frequency performance, and avoid using bypasses at all. But then, I design stuff and don't have to deal with getting the most out of a capacitor I already have - I can go get the one I want. For electrolytics, there are now many on the market that are rated for use in switching power supplies and remain capacitive well above 100kHz. Mylar has some problems at higher frequencies but polypropylene is nearly as cheap and maintains performance much higher.
There are several more subtle ways that capacitors can degrade sonics; microphonics and dielectric absorption are two well-known examples. Bypassing to improve those problems is not so well understood and capacitors are rarely specified for those issues, making it nearly impossible to analyze the situation. Experimenting is the only practical answer, and it requires substantial reserves of patience (and often money!). Experimenting with bypass capacitors is less likely to make a big difference here, compared with swapping the whole capacitor. But it's also less expensive, so the two may be equally cost effective.
