As PJ mentioned removing the cathode bypass cap will lower gain. If you mean it is quieter at the same volume, once you crank it up to make up the difference, then that is odd. It should be much the same. You might have jostled the offending component when you removed the bypass caps.
Have you narrowed the noise to the Quickie, that is eliminated the interconnects and all other components?
The Quickie is battery powered. So try swapping batteries. Probably not the problem but it is quick. If that changes nothing put the old batteries back in and go from there.
Short the inputs of the Quickie with jumpers, remove the input interconnects and see if it thumps. If not, it is the interconnects or the source. If it persists then short the inputs of your amp and see what it does. Note: this does not absolve the interconnects between the Quickie and the amp. If it still thumps it is the amp. If it doesn't then you have to back up and try shorting the RCA plugs that were in the Quickie's outputs, plug the interconnects back into your amp and remove the amp shorting jumpers. See if the interconnects are the cause.
After all this you can say for sure that it is the Quickie if it is so indicated.