Maybe this can help... I finished my build the weekend before this past one. I built the Crack stock with the excpetion of using Mills resistors in place of the stock cement/wirewounds and used an Alps Blue Velvet volume pot. The amp is, for all intents and purposes, silent. I was and still am incredibly impressed with the lack of any preceptible background noise. I havent shorted the inputs to check whether the background is actually "black", but in terms of normal listening, there is nothing there. For all intents and purposes, it's silent (Speedball must be really impressive).
My point is that the amp, in stock form, is capable of a near silent background during normal listening (including quiet passages and pauses in the music). I use the word near as a qualifier only because an auidophiles description of silent may be different from mine, but you get the point I think. So follow the advice of the more knowledgable folks who have offered suggestions.
For the record, my main phones are Beyer 990's (250 ohm).