There's a decent record shop in Athens, GA downtown (I forget the name). New and used at good prices, good selection, and good condition.
Back in the early '90s I bought a ton of excellent quality vinyl for dirt cheap at Poo-Bah's in Pasadena, CA. Don't know if they're still as good as back then.
Occasionally you can get lucky with online vendors. I mainly buy new vinyl now, but have recently scored with a NM copy of Beefheart's Doc at the Radar Station from a seller at amazon.com for only $10.
NYC's used vinyl stores are almost all gone. Save for Downtown Music Gallery, but that's jazz only. So much has disappeared and gone online. And prices have sky-rocketed, too. It's too bad.