Most often when I have experienced this when doing DIY speaker projects, it is because the cone is hitting something that shouldn't be there (like a piece if wire or random scrap of something), or there is a major air leak around the woofer and what you are hearing is air leaking and not actually the woofer itself.
To debug this, start by using your phone as a signal generator and play a bass tone that really sets it off, even at low volumes, then use your ears to try to precisely locate where the sound is coming from. Once you've done that, try to determine if there is air leaking there.
If there isn't, then you need to remove the woofer from the cabinet and test it in free air to see if it makes the noise.
