I'll offer a little tidbit of thought on this matter.
The maximum excursion on that particular driver is 4mm one way (noted for bass peaks, not sustained input), this would be 1 Watt of power at 55Hz. Loading may or may not buy you a little more power handling. At 75Hz, you can take more like 4W.
I'd still try a bigger push-pull amp, just for a point of comparison.
-PB