If the AC/DC mode switching is automatic it may be that the waveform is not being recognized as AC since it doesn't look like a sine wave. It looks more like a series of pulses. Try to shrink the vertical swing by lowering the signal or by changing the vertical scale to a bigger volts per division setting and see if the mode stabilizes when you have the full sine wave showing on the screen.
Haven't been here in a while so I missed this, Dan. Thanks for the suggestion. After much playing around, I kind of wrote the device off as faulty. I might dig it out and try it again sometime with a lower input signal, but in the meantime, I confirmed the usable frequencies of my multimeter and have been using that successfully (I think)
