0.1% off where and from what? (the deviation of the total trace from 100K, for instance, tells nothing about tracking) .
Padding the pot is a solid plan, - 20dB of padding would be a good place to start. (10.5K across the outer lugs, 100K in series with the input to the pot)
-PB