It makes life harder for the sources that have to drive the pot, and so might slightly increase the distortion from them.
If the preceding circuit has a small output cap, you might get bass loss.
On the plus side, the noise floor, at settings near the mid point (-6dB) will be slightly better.
I can't remember what comes before the pot in a 32.5. Is it some sort of buffer circuit, or is it straight from the external source? If buffered, I think thah will drive 10K happily.