You are not alone and there are several reasons for why this happens.
For one us audiophiles are middle aged and older and grew up listening to a particular presentation which has a reasonable level of distortion(s).
Also, depending on the types of music, some recordings are either overly "dry" (they lack reverb and ambience) or bright or overly compressed, etc. and a more "tailored" and forginving system will make them less harsh or more compeling to listen to.
And finally there's personal preference, something which ASR seems to be acknowledging, even though it has been discussed by scholars(psychoacoustics) for quite some time now...