I usually agree with your opinions, Jez, but this is one of those times I have to take issue with your comment about all competently-designed headphone amps sounding the same! As just one recent example, I auditioned an Astell&Kern SR25 and there was a positively HUGE difference in the way my HD600 sounded on its single-ended and differential output. AFAIK the HD600 isn't that much more difficult to drive than your HD580 (300Ω and 104dB/1Vrms), however the soundstage was narrow and congested on the former yet stretched out and spacious on the latter. Even my totally-not-interested-in-anything-relating-to-hifi Mum could hear the difference!
This is consistent with what I generally observe with single-ended and differential headphone amps. All of my headphones, even the super-sensitive in-ear-monitors that require less than 1/10th of a volt to reach over 100dB SPL, sound freer, more open/spacious and effortless when driven differentially. The differential output provides twice the output voltage, but I don't think this is what's responsible for the difference as the single-ended outputs of said amps still had more than enough voltage to drive my headphones plenty loud. Unless there's some other magic involved, it must be at least part due to the elimination of the common ground between the left and right channels.
PS - I do agree that there are headphone amps out there that add character/colouration to the sound with non-linear frequency responses and added harmonic distortions, just as there are loudspeaker amps. Nothing wrong with folks choosing one of these options if they prefer its 'synergy' with their chosen headphone, though it probably wouldn't be my default approach.