F1,
Glad you posted, I've been wanting to comment a post you made a while ago that's relevant here.
... don't underestimate the power of expectation bias.
This works in ever field, not just hifi.
I first experienced the power of it when involved for the first time in F1 racing. A driver, who was excellent and went on to be World Champion, wanted an adjustment to his car which wasn't available. One of the mechanics said never mind, I'll go to the truck and modify the part to suit. In fact the mod was impossible with the kit/time available and the mechanic knew this. He took the bit off, went to the truck for a few minutes then came back and re-fitted it. He said to the driver "You should be OK now". The driver went out and went 0.5 secs quicker and came back in to announce that that was exactly what was needed...
This is analogous to someone hearing a difference when expecting to hear expensive DAC A when it's DAC B. Placebo. I know this works because I've experienced similar, most recently at the last bake-off (where the first tranche of six trials were bizarrely all the same source, heads being flipped six times in a row by Vital!)
Changing the
mental state of the driver
changed the lap time, so I don't see an analogue between lap time and measured distortion of a playback system. Rather, a better analogue is perceived sound quality which, like lap time, depends on BOTH the physical system AND the mental state of the person.
The point here is there are two variables, the system and the person, causing the result (the lap time or perceived sound quality).
Your anecdote shows that changing the person
alone changes the result. We all agree.
However, we might change the system, for example, use body panels with a different surface texture, without telling the driver anything positive or negative, and the lap time could stay the same. From this, we might conclude that the change in surface texture of the panels makes no difference.
It's only when you take the panel into the lab and test it in isolation, that you find that is reduces drag. And the reason the lap time didn't change is that the reduction is drag was accompanied by a different air flow which increased drag around certain parts of the car. If you can alter the shape of the car to be compatible with the new air flow, you can realise the improvement from the changed surface texture. Then and only then, the lap time will improve.
However, if you used the new panels AND changed the shape of the car (realised improvement in the physical system) BUT you told the driver the panels have a problem, he might actually get an even worse lap time than before, even though the system is technically better.
If the "mental noise level" is greater than the degree of physical difference being tested, it can make it very difficult to draw hard conclusions from the result (lap time or perceived sound quality).
That doesn't mean we abandon all incremental improvements that are smaller than the mental noise level. Eventually, they feed through in a race situation (or real listening).
Also, unchanged measurable whole-system performance doesn't prove the part being tested couldn't realise an improvement in a slightly different physical system. If the item measures better in isolation, we might still realise better lap time or perceived sound quality if other conditions are met.
Finally, in F1 good enough is never good enough, and that's why objective improvements (maybe shaving a gram in weight) are made one upon another whether they're distinctly detectable in lap time or not. It's only at the end of the season that a small decrease in lap time comes out in the wash. Then it's hard to quantify how much some technical improvements contributed individually - but who cares.