Anyone tried blind testing DACs?

[Thetiminator]

This has to be the dorkiest thing I’ve ever read on the tinternet....truly world class entertainment Images of grown men sitting listening to music (very sternly I might add) with blindfolds on should be prescribed by the NHS. Just when you feel at your lowest....here you go, take a look at these dipsticks...think you have problems now?

And is it just hifi? Or do we get the ole blindfolds out when the wine waiter appears? Oo. Test driving a new car could get very interesting...

Brilliant.

 

[Blackmetalboon]

This has to be the dorkiest thing I’ve ever read on the tinternet....truly world class entertainment Images of grown men sitting listening to music (very sternly I might add) with blindfolds on should be prescribed by the NHS. Just when you feel at your lowest....here you go, take a look at these dipsticks...think you have problems now?

And is it just hifi? Or do we get the ole blindfolds out when the wine waiter appears? Oo. Test driving a new car could get very interesting...

Brilliant.

I guess you wouldn’t need a blind fold to hear the hum and fan noise from your previous MF Titan Amps though. Funny how you excepted that at the time, but have problems with blind testing!

 

[JensenHealey]

Wine tastings...aren't they undertaken with the wine bottle labels well covered or removed? To remove confirmation or label bias.

Seems clear to me that a fair assessment of differing audio devices take place under similar conditions. But I do think the level matching business is a real problem to get 'fair' results.

 

[Thetiminator]

Wine tastings...aren't they undertaken with the wine bottle labels well covered or removed? To remove confirmation or label bias.

Seems clear to me that a fair assessment of differing audio devices take place under similar conditions. But I do think the level matching business is a real problem to get 'fair' results.

Oh God no, not wine tastings. I said the wine waiter. Wine tastings...they’re just hideous...strictly for those who know nothing about and can’t afford good wine.

 

[Thetiminator]

I guess you wouldn’t need a blind fold to hear the hum and fan noise from your previous MF Titan Amps though. Funny how you excepted that at the time, but have problems with blind testing!

Hmmmm...I’m not sure either of us know what you’re trying to say there...

 

[JensenHealey]

Oh God no, not wine tastings. I said the wine waiter. Wine tastings...they’re just hideous...strictly for those who know nothing about and can’t afford good wine.

I had in mind the professional and competitive wine tastings - those 'gold' medal winning wines have to be assessed somewhere. It would be 'interesting' (perhaps, perhaps not) to see a HiFi equivalent performed on various components. A bit trickier than covering up a wine bottle label however......

 

[h.g.]

A scientifically useful, reasonably sensitive test is possible. But that's a major undertaking and we punters have no chance.

There is nothing difficult about designing an audibility test for those that think in a rational and straightforward manner. All teenage school children are taught the scientific method and how to design experiments without experiencing any great difficulty. Difficulties arise however for adults that have adopted audiophile magical thinking and pick up whatever supports it while pushing away what doesn't regardless of how well individual facts and observations stand up to close examination or fit with related facts and observations. The point of my post was that if some of those that have adopted audiophile magical thinking were to attempt to follow the scientific method in designing an audibility experiment it may start to reveal to them that they have lost the plot. Simply telling such people they have lost the plot doesn't work as the almost infinite number of threads like this demonstrate. Perhaps quietly trying to work something out for themselves may be more effective?

 

[Del monaco]

Oh God no, not wine tastings. I said the wine waiter. Wine tastings...they’re just hideous...strictly for those who know nothing about and can’t afford good wine.

Like wine masters who taste regularly?

 

[Sue Pertwee-Tyr]

There is nothing difficult about designing an audibility test for those that think in a rational and straightforward manner. All teenage school children are taught the scientific method and how to design experiments without experiencing any great difficulty. Difficulties arise however for adults that have adopted audiophile magical thinking and pick up whatever supports it while pushing away what doesn't regardless of how well individual facts and observations stand up to close examination or fit with related facts and observations. The point of my post was that if some of those that have adopted audiophile magical thinking were to attempt to follow the scientific method in designing an audibility experiment it may start to reveal to them that they have lost the plot. Simply telling such people they have lost the plot doesn't work as the almost infinite number of threads like this demonstrate. Perhaps quietly trying to work something out for themselves may be more effective?

I rather suspect most contributors on here are of an age where the scientific method was not formally taught as teenagers. But whatever. I've thought a little about this over time, and I'm aware of the possible biases in sighted listening. But the advocates of blind testing seem almost wilfully blind to the possibility of bias or other confounding factors in a blind test, such that they decline to control for any such bias by first establishing that the test methodology reliably (to the desired standard of statistical probability) distinguishes between two known differences, of a similar degree to those you want to use the test for.

I am aware that when listening blind, my 'mode' of listening is different to when I am relaxed and enjoying my music, and also different to when I am listening in a sighted audition. I find myself in 'analytical' mode, which is somewhat different to how I listen normally. I would design a blind test to see if this was a confounding factor, by first using the proposed test methodology to distinguish between two known, different, sound sources, perhaps CD vs a decent bitrate MP3 or equivalent. If the test showed I could identify the difference reliably, then the test is likely to be sufficiently sensitive. I simply can't understand why those who demand blind tests, or advocate them as a gold standard, don't do this. I can only surmise it is because they have doubts that the differences would show themselves.

 

[Purité Audio]

You are in ‘analytical’ mode whether the comparison is sighted or not, as to whether you can detect differences unsighted start with something major such as switching off a sub.
Keith

 

[h.g.]

But the advocates of blind testing seem almost wilfully blind to the possibility of bias or other confounding factors in a blind test, such that they decline to control for any such bias by first establishing that the test methodology reliably (to the desired standard of statistical probability) distinguishes between two known differences, of a similar degree to those you want to use the test for.

This makes no sense. If you were to try to design a scientifically valid audibility experiment where you would have to fully define the quantity being measured you would see why. It can be remarkably difficult to get people that have lost the plot to see they have lost the plot. Like most audiophiles both subjective and objective you would seem to be hanging a lot of meaning on "blind test" that likely doesn't exist in a scientific sense.

I am aware that when listening blind, my 'mode' of listening is different to when I am relaxed and enjoying my music, and also different to when I am listening in a sighted audition. I find myself in 'analytical' mode, which is somewhat different to how I listen normally. I would design a blind test to see if this was a confounding factor, by first using the proposed test methodology to distinguish between two known, different, sound sources, perhaps CD vs a decent bitrate MP3 or equivalent. If the test showed I could identify the difference reliably, then the test is likely to be sufficiently sensitive. I simply can't understand why those who demand blind tests, or advocate them as a gold standard, don't do this. I can only surmise it is because they have doubts that the differences would show themselves.

Again this makes little sense because you are vaguely lumping together and associating all sorts of things and adopting a scientifically invalid view of sound perception. You will have to stop doing this if you wish to design a valid scientific experiment. Of course the scientific bit is optional but if you wish to persuade those that accept scientific knowledge instead of audiophile magical thinking you will have to play by their rules and not your own. Tough I know since I won't pretend to know how to play by audiophile rules. I think I can see who is successful at it but as to why...

 

[SteveH]

Wine tastings...aren't they undertaken with the wine bottle labels well covered or removed? To remove confirmation or label bias.

Yes they are, and I remember one at Christies a good few years ago when the "guru" I was with was unable to tell red from white when the glass was covered as well.
Go figure. Sighted bias is extraordinarly powerful. No different for hi-fi.

 

[camverton]

You are in ‘analytical’ mode whether the comparison is sighted or not, as to whether you can detect differences unsighted start with something major such as switching off a sub.
Keith

Quite so! Analytical listening to the sound is very different from relax and enjoy the music. They possibly use different processes within the mind, and if so make the comparative process of little benefit to a music lover! As an example, if I listen to the ‘sound’ of a “finest measuring” monitor speaker it is extremely impressive; move into enjoy the music mode and I want to give up after an hour or so. Given a speaker designed for enjoying music I’ll want to continue listening through the night and sometimes beyond dawn.

Little wonder, given your stock, you favour blind testing etc. Not that there is anything wrong with what you stock for anyone, such as a recording engineer, who needs to concentrate on the sound but for many music lovers better choices may be available.

 

[Sue Pertwee-Tyr]

You are in ‘analytical’ mode whether the comparison is sighted or not, as to whether you can detect differences unsighted start with something major such as switching off a sub.
Keith

Well thank you Keith for your insights into how I feel inside my own head.

 

[camverton]

Well thank you Keith for your insights into how I feel inside my own head.

As long as you don’t try to feel inside Keith’s head you should be OK

 

[Sue Pertwee-Tyr]

This makes no sense. If you were to try to design a scientifically valid audibility experiment where you would have to fully define the quantity being measured you would see why. It can be remarkably difficult to get people that have lost the plot to see they have lost the plot. Like most audiophiles both subjective and objective you would seem to be hanging a lot of meaning on "blind test" that likely doesn't exist in a scientific sense.

Again this makes little sense because you are vaguely lumping together and associating all sorts of things and adopting a scientifically invalid view of sound perception. You will have to stop doing this if you wish to design a valid scientific experiment. Of course the scientific bit is optional but if you wish to persuade those that accept scientific knowledge instead of audiophile magical thinking you will have to play by their rules and not your own. Tough I know since I won't pretend to know how to play by audiophile rules. I think I can see who is successful at it but as to why...

I'm sorry it makes no sense. It does to me, but we are fundamentally failing to communicate, because your post, and the preceding one from you (#126) genuinely don't make any sense to me either. My replies were an attempt to bridge the gap by explaining where I'm coming from. It appears to be beyond my abilities, either to understand your points, or to express mine. It does seem pointless continuing, therefore.

 

[gingermrkettle]

"Fully define the quantity" raises another unproven axiom commonly stated as fact by some on here, that the measurements commonly used are sufficient to completely define all aspects of audio performance over any time period.

 

[h.g.]

Yes they are, and I remember one at Christies a good few years ago when the "guru" I was with was unable to tell red from white when the glass was covered as well.
Go figure. Sighted bias is extraordinarly powerful. No different for hi-fi.

Bias is not the correct word when applied to perception because there is no correct sound to reference. Sound perception does not exist as a fixed quantity but varies with context. Sound perception is influenced (not biased) by the other senses like sight and what we have learnt. People hearing the same cable but receiving different relevant cues will perceive different sounds. These are not imaginary sounds but actual sounds because there is no way to perceive sound except by listening under particular conditions. Ditto wine tasting.

 

[Purité Audio]

Well thank you Keith for your insights into how I feel inside my own head.

Not just your head Sue, everyone goes into ‘critical’ mode when asked to compare something it is often why you hear something for the first time when comparing only to find that the same artefact is there on your usual system.
Unsighted comparison is just a useful way of deciding whether there is a difference between two components.
If more listeners used it then the snake-oil merchants would be stopped in their tracks.
Keith

 

[h.g.]

I'm sorry it makes no sense. It does to me, but we are fundamentally failing to communicate, because your post, and the preceding one from you (#126) genuinely don't make any sense to me either.

As I mentioned above I am perfectly well aware that conversation in threads like this has pretty much zero chance of getting science across to the audiophile faithful which is why I was suggesting you tried to design a scientific experiment yourself to help understand how it works. If you are interested it might get a bit further by hopefully removing the attack from outside the tribe response inevitably present in threads like this. Unlike some objective audiophile missionary types I feel no great need to convert you to the path of truth and light but I am curious about what supports the audiophile faith given the continual drip, drip, drip of reality pushing against it.