Is blind testing suitable for audio use?

[Alan Sircom]

Does blind testing only remove visual bias from the process of assessing the sound of a product or does it introduce a series of confounding variables into the process that might not be well understood and can easily skew the results?

I fall on the side of it just removing bias, but then I'm comfortable with the process and don't have any hangups about test stress etc.

Discuss...

Having, like Guy, been involved with a lot of these tests in the past (and having even conducted a few), I'd have to say... it depends!

I think the 'stress' element as an argument against blind testing is somewhat overblown, but the performance of most listeners is in direct proportion to their training, whether that be some kind of formal training or simply turning up for listening sessions on a regular basis. There are always a few golden-eared and tin-eared outliers, but in the main what makes a good blind test listener is consistency, in terms of describing what they hear, being able to consistently pick out the same product when retested (perhaps surprising for some, the best listeners could successfully spot retest electronics under blind conditions months after the first listen, even when every other product in the test is different) and just consistently being there month in, month out.

The more thorough the blind test, the more consistent - or more trained - your listeners need to be. I still maintain this is a 'training' issue rather than a 'stress' issue, although the two are linked - I imagine taking your driving test would be a hell of a lot more stressful if you'd had no driving lessons!

In some ways, the confounding variable of knowing what category of product is being tested throws out its own bias. The stronger the views held, the greater the bias. And, as evidenced by the reaction to any poor innocent sap daring to ask a cable question in most UK forums, strongly held views are all too common. In most blind test protocols, such strongly held positions are useless because they potentially cloud the judgement of the listener. If possible, a position of 'open-minded sceptic' is better.

If that's impossible, you need some extremely well trained and willing listeners to volunteer to sit down and listen to a few 'somethings' in a day without knowing what that 'something' is.

The problem there is few people are willing to spend hours or days listening to 'somethings' without wanting to know at least what category of product they are listening to up-front. That's when you get 'I feel like a lab rat' comments.

 

[darrylfunk]

Agree with Alan B in post #6.

Q. Can anybody asking Steven why he would like a sighted session before a blind test answer the question of why not? Nobody has answered him and it's a fair question.

we have been asking him why he holds these views for a couple of years and he hasn't had the polite normal decency to answer the 2 simple questions....

and then he calls people names and generally throws his toy's out of the pram...

is our question and there are about 15 of us who have asked him, not a 'fair' question?

 

[Joe]

Hello Mr Brave Keyboard warrior! I bet you wouldn't say that to my face although no doubt you will insist that you would from behind your keyboard.

Blind tests are fine. They just need to be conducted thoroughly.

No warrior here... you come over as pompous and incredibly dismissive of anyone with a contrary opinion.

And you are rude.

( Blind tests just need to be conducted blind. )

 

[whatsnext]

Agree with Alan B in post #6.

Q. Can anybody asking Steven why he would like a sighted session before a blind test answer the question of why not? It's a fair question.

Regarding the OP. SQ hit it on the head in post #15.

For an individual looking for an absolute where "A" = "B", or "A" is not the same as "B", and where that individual is easily influenced by magazines (post #22 by Stevie A), gushing comment or by so-called "dodgy" dealers, blind tests are probably helpful to help prevent expensive mistakes. However, as S-Man showed in post #11 and I have many personal experiences of, it is entirely possible in a sighted test to reject a component despite the opposite result being assumed a formality due to expectation bias.

If the point of blind testing is to remove expectation bias then it's clear blind testing is not needed for everyone. Enough people are not taken in by foo. Like S-Man, and despite "accepted wisdom", I use my speakers without spikes, nor does my stuff sit on a dedicated equipment rack though I've had one for the convenience of storing a number of components. Now I have less stuff it's returned to the wooden cabinet and sounds the same as before.

So, to reply to the original question - no, of course not.

Back to painting the hoose....

It is a fair question to use to counter another question.

Why do you want a sighting? It is only a blind test if there is no sighting - too simple.

 

[onlyconnect]

Two main reasons for blind tests:

1. Remove expectation bias caused by appearance/knowledge of price etc.

2. Test whether two different sources are audibly distinguishable.

Both important; but the second is more so IMO.

Tim

 

[Markus S]

Q. Can anybody asking Steven why he would like a sighted session before a blind test answer the question of why not? It's a fair question.

I happen to agree with Steven here. I find it easier to home in on differences in sighted testing. Once I think I have a handle on sound differences, I'm happy to take a blind test to check if I have been fooling myself.

 

[onlyconnect]

I think the 'stress' element as an argument against blind testing is somewhat overblown, but the performance of most listeners is in direct proportion to their training, whether that be some kind of formal training or simply turning up for listening sessions on a regular basis.

We've had a few "what to listen for" discussions on here I think; would like more of these.

Tim

 

[Cav]

I want to eliminate as many variables as possible...

But you want pre-sighted listening?

For some in explicable reason you seem resistant to eliminating a false negative.

Why do you continue to say such things? Where have I ever expressed such a view?

 

[Steven Toy]

I happen to agree with Steven here. I find it easier to home in on differences in sighted testing. Once I think I have a handle on sound differences, I'm happy to take a blind test to check if I have been fooling myself.

That sums it up beautifully and I would be happy to conduct blind tests under these conditions any day.

Any more questions, rhetorical or otherwise, straw men, accusations of having an agenda, obtuseness, keyboard warrior bravery?

 

[Steven Toy]

But you want pre-sighted listening?



Why do you continue to say such things? Where have I ever expressed such a view?

See Markus's post. Beyond that you are arguing for the sake of it, just like darryl.

 

[Nic P]

A true story I like - France being a civilised country has a wine research institute. Using blind testing they proved that opening a bottle of red wine for an hour before drinking made no discernible difference to the taste. When published the results caused a furore with almost all French Masters of Wine saying whatever the French is for "bollocks". The wine institute considered the research and the reaction to it ... and announced that blind testing doesn't work

Nic P

 

[Steven Toy]

A true story I like - France being a civilised country has a wine research institute. Using blind testing they proved that opening a bottle of red wine for an hour before drinking made no discernible difference to the taste. When published the results caused a furore with almost all French Masters of Wine saying whatever the French is for "bollocks". The wine institute considered the research and the reaction to it ... and announced that blind testing doesn't work

Nic P

Foutaise.

 

[Brian]

we have been asking him why he holds these views for a couple of years and he hasn't had the polite normal decency to answer the 2 simple questions....

and then he calls people names and generally throws his toy's out of the pram...

is our question and there are about 15 of us who have asked him, not a 'fair' question?

I've no interest in the history between yourself and Steven. I don't care whether you think he's answered your question or not, nor do I care if your question is fair or not.

In response to being asked by a few members why Steven would like a sighted listening session before a blind test he asked why not? I think that's a good question and I'm interested in the response from those advocating blind tests.

So, do you intend answering the question? If you do, then answer it. If you don't just say so.

Here's a reminder of the question...
Q.Why not have a sighted listening before a blind test? What is wrong with that suggestion?

 

[Brian]

It is a fair question to use to counter another question.

Why do you want a sighting? It is only a blind test if there is no sighting - too simple.

In this case, yes it is fair. Very fair.

The fact I may have compared sighted a couple of DACs and come to a conclusion does not alter the fact that if I then listen to them "blind" I am listening to them blind.

 

[avole]

I like the old HiFi Choice way, which was blind testing first, followed by sighted, followed by lab.

 

[Steven Toy]

I've no interest in the history between yourself and Steven. I don't care whether you think he's answered your question or not, nor do I care if your question is fair or not.

In response to being asked by a few members why Steven would like a sighted listening session before a blind test he asked why not? I think that's a good question and I'm interested in the response from those advocating blind tests.

So, do you intend answering the question? If you do, then answer it. If you don't just say so.

Here's a reminder of the question...
Q.Why not have a sighted listening before a blind test? What is wrong with that suggestion?

What is wrong with the suggestion?

I have given my answer, Markus has given an answer and Alan Sircom has given an answer regarding training or familiarisation. Now it is time for this question to be answered.

 

[darrylfunk]

I've no interest in the history between yourself and Steven. I don't care whether you think he's answered your question or not, nor do I care if your question is fair or not.

In response to being asked by a few members why Steven would like a sighted listening session before a blind test he asked why not? I think that's a good question and I'm interested in the response from those advocating blind tests.

So, do you intend answering the question? If you do, then answer it. If you don't just say so.

Here's a reminder of the question...
Q.Why not have a sighted listening before a blind test? What is wrong with that suggestion?

why not indeed?

why hasn't he answered our questions though and where is his evidence that he claimed about the weakness of blind testing?

 

[avole]

What use would sighted listening to cables have?

 

[Steven Toy]

In this case, yes it is fair. Very fair.

The fact I may have compared sighted a couple of DACs and come to a conclusion does not alter the fact that if I then listen to them "blind" I am listening to them blind.

Indeed. It requires obtuseness that beggars belief not to see this.

 

[merlin]

I'd have no problem with sighted tests first.

My experience has been that the differences simply disappear once my eyes aren't helping (case of point being different file types) but I can't see any harm in letting people "familiarise" themselves with the items in the first place. Indeed it helps people accept the blind findings a lot more easily IME.