advertisement


Is blind testing suitable for audio use?

sq225917

sq225917

Bit of this, bit of that
Does blind testing only remove visual bias from the process of assessing the sound of a product or does it introduce a series of confounding variables into the process that might not be well understood and can easily skew the results?

I fall on the side of it just removing bias, but then I'm comfortable with the process and don't have any hangups about test stress etc.

Discuss...
 
sq225917 said:
Does blind testing only remove visual bias from the process of assessing the sound of a product or does it introduce a series of confounding variables into the process that might not be well understood and can easily skew the results?

I fall on the side of it just removing bias, but then I'm comfortable with the process and don't have any hangups about test stress etc.

Discuss...
Click to expand...

I think you've largely answered your own question. The only thing I can add is that all subjects should have ample opportunity to make hands-on sighted comparisons before the blind testing begins. The only people who will argue against this are those seeking a false null result. I see that you acknowledge the need to remove such confounding variables.
 
Steven, can you explain what effect the hands-on sighted comparisons have on the blind tests which follow, and why?
 
I used to be perfectly happy doing the HiFi Choice listening tests where perhaps 6-8 items were appraised, level matched, over the course of a day. Perhaps 6 x two minute excerpts of different kinds of music were used for each. One or two of the items would be re-introduced later to double check the findings of the listening panel. Quite often these group tests would spread over 3 days with some different listeners and perhaps 2 or 3 of the items listened to on the first day re-introduced on the second & third days.

This seemed a fairly relaxed yet thorough way of doing it & I always had some faith that the right outcomes had been reached certainly when Paul Miller had conducted the tests anyway.

For some reason DBT puts listeners in a different frame of mind, not sure why, and I'd be more wary of the conclusions drawn under those conditions. The HiFi Choice method ticked the blind & level matched boxes but is perhaps more involved to organise.
 
sq225917 said:
Does blind testing only remove visual bias from the process of assessing the sound of a product or does it introduce a series of confounding variables into the process that might not be well understood and can easily skew the results?

I fall on the side of it just removing bias, but then I'm comfortable with the process and don't have any hangups about test stress etc.

Discuss...
Click to expand...

You have covered both sides quite well Si. I haven't ever been involved in blind testing, so I guess there would be some stress there. I prefer to evaluate stuff comfortably, without time constraints and by myself. Perhaps in a test, I would feel pressure to perform, though I am not sure why that should be so.

I am sure blind testing is perfectly suitable for whoever wants to indulge, though I would disagree if the results were considered as some kind of definitive statement. It's just another opinion.

More relevantly; to me this is just a hobby so it doesn't matter much. :D I am happy trialling changes in my system my own way and trust what I hear, whether it's positive or negative.
 
@avole, cav,

On the Wam they often conduct blind tests and usually they follow a sighted/hands-on session.

The reasons for this should be obvious. The reasons for you questioning this are also pretty obvious; you seek a predetermined outcome.

It is also interesting to observe how people's findings may alter between the sighted and blind stages. The blind stage removes expectation bias; the preceding sighted test may confirm that it exists amongst at least some of the participants.

The other answer to your question is contained in SQ's opening post but to sum up in one word: thoroughness.

The question I ask of you is why not?
 
Steven, I don't find the reasons obvious at all, or I wouldn't have asked.
 
In general I agree with Alan B's statement.

However, I'n not quite sure what the rules for blind testing are...

1) The listener is told item A or item B is playing
2) The listener hears, for example, 10 X the same music and has to rate each one to see if it corresponds with when A or B was actually playing
3) Something else

My most recent audition (at home) went like this:
New item A is fitted and my current fav CD is played.
Turned it up loud and thought, wow this is great.
2 tracks in, turned it down and thought. Hmmm not sure.
Tried another disc and had an urge to listen item old item B.
Reistalled item B and had an enjoyable listening session.
Over the next few days kept eyeing item A and thinking I ought to give it another go.
Left item A powered up for 24hrs and tried it again. Not happy.
Tried to forget item A was installed and listened to music as normal for a day or two, except I couldn't find much mosic that I could settle on.
Reistalled item B, breathed a sigh of relief.
Listed item A for sale.

I can't image much of a difference if my listening had been by method #1 above over the same time period.
Item A is better looking and more expensive than item B, it also has a better specified SNR :)
 
I have already answered a question but I suspect I'm going to be badgered.

Why why why whywhywhywhywhywhy....... ?
 
I assume Steven feels, somewhat as I do that as the sound of an item isn't 100% of most people's reason for purchasing that there has to be some element of having experienced the device under test outside of those test conditions. I use blind testing to indicate if something does indeed sound different, that's all. I'll live with it for a while before I decide if I prefer the combination of sound and user experience.

If knowledge of the item you are listening effects how you think it sounds, then it does indeed change how it sounds, it may be a false positive but if the feeling stays then surely there's something to be said for acknowledging the effect exists. (For me i wish to rule that out as early as possible, but it has the chance to creep back in once I own the item, I don't keep them under lock and key in the dark).
 
darrylfunk said:
why? what's your reasoning?
Click to expand...

It is because people want to make themselves more at ease. Fear of making a truely unbiased verdict that contradicts all the opinions they have posted in forums before the test.

I accept audio persons are rarely ex SAS members but I am amazed that grown men are stressed by a blind test. Do they have cold sweats, wet themselves, uncontrollable shakes?

Basically they aren't as discerning as they like to think they are.

I would love to do a blind test of amps, cables IC and mains cables (without sighted hearing) I would learn something about myself (anf the kit)

What have they got to loose one asks
 
When changes may be subtle, subtle causes of stress (not enough to bring on somatic symptoms like sweats etc) need to be removed as much as possible. Are we testing the equipment or the subjects?

If you think a blind test is a foregone conclusion (a null result) you will want to test the subjects of course for the purposes of your own entertainment and will be asking these whywhywhywhy questions. Is a sighted/hands-on session before the blind session likely to spoil your fun?
 
That's where I don't follow you, Steven. You seem seriously concerned about null results, yet, again thanks to Wikipedia, there's nothing to be concerned about:

"In science, a null result is a result without the expected content: that is, the proposed result is absent.[1] It is an experimental outcome which does not show an otherwise expected effect. This does not imply a result of zero or nothing, simply a result that does not support the hypothesis. The term is a translation of the scientific Latin nullus resultarum, meaning "no consequence"."

If a null result means that an expected outcome that all cables sound the same is incorrect, where do you stand?

There's a big difference between foregone conclusion and expected result.
 
Steven Toy said:
When changes may be subtle, subtle causes of stress (not enough to bring on somatic symptoms like sweats etc) need to be removed as much as possible. Are we testing the equipment or the subjects?

If you think a blind test is a foregone conclusion (a null result) you will want to test the subjects of course for the purposes of your own entertainment and will be asking these whywhywhywhy questions. Is a sighted/hands-on session before the blind session likely to spoil your fun?
Click to expand...

Steve

You often say you have an open mind. That is a predetermined ramble.

I have never done a blind test and would welcome the experience without pre sighting.

I repeat why are grown men experiencing stress, what stresses them?
 
whatsnext said:
Fear of making a truely unbiased verdict that contradicts all the opinions they have posted in forums before the test.
Click to expand...

I've posted many opinions that seem to go against the herd. It doesn't really matter who is "right". Here's 2:
I prefer freebie interconnects to all the fancy/branded ones I've tried.
I prefer my speakers without spikes.

Neither of these opinions were reached with my eyes closed.
 
You must log in or register to reply here.


advertisement


Back
Top