Alan Sircom
I dü werds, me
Does blind testing only remove visual bias from the process of assessing the sound of a product or does it introduce a series of confounding variables into the process that might not be well understood and can easily skew the results?
I fall on the side of it just removing bias, but then I'm comfortable with the process and don't have any hangups about test stress etc.
Discuss...
Having, like Guy, been involved with a lot of these tests in the past (and having even conducted a few), I'd have to say... it depends!
I think the 'stress' element as an argument against blind testing is somewhat overblown, but the performance of most listeners is in direct proportion to their training, whether that be some kind of formal training or simply turning up for listening sessions on a regular basis. There are always a few golden-eared and tin-eared outliers, but in the main what makes a good blind test listener is consistency, in terms of describing what they hear, being able to consistently pick out the same product when retested (perhaps surprising for some, the best listeners could successfully spot retest electronics under blind conditions months after the first listen, even when every other product in the test is different) and just consistently being there month in, month out.
The more thorough the blind test, the more consistent - or more trained - your listeners need to be. I still maintain this is a 'training' issue rather than a 'stress' issue, although the two are linked - I imagine taking your driving test would be a hell of a lot more stressful if you'd had no driving lessons!
In some ways, the confounding variable of knowing what category of product is being tested throws out its own bias. The stronger the views held, the greater the bias. And, as evidenced by the reaction to any poor innocent sap daring to ask a cable question in most UK forums, strongly held views are all too common. In most blind test protocols, such strongly held positions are useless because they potentially cloud the judgement of the listener. If possible, a position of 'open-minded sceptic' is better.
If that's impossible, you need some extremely well trained and willing listeners to volunteer to sit down and listen to a few 'somethings' in a day without knowing what that 'something' is.
The problem there is few people are willing to spend hours or days listening to 'somethings' without wanting to know at least what category of product they are listening to up-front. That's when you get 'I feel like a lab rat' comments.