Art Dudley on blind 'tests'

[Sue Pertwee-Tyr]

Hopefully, it's clear I exaggerate for effect and, possibly, amusement, but there is, I'm quite sure, a germ of truth in the caricature I painted a page back.

To answer Avole's question, I did very well in all my exams (I'm too old for GCSEs, having done 'O' levels which, as any fule kno, were much harder ) including my degree, which is a first class honours, thanks for asking. I'm not intellectually insecure, which is a product of my rather better than average education.

So I'm not afraid to admit that blind testing produces at least a suspicion of such thoughts in my mind, nor do I lightly shrug off the possibility that such thoughts will have an effect on my performance. My argument is that we all fall prey to such thoughts, albeit to varying degrees, and that this is but one possible factor in their propensity to produce null results. I think a lack of awareness of this possibility is at least as insidious an effect as any putative lack of awareness of 'expectation bias' in sighted tests.

 

[whatsnext]

So I'm not afraid to admit that blind testing produces at least a suspicion of such thoughts in my mind, nor do I lightly shrug off the possibility that such thoughts will have an effect on my performance. My argument is that we all fall prey to such thoughts, albeit to varying degrees, and that this is but one possible factor in their propensity to produce null results. I think a lack of awareness of this possibility is at least as insidious an effect as any putative lack of awareness of 'expectation bias' in sighted tests.

Assumptions again. I am quite happy to have a different view to the collected socially acceptable, agree with everyone for harmony, not upset anyone results I have observed so far. If I think something is so I say so, if I don't know I say so - it's how I earn my living.

I also respect those who can communicate in plain English.

 

[avole]

Hopefully, it's clear I exaggerate for effect and, possibly, amusement, but there is, I'm quite sure, a germ of truth in the caricature I painted a page back.

To answer Avole's question, I did very well in all my exams (I'm too old for GCSEs, having done 'O' levels which, as any fule kno, were much harder ) including my degree, which is a first class honours, thanks for asking. I'm not intellectually insecure, which is a product of my rather better than average education.

So I'm not afraid to admit that blind testing produces at least a suspicion of such thoughts in my mind, nor do I lightly shrug off the possibility that such thoughts will have an effect on my performance. My argument is that we all fall prey to such thoughts, albeit to varying degrees, and that this is but one possible factor in their propensity to produce null results. I think a lack of awareness of this possibility is at least as insidious an effect as any putative lack of awareness of 'expectation bias' in sighted tests.

What is your degree in? Genuine curiosity, SPT, nothing else.

 

[maxflinn]

Isn't this pursuit supposed to be fun?

From what I can gather, the next DBO will entail sighted listening first, impressions given and noted, and then a bit of listening without knowledge of what DAC is in use at the time, impressions given, and correlated with the sighted impressions.

Fun, and informative, I'd have thought.

 

[Sue Pertwee-Tyr]

It's a multidisciplinary degree, approximately half music, some technology (renweable energy) and some German (most of the vocab for which I've forgotten through lack of regular use).

Apologies for the failure of plain English. I can do it, but for some probably subconscious reason chose not to. Will try harder.

 

[avole]

That's a damn sight more interesting than mine. I did computing, religious history (don't ask) and the pick of the bunch, cooking.

 

[Sue Pertwee-Tyr]

'Interesting' is, dare I say it, a subjective impression. To my mind, what matters is whether you enjoyed it and benefited from the experience. I did, which matters more to me than the grade, if I'm honest.

 

[Whaleblue]

From what I can gather, the next DBO will entail sighted listening first, impressions given and noted, and then a bit of listening without knowledge of what DAC is in use at the time, impressions given, and correlated with the sighted impressions.

Fun, and informative, I'd have thought.

Precisely.

 

[Steven Toy]

Assumptions again. I am quite happy to have a different view to the collected socially acceptable, agree with everyone for harmony, not upset anyone results I have observed so far. If I think something is so I say so, if I don't know I say so - it's how I earn my living.

I also respect those who can communicate in plain English.

I think the English word you are looking for is admiration otherwise the ironyometer just went off the scale.

 

[Joe Hutch]

My degree is in English, quite simply the best subject there is.

 

[avole]

'Interesting' is, dare I say it, a subjective impression. To my mind, what matters is whether you enjoyed it and benefited from the experience. I did, which matters more to me than the grade, if I'm honest.

You're a dreamer, SPT. I did mine to get a job. Enjoyment was secondary.

 

[sbgk]

My degree is in English, quite simply the best subject there is.

now that's a subjective statement

 

[Ciunas Audio]

....

So I'm not afraid to admit that blind testing produces at least a suspicion of such thoughts in my mind, nor do I lightly shrug off the possibility that such thoughts will have an effect on my performance. My argument is that we all fall prey to such thoughts, albeit to varying degrees, and that this is but one possible factor in their propensity to produce null results. I think a lack of awareness of this possibility is at least as insidious an effect as any putative lack of awareness of 'expectation bias' in sighted tests.

I agree with this when it's done as a blind listening session & the propensity to call it a "blind test" further conditions the participants to the idea that they are under scrutiny - to stick with examinations as mentioned above - it's a pass/fail multiple choice test rather than a discursive essay style test.

This factor, of second-guessing one's first thought, was revealed & discussed in the thread where ABX positive results were reported - it's a naturally arising stress factor for most people, any time an individual feels they are being tested or can be definitively exposed as being wrong in their choice - it's an ego thing. (This doesn't arise when one does a personal blind test)

Furthermore, the argument often used is that a sighted test which shows differences followed by a blind test which no longer reveals differences "proves" that blind testing is "more valid", "more revealing", "the truth" is obviously logically flawed.

What we have are sighted test which have a bias towards false positives followed by a blind test which have a bias towards false negatives. What conclusions can be drawn about contradictory results from these two flawed tests?

That is all that's being said - both tests are flawed in their own way. Yes, some can point out obvious bias possibilities in sighted "tests" but fail to recognise other biases in "blind tests"

 

[MVV]

I never thought of the idea of doing a degree to get a job; thats almost as bad as working! Probably explains why I've been self employed for 25 years.

 

[avole]

My background is poverty.

 

[MVV]

My background is poverty.

I can identify with that because it is my future. Pensions? I spit on your pensions!

 

[Ciunas Audio]

Can I make a practical suggestion to Ian for his next DB IV - include a control which evaluates the propensity or otherwise of this blind test returning false negatives. The control I suggest is the Winer DA/AD test that he & a number of PFMers could hear a subtle difference between the generational copies.

This is probably at the same level as the type of subtle differences normally expected between DACs?

I've always said that blind tests need to prove themselves as capable of living up to the claims that are made for them by including such controls. Here's the chance to now prove this. We have test signals (muzak) which are proven (through ABX testing) to be differentiable between one pass & ?no of passes. People here heard these differences so they are perfect test signal for control usage

 

[neil]

There's an interesting quirk that means DB IV may not be blind for all participants. Many of the blind testing advocates point out that in sighted testing the participants are subject to expectation bias and that their brains will invent differences where none are their giving a false positive. Anyone taking part in this latest exercise who subscribes to the view that bits are bits and modern DACs that are competently implemented all sound the same already knows that DACs will be the variable in the exercise. In a sense for them the test is not really blind as the specifics of the DAC are irrelevant. As expectation bias works both ways it is entirely possible that they will hear no difference even when one is present because they expect to hear no difference.

 

[Ciunas Audio]

There's an interesting quirk that means DB IV may not be blind for all participants. Many of the blind testing advocates point out that in sighted testing the participants are subject to expectation bias and that their brains will invent differences where none are their giving a false positive. Anyone taking part in this latest exercise who subscribes to the view that bits are bits and modern DACs that are competently implemented all sound the same already knows that DACs will be the variable in the exercise. In a sense for them the test is not really blind as the specifics of the DAC are irrelevant. As expectation bias works both ways it is entirely possible that they will hear no difference even when one is present because they expect to hear no difference.

Yes, Neil, this is the nocebo bias that is among some of the biases that are ignored in these types of blind tests.

Apart from this even if someone doesn't come into the test with this nocebo expectation, the psychology of a group blind test is towards not sticking your neck out & hence a null test is the easiest way to do this.

That's why I suggest using this control to provide some validation of the results.

 

[Joe Hutch]

I suggest that each participant is heavily dosed with a truth serum pre-test to eliminate any form of expectation bias.