Art Dudley on blind 'tests'

[Julf]

Darren and Julf, what kit have you ended up with after your extensive testing, surely the proof of the puddiing is in the eating ?

Confusing personal preferences with more general "betterness" is an all-too-common mistake.

What kit I have ended up with is dictated by my own personal tastes and preferences. The only conclusion you can draw from the gear I like is that that is the kind of gear I like.

 

[oldius]

I would be wary of those with vested interests disputing the validity of blind testing.

I remember a foot tapping imbecile of a dealer trying to convince me that, and I quote, I must be deaf if I could not hear a huge jump in quality between what he was selling and the equipment that I already had. He almost did the 'electric boogaloo' as soon as his own CD player was introduced but became a 'busking statue' playing the same track on my player.

I am the happiest I have ever been with equipment that I have owned and yet most of it was on the market between 1970 and 1990. In my opinion most audio has not improved since that time with the exception of digital products of course and I have heard huge amounts of gear during that time.

 

[oldius]

I am also in complete agreement with Ian. Why spend thousands of pounds on DAC upgrades if there is little benefit in absolute quality when the test is both level matched and blind.

I had the same result at a DAC bakeoff of my own. In my own room and with my own system, differences were very small once level matching was introduced and the "eye candy" was removed. In my younger days I would often seek to 'prefer' the sound of the gear that I liked the look of most. I'm sure most of us have been guilty of that.

 

[Steven Toy]

This is how to sustain the cyclic argument. Keep to the script and ignore anything that deviates from what you perceive or want the opposing argument to be.

For good measure keep repeating questions and then ignoring the answers only to repeat the same question again.

I'll ask Julf, Odius and others:

What will I be doing on the 10th January?

Answers on a postcard please.

A clue: the validity of blind testing is not being questioned by me. Unless the test has been rigged.

 

[neil]

The simplest form of blind test ("no peaking") only takes a willing helper, and some volume levelling.

What I think DBO I did was to show those present that perceived significant differences disappeared the moment the tests went blind. Nothing else changed (all equipment was plugged in when sighted and significant differences commented on), but suddenly everyone subjected to this blind scenario became somewhat perturbed that the differences had disappeared.

OK, I've since had a change of mind over whether differences exist, and accept that even this simple scenario has flaws, but nevertheless my feeling is that before I spend £'000s or even £'0000s in future I'd do this whenever possible.

Vital, not wishing to dis the DBO series at all, more power to your elbow for investing your time in doing this, but your statement is perhaps not quite accurate. You say nothing changed but if the sighted tests were at the start of the session and the blind tests occur afterwards then things have changed. The participants have had more time listening to the kit and to become familiar with its performance within what for most will be an unfamiliar listening environment (along with mostly unfamiliar kit). So whilst there is a correlation with the move to blind auditioning we have no idea whether that is the cause of th e different results. I still find the results you guys are getting interesting and hope you'll continue your efforts and to report them here.

 

[oldius]

Steven

You will be part of a DAC bakeoff. I hope that answers your question.

To my knowledge you also have or had your own forum that was based on subjectivist principles. I find it somewhat ironic that someone who vociferously promotes their own opinions, would highlight the 'scripting' of other poster's arguments when their own are apparently equally intransigent.

 

[Joe Hutch]

A clue: the validity of blind testing is not being questioned by me. Unless the test has been rigged.

So, if you can't tell one lager from another, or Coke from Pepsi in a blind tasting, how has that test been 'rigged'? This is the bit I don't understand.

 

[darrenyeats]

Vital, not wishing to dis the DBO series at all, more power to your elbow for investing your time in doing this, but your statement is perhaps not quite accurate. You say nothing changed but if the sighted tests were at the start of the session and the blind tests occur afterwards then things have changed. The participants have had more time listening to the kit and to become familiar with its performance within what for most will be an unfamiliar listening environment (along with mostly unfamiliar kit). So whilst there is a correlation with the move to blind auditioning we have no idea whether that is the cause of th e different results. I still find the results you guys are getting interesting and hope you'll continue your efforts and to report them here.

I recall after most (not all) heard big differences sighted, we proceeded to blind using randomised source, A or B for each trial. Unluckily the coin tosses meant it took - at least - seven trials till the source actually changed (I know - bad luck!)

Leaving aside whether ABX or AB is better, that's a different discussion - we'd already been through >=6 difficult blind trials before ever the source really switched! I would never describe that as "the differences disappeared the moment the tests went blind".

 

[zarniwoop]

This is how to sustain the cyclic argument. Keep to the script and ignore anything that deviates from what you perceive or want the opposing argument to be.

Your mission statement?

 

[darrenyeats]

Darren and Julf, what kit have you ended up with after your extensive testing, surely the proof of the puddiing is in the eating ?

I switched transports from an older expensive one to the Touch based partly on a blind test. I've switched from using Spotify Premium back to CD/lossless based partly on a blind test. They're too impractical for a lot of other things.

I don't think any of my blind tests have - or should have - any meaning to others. Without it being a properly scientific test (which it never is in real life) it only has personal value IMO. I still rely on sighted listening too though - I think null blind results are suggestive but not reliable and positive sighted results are suggestive but not reliable. This leaves a practical gap but hey, life is short.

I've spent more time blind testing than arguing about blind testing on forums, and I encourage others to do the same, they are very illuminating.

 

[neil]

I recall after most (not all) heard big differences sighted, we proceeded to blind using randomised source, A or B for each trial. Unluckily the coin tosses meant it took - at least - seven trials till the source actually changed (I know - bad luck!)

Leaving aside whether ABX or AB is better, that's a different discussion - we'd already been through >=6 difficult blind trials before ever the source really switched! I would never describe that as "the differences disappeared the moment the tests went blind".

Darren, I'm pretty certain that if you blindfolded me and put me in an unfamiliar room with unfamiliar kit you could play me excerpts of music on exactly the same pieces of kit for quite some time and I'd say I was hearing differences. In sighted auditions of kit at dealers I've often initially thought I was hearing differences which once I'd become familiar with the environment and gear diminished significantly or vanished. I have however never experienced this strange phenomenon of dancing staff which so many on here seem to have had inflicted upon them. They've mostly disappeared to make tea, deal with customers, catch up on paperwork or occasionally sat quietly somewhere off to the side or behind the listening position. I do also have a tendency to close my eyes whilst auditioning gear so maybe a few were frantically wasting energy. This is why I prefer to get kit on home loan for a week or so. Alternatively for secondhand gear I'll buy based on initial audition and then do extended home auditioning before making a decision. If I find I don't like it I'll sell it on.

 

[Steven Toy]

Your mission statement?

What am I doing on 10th Jan?

 

[zarniwoop]

What am I doing on 10th Jan?

Time will tell.

 

[Whaleblue]

Darren and Neil,

You are quite right I have oversimplified and summarised. It's undoubtedly more involved and complex a discussion than that previous post of mine suggests.

However, my aim was to point out how a simple blind test can shed a lot of light on the "night and day" findings from sighted listening.

I need to stress I DO believe differences exist, and I'm hoping we can get Mark to lend us the Mirus for January 10th as it did make me sit up. It's just my psyche makes me want to take expectation bias (yes even sceptical me has some!) out of the picture for at least some part of the assessment.

 

[Ciunas Audio]

Darren and Neil,

You are quite right I have oversimplified and summarised. It's undoubtedly more involved and complex a discussion than that previous post of mine suggests.

However, my aim was to point out how a simple blind test can shed a lot of light on the "night and day" findings from sighted listening.

I need to stress I DO believe differences exist, and I'm hoping we can get Mark to lend us the Mirus for January 10th as it did make me sit up. It's just my psyche makes me want to take expectation bias (yes even sceptical me has some!) out of the picture for at least some part of the assessment.

Ian, why didn't you just do your own self-blinding at Steven's & evaluate whether you were falling foul to expectation bias? I would, if I had this fear. I don't understand your equivocating what you heard & setting up another session but a blind one. You may not mean this but it is interpreted as if it's the real listening session? It almost seems to me that there is a fear of saying what is genuinely heard for fear of ridicule & so equivocating is done? I'm not trying to get at you but just to point up some psychological factors that I believe are prevalent among us & can cause us to second guess what we are hearing.

It has been stated here by Darren - we use sighted listening (knowing it's possibility for false positives) & report what we heard in the full knowledge by us & anybody reading our reports that they could be flawed. That's why the better listening reports specify the music track & what we (think) we heard on a track - so that others can evaluate this for themselves. Some might not hear the same & others might hear some or all of what is reported - different systems/ different rooms/ different ears.

One of the biggest issues with blind sessions (& where the greatest damage comes from) is that they are almost never recognised as being prone to false negatives. You are one of the few who have recognised & admitted this based on your own personal experience. But I'm not sure you fully believe this?

As many have said before - if blind listening sessions used some controls which showed that the whole setup/people/system were not prone to false negatives then they would gain more validity, for me, anyway.

 

[maxflinn]

 

[Ciunas Audio]

I'm of the opinion that the real difference between blind & sighted listening is not the taking away of knowledge - it's actually to do with the fact that sighted listening is usually done many times on many different days, with different music, etc. - in other words changing many variables over time which gives us a triangulation on the sound of the device.

Blind listening is almost always done once off as a "test" & the result taken as "evidence" of something - as said the notion of false negatives is seldom, if ever admitted. Please don't say that there's nothing stopping anyone from doing blind tests over an extended period - we are talking about the differences between blind tests as we all know them & sighted tests

If people are honest about this they will realise that they are not slaves to expectation bias as a result of sightedness - they are more likely to be slaves to expectation bias as a result of nocebo (it's much more hidden & difficult to recognise) or much more prone to second guessing themselves (because of self-doubt, fear of being made to look a fool, inability to describe the difference, etc.) - much safer & easier to believe they hear no difference.

I have many instances where I thought I heard something different between two playbacks but when I immediately repeated it realised it was a false positive - all done sighted.

Edit: Oh no, the Israeli/palestinian conflict must be over - here comes trouble!

 

[Whaleblue]

John, not sure where I've been guilty of equivocation? Anyway, I'll not take offence as I'm sure none was intended.

DBO III was run entirely sighted at Steven's request, and I wasn't going to go against his wishes. The Miras was introduced quite late in proceedings and I was already thinking about the 2 1/2 hour drive back by the time we were listening to it.

I've had no problem admitting I'm wrong in the past (on this, or any other subject!), and I've openly moved from a certainty that all DACs sound the same to a stance where I've heard DAC related differences both sighted and blind. I continue to stress that I am now interested in whether the differences matter. The Mirac appeared to show the biggest difference I've heard, so seems to make the best subject for the blind tests that I personally believe are a very significant part of any assessment of this.

 

[neil]

Ian, why didn't you just do your own self-blinding at Steven's & evaluate whether you were falling foul to expectation bias? I would, if I had this fear. I don't understand your equivocating what you heard & setting up another session but a blind one. You may not mean this but it is interpreted as if it's the real listening session? It almost seems to me that there is a fear of saying what is genuinely heard for fear of ridicule & so equivocating is done? I'm not trying to get at you but just to point up some psychological factors that I believe are prevalent among us & can cause us to second guess what we are hearing.

It has been stated here by Darren - we use sighted listening (knowing it's possibility for false positives) & report what we heard in the full knowledge by us & anybody reading our reports that they could be flawed. That's why the better listening reports specify the music track & what we (think) we heard on a track - so that others can evaluate this for themselves. Some might not hear the same & others might hear some or all of what is reported - different systems/ different rooms/ different ears.

One of the biggest issues with blind sessions (& where the greatest damage comes from) is that they are almost never recognised as being prone to false negatives. You are one of the few who have recognised & admitted this based on your own personal experience. But I'm not sure you fully believe this?

As many have said before - if blind listening sessions used some controls which showed that the whole setup/people/system were not prone to false negatives then they would gain more validity, for me, anyway.

JK as I understand it the session at Steven's was designed to be sighted from the outset. As such it may not have been possible to switch just like that.

 

[Ciunas Audio]

John, not sure where I've been guilty of equivocation? Anyway, I'll not take offence as I'm sure none was intended.

No offence intended & thanks for not pushing the all-out-war button. I just gathered from your posts that you were holding back your thoughts of the difference you heard with the Mirus, admitting that there were some small differences but waiting until you did your own DBO IV (blind) before really telling us what you thought?

DBO III was run entirely sighted at Steven's request, and I wasn't going to go against his wishes. The Miras was introduced quite late in proceedings and I was already thinking about the 2 1/2 hour drive back by the time we were listening to it.

Sure, I understand that circumstances may not have been conducive to doing a quick self-blind test

I've had no problem admitting I'm wrong in the past (on this, or any other subject!), and I've openly moved from a certainty that all DACs sound the same to a stance where I've heard DAC related differences both sighted and blind.

Sure, I know & acknowledge your openness to change. It's less about being right or wrong & more about being true to ourselves & what we genuinely hear - knowing full well that we might be biased but trusting that over time this will be ironed out.

I continue to stress that I am now interested in whether the differences matter. The Mirac appeared to show the biggest difference I've heard, so seems to make the best subject for the blind tests that I personally believe are a very significant part of any assessment of this.

No problem with any tests you do or consider important for you.