Sorting out evidence-based from faith-based

[AndrewM]

1. That rarely happens.....

Well, the usual boilerplate caveats are generally inserted to cover their arse, but the implication is fairly clear.

Despite Max's kind endorsement, I don't subscribe to his world view that there can be no difference between DACS (or even USB cables, although I'd be much harder to convince of that). But I do also believe that sighted bias is a factor and any testing that doesn't exclude it is suspect in my view.

 

[Whaleblue]

Ian, amps wouldn't be on my agenda anyway, and DACs?

At the risk of annoying the only person who seems to be still behaving friendly towards me - what has made you want to re-evaluate them?

Max, you should know by know I think you're a decent guy, but as I've said before you were too sure of the facts for my liking. Hope that makes sense.

Having experienced differences now under controlled conditions that I'd not heard sighted my mind is open to the distinct possibility that "weird stuff happens in one's brain". I'm therefore more inclined to accept the sighted preference for my Vibe that didn't show up in quick ABing!

 

[adamdea]

Anything other than open-mindedness amongst consumers with no vested interest strikes me as bizarre.

I guess I owe an apology to the many I've challenged on their sighted results, but I hope I was generally polite enough to have offended very few.

With that said, I still value the blind test approach, but will now just see it as one tool in the box.

I think your open mindedness is admirable, but I'm not really following why your ABX experience has led you to radically reassess your view of sighted tests.

I can see though why you might have concluded that nocebo effect is a possibility and that it might be a good idea to test rather than throw in the hand too early.

The other thing that puzzles me is that I was under the impression that you generally carried out level matched sighted tests. Have i missed something.

I can see though how you might feel that it would be a good idea to err on the side of confessing to errors you have not made in order to bring down the forum average.

 

[onlyconnect]

I can see though how you might feel that it would be a good idea to err on the side of confessing to errors you have not made in order to bring down the forum average.

Tim

 

[Whaleblue]

I'm certainly not going to argue that one phono stage cant sound better than another. But I'd be more convinced by the efficacy of this sort of test if you could show that you could reliably distinguish using a blind test even using much longer samples - a bit of bind to do I know, but if the difference is clear, then it should be possible to devise a blind test that confirms it somehow.

The problem I'd have with that is that my successful ABX of the SoundBlaster test was achieved using very short clips (appx 2 seconds) of the "music", and it allowed very quick switching. I can't see how that could be readily achieved with kit changes.

It's odd, having crossed the divide, how I can see why you're suggesting what you're suggesting whilst simultaneously not feeling the desire to try it!

 

[AndrewM]

Andrew if it concerns you so much why don't you go and do the blind test on those products and as Darrenyeats advised take the hit on all the hassle doing it 'properly' One obvious thing that needs to be constantly restated as it is ignored or forgotten the main champions of 'proper' DBT x whatever just don't impose that on themselves. You would except such believers to be doing this stuff all the time.

Well the main reason not, is that I don't need a phono as I don't own any vinyl.

More generally, I thought we were discussing the general applicability/reliability of this sort of testing. I wouldn't seek to impose any sort of testing on anyone else choosing equipment - that's their own business. But if anyone makes a claim and seeks to validate that claim by offering or referring to test results, then I think we are all entitled to look carefully at the methodology.

 

[maxflinn]

Max, you should know by know I think you're a decent guy, but as I've said before you were too sure of the facts for my liking. Hope that makes sense.

Having experienced differences now under controlled conditions that I'd not heard sighted my mind is open to the distinct possibility that "weird stuff happens in one's brain". I'm therefore more inclined to accept the sighted preference for my Vibe that didn't show up in quick ABing!

Ian, no worries. FWIW I have no interest in testing of DACs because unless they're rubbish, they'll sound identical as distortion will be below audibility.

However I hope you enjoy whatever new testing methods you try and that if you do end up believing you can hear differences between all sorts of things that you stay as friendly and reasonable as you've always been, and raise the level of discussion of the 'other' side

 

[Ciunas Audio]

Well, the usual boilerplate caveats are generally inserted to cover their arse, but the implication is fairly clear.

Despite Max's kind endorsement, I don't subscribe to his world view that there can be no difference between DACS (or even USB cables, although I'd be much harder to convince of that). But I do also believe that sighted bias is a factor and any testing that doesn't exclude it is suspect in my view.

Yes, Andrew but any blind testing that ignores all other biases/factors is similarly suspect.

 

[adamdea]

A bit too simplistic, Adam. As I said ABX shows results which remove the need for people to describe the differences they hear - they can listen in a more relaxed way without anxiety to what way they are going to describe any slight differences they hear. This anxiety, in itself, is often enough to divide attention away from actual listening.

Some people are able to avoid this anxiety & just relax in blind testing, not caring about trying to analyse what they are hearing but just experience how the music is making them feel - a more holistic style of listening.

That's why I criticise the usually organised forum blind tests - they ignore other biases & can insert this type of new bias into the mix - all of which favours a null result

So, mixing up ABX with the usual forum blind tests is a complete over-simplification.
How many ABX tests have been done on this forum?
How many blind tests have been done & called for on this forum?
They are NOT both the same thing

But who is mixing up ABX tests with the "usual forum blind tests"?

I'm not even sure what you mean by "the usual forum blind tests" but (as I have pointed out before) A/B blind preference tests (which is certainly what Archimago's tests were and IIRC Werner's and Robert's) produce a similar result across the population which is to see whether the preferences are consistent.
Of course they don't really test individual responses because the individual responses are not generally significant.

They equally require no need to explain the difference in words- one can just plump for an option.

It's true that a mere blind listening session without any test isn't producing very concrete evidence. But that's not because it is blind. It's because it isn't a test.

 

[Purité Audio]

Phono stages will sound different depending upon the implementation and accuracy of the equalisation,
Keith.

 

[Ciunas Audio]

Ian, no worries. FWIW I have no interest in testing of DACs because unless they're rubbish, they'll sound identical as distortion will be below audibility.

Just like the XFi spec shows a all distortions below audibility, yet one loopback pass jumps it into audibility. Mind you Winers next DAC he claims is better so I guess all DACs are not the same, eh?

 

[Whaleblue]

I think your open mindedness is admirable, but I'm not really following why your ABX experience has led you to radically reassess your view of sighted tests.

Well, it's only HiFi, so "radical" is within a small sphere of influence in a life. I expect there's an element of enthusiasm of "the converted" in my recent postings. Hallelujah!

I can see though why you might have concluded that nocebo effect is a possibility and that it might be a good idea to test rather than throw in the hand too early.

The other thing that puzzles me is that I was under the impression that you generally carried out level matched sighted tests. Have i missed something.

The nocebo effect certainly seemed to affect my sighted results. I conducted lots of sighted tests, and latterly realised the importance of carefully matched levels for ABing.

I have also used blind tests, but these had generally involved a helper.

In retrospect I wonder whether even with a willing helper (or sighted other testees) there is social pressure to not waste their time, and that might precipitate a premature negative outcome?

I can see though how you might feel that it would be a good idea to err on the side of confessing to errors you have not made in order to bring down the forum average.

I think that's a compliment, but might have to AB it.

 

[AndrewM]

The problem I'd have with that is that my successful ABX of the SoundBlaster test was achieved using very short clips (appx 2 seconds) of the "music", and it allowed very quick switching. I can't see how that could be readily achieved with kit changes.

It's odd, having crossed the divide, how I can see why you're suggesting what you're suggesting whilst simultaneously not feeling the desire to try it!

The sort of (extremely impracticable, I know) test I had in mind was something along the lines of getting a third party to set up the kit behind a screen say once a day, randomly A or B, for 10 days or so, and seeing if you reliably distinguish which day was A and which B. I can quite understand why that would be something you don't want to do!

Clearly you feel the Vibe makes a significant difference to the pleasure you get from your system and you'd be daft to sell it under those circumstances. Enjoy the music!

 

[adamdea]

The problem I'd have with that is that my successful ABX of the SoundBlaster test was achieved using very short clips (appx 2 seconds) of the "music", and it allowed very quick switching. I can't see how that could be readily achieved with kit changes.

It's odd, having crossed the divide, how I can see why you're suggesting what you're suggesting whilst simultaneously not feeling the desire to try it!

It can be done with some items (different transports into a dac, or different dacs into a pre). But the real problem is the shortness of audio memory. That's why quick changes are thought to be more rather than less revealing (conducive to accurate judgment not "revealing" as in conducive to thinking there is a difference.)

 

[maxflinn]

Just like the XFi spec shows a all distortions below audibility, yet one loopback pass jumps it into audibility. Mind you Winers next DAC he claims is better so I guess all DACs are not the same, eh?

Ethan tells me that the D/A function of the XFi is transparent. Can you prove otherwise?

Any other DAC that is also transparent will sound the same, including yours, if it's transparent. Can you prove otherwise?

Some may measure further below transparency than others and thus will not jump into audibility as fast in comparison when put in a loop for multiple passes.
Can you prove otherwise?

You're the DAC designer, give us something more than hand-waving. What can you tell me that will convince me that current scientific understanding is wrong?

Off you go now, good lad...

 

[adamdea]

Well, it's only HiFi, so "radical" is within a small sphere of influence in a life. I expect there's an element of enthusiasm of "the converted" in my recent postings. Hallelujah!



The nocebo effect certainly seemed to affect my sighted results.

Sighted tests may produce false negatives as well as false positives. In the infamous audio myths workshop JJ makes that point a few seconds in.

But without some form of test (ie something which can independently checked against the facts) how can one ever tell?

I conducted lots of sighted tests, and latterly realised the importance of carefully matched levels for ABing.

I have also used blind tests, but these had generally involved a helper.

In retrospect I wonder whether even with a willing helper (or sighted other testees) there is social pressure to not waste their time, and that might precipitate a premature negative outcome?

Possibly I suppose. You are one of the few people around these parts for whom I could imagine that being an issue. For others I imagine the prospect of wasting other people's time would be irresistible.

Anyway, I can see that you have found your ABX experience revelatory and that you are re-thinking your position (Good: "When the facts change, I change my mind. What do you do, sir?"). I look forward to hearing more as your thoughts evolve. As always, a pleasure.

 

[Whaleblue]

Ian, no worries. FWIW I have no interest in testing of DACs because unless they're rubbish, they'll sound identical as distortion will be below audibility.

But we've both experienced differences with the Original vs. 1st pass!

OK, it may have been a ropey old Soundblaster (was it ropey and old?), but it was an AB'able difference! What if those subtle differences add up to a better sound?

OK, speakers make more obvious differences, but approach the whole thing with an open mind and think for yourself beyond the mantra others may feed you from either side. To be clear (how many times do I have to type those words!) I'm not suggesting you're a stooge or what-have-you, but you do seem to believe what the "first guy" said to the exclusion of all the experiences and advice you may be receiving ever since...

 

[AndrewM]

Yes, Andrew but any blind testing that ignores all other biases/factors is similarly suspect.

True, but it doesn't validate any other method that doesn't excluded sighted bias.

 

[sbgk]

Ethan tells me that the D/A function of the XFi is transparent. Can you prove otherwise?

Any other DAC that is also transparent will sound the same, including yours, if it's transparent. Can you prove otherwise?

Some may measure further below transparency than others and thus will not jump into audibility as fast in comparison when put in a loop for multiple passes.
Can you prove otherwise?

You're the DAC designer, give us something more than hand-waving. What can you tell me that will convince me that current scientific understanding is wrong?

Off you go now, good lad...

given how many dac manufacturers there are, a position of they all sound the same seems rather eccentric. Who is more correct - the DAC manufactueres, their customers, reviewers and sellers or ?

 

[AndrewM]

.....I have no interest in testing of DACs because unless they're rubbish, they'll sound identical as distortion will be below audibility....

I know I'm probably beating my head against a brick wall, but......

DACS contain analogue stages to amplify the (no doubt bit perfect) converted digital output from the chip.

Analogue amplifiers can sound slightly different due to design decisions made by the designer. Analogue circuits can be affected by RFI, mains pollution etc depending on how well the designer has coped with all these factors.

Therefore it is possible that different DACS might sound different. QED.

I would agree that such differences are very, perhaps even vanishingly, small, but even a rigid objectivist can't claim that there is scientific justification for denying the possibility of any difference.