advertisement


Sorting out evidence-based from faith-based

Status
Not open for further replies.
Ciunas Audio

Ciunas Audio

Trade: Ciunas Audio
So here's some evidence I posted of a blind listening test but it was buried in another thread.
It might be useful to examine it & determine some issues related to blind testing & evidence?

But first an explanation - the test uses Foobar & its add-on ABX. Anybody who has used Foobar ABX will know what the figures mean but it's worth explaining a bit about ABX & what the results mean.

Essentially, ABX is a utility add on to Foobar that can allow you blind test whether you can hear the difference between two selected audio samples/files & identify them correctly & give you the statistical importance of your test trials i.e how close to guessing you are. It's a blind test - you don't know which file is which - they are randomly named X & Y for each trial & you have to say which original file X is? Is it the original file A or B

The numbers show each guess/trial & whether correct or not & the third number is the cumulative probability that you are guessing - a figure below 0.1% is very significant & means that it's almost certain (statistically significant) that you are not guessing in the test.

So for a 95% confidence level

Number of trials operated 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Minimum number correctly 9 9 10 10 11 12 12 13 13 14 15 15 16 16 17 18
Click to expand...
 
So here are the results: The test was Ethan Winer's test of whether anybody could audibly identify the insertion of an A/D -> D/A stage in the audio path from his audio myths video - download files [TL: link to copyrighted material deleted]. Just to explain Ethan has 5 Soundblaster files hosted on his site - which represent files that have audio passed a certain number of times through the A/D D/A of the Soundblaster card - original, 1, 5, 10, 20 passes. You can download the files from the above link & try the test yourself

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:34:21

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass5.wav

06:34:21 : Test started.
06:35:00 : 01/01 50.0%
06:35:10 : 01/02 75.0%
06:35:21 : 01/03 87.5%
06:35:46 : 02/04 68.8%
06:35:58 : 03/05 50.0%
06:36:19 : 03/06 65.6% <----- Difference found
06:36:28 : 04/07 50.0%
06:36:40 : 05/08 36.3%
06:36:51 : 06/09 25.4%
06:37:02 : 07/10 17.2%
06:37:11 : 08/11 11.3%
06:37:25 : 09/12 7.3%
06:37:36 : 10/13 4.6%
06:37:47 : 11/14 2.9%
06:37:58 : 12/15 1.8%
06:38:10 : 13/16 1.1%
06:38:24 : 14/17 0.6%
06:38:34 : 15/18 0.4%
06:38:50 : 16/19 0.2%
06:38:58 : 17/20 0.1%
06:39:12 : 18/21 0.1%
06:39:21 : 19/22 0.0%
06:39:38 : Test finished.

----------
Total: 19/22 (0.0%)

Above I am showing my search for critical section. So when I tested the single generational loss (i.e. "most difficult") I knew what to listen for:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav

06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.

----------
Total: 11/11 (0.0%)
Click to expand...
 
I would have pointed out that this is a lot of fuss over one third-hand-reported data point, but it is too warm and sunny for me to bother...
 
So those three responses mean what?
Avole, isn't bothered with real data, has no inquisitiveness with the results because they don't support his world view? Oh, just what he accused me of "afraid of blind tests" because they don't support my world view. Here's a real blind test, Avole with real statistically relevant data.
Julf uses some mangled excuse for ignoring it - datapoint, third party, not bothered, lot of fuss?
And Max, well ..........

Lot's of possibilities why this blind test was now passed with flying colours but one question arises - it has been on Winer's site for how long & nobody has shown any blind test results that show 1 pass A/D D/A through his Soundblaster card was audible. Why?
 
Ok, so maybe this is something that might be of more interest as there already was a 10 page thread about Archimago's 16/44 Vs 24/96 audibility test results.

These files were prepared by ArnyK (he of ABX fame) along the lines of what he claims is the correct way to prepare files of different sample rates to avoid issues.

The results here are from the same person as the results above so don't use the 50% of people get it right & 50% wrong. This is a statistically analysed test of a number of trials not a single shot guess.
32 Khz versus 96 Khz
=================================
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/09 06:10:07

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling band resolution limited 3216 2496.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling full band 2496.wav

06:10:07 : Test started.
06:10:38 : 01/01 50.0%
06:10:50 : 02/02 25.0%
06:11:07 : 03/03 12.5%
06:11:23 : 04/04 6.3%
06:11:36 : 05/05 3.1%
06:12:00 : 06/06 1.6%
06:12:14 : 07/07 0.8%
06:12:26 : 08/08 0.4%
06:12:38 : 09/09 0.2%
06:12:49 : 10/10 0.1%
06:13:00 : 11/11 0.0%
06:13:23 : 12/12 0.0%
06:13:42 : 13/13 0.0%
06:13:48 : Test finished.

----------
Total: 13/13 (0.0%)


44.1 versus 96 Khz
---------------------------------

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/09 06:32:02

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling band resolution limited 4416 2496.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling full band 2496.wav

06:32:02 : Test started.
06:33:07 : 01/01 50.0%
06:33:17 : 02/02 25.0%
06:33:24 : 03/03 12.5%
06:33:36 : 04/04 6.3%
06:33:47 : 05/05 3.1%
06:33:58 : 06/06 1.6%
06:34:12 : 07/07 0.8%
06:34:15 : Test finished.

----------
Total: 7/7 (0.8%)
Click to expand...
 
Let's sort out the evidence-based from the faith-based!
Let's see what factors might be responsible for this result & what bearing it might have on other blind tests.
 
John, perhaps there might be more interest in the thread if you gave a rundown, in layman's terms, of what exactly was being tested and what the results indicate.
 
Jk, Im struggling with numbers and layout of the data, perhaps titled columns might help. I'd assume most others similarly struggle and lack interest as a result. However I'm dead interested, keep it coming.
 
Col 1= time of trial
col 2 = result 1/1 is one correct out of one trial 1/2 = one correct out of 2 trials, etc
col 3 = probability that the test so far is based on guesswork 50% = expected result if guessing; 0% result if identified all samples correctly.

To run the test you select two files you want to see if you can identify differences between - these are named File A & B. The ABX utility presents you with a screen on which you have buttons for playback of A B & X Y. X & Y are randomised files A & B - you don't know if X is A or file B. It is randomised for each trial i.e each guess

You can select to play X or Y. Your task is to identify if X is A or B
So you register your choice by pressing your selection

It is a completely blind test & you can play file A or B at any time but the identities of X & Y are hidden throughout the test.

Here's a screenshot which might make it clearer:
KXrEWqA.png
 
maxflinn said:
John, perhaps there might be more interest in the thread if you gave a rundown, in layman's terms, of what exactly was being tested and what the results indicate.
Click to expand...
Sure, these tests are blind tests of the audibility of 2 issues considered inaudible by many.

The first results show that inserting an A/D converter followed by D/A converter in an audio chain is audible when using Winer's Soundblaster sound card as A/D & D/A. Winer is often cited by evidence-based audio people as the yardstick by which to bust audio myths (you probably have linked to him, yourself, Max?). Have a look at what he says about this at about 43:39 into this video https://www.youtube.com/watch?v=BYTlN6wjcvQ "Audio Myths Busted"

The second set of results show that 16/32 & 16/44 are audibly distinguishable from 24/96.

The importance of these results is that there's no level matching issues, no room issues (done on headphones), no group psychology (done solo).

But there are other factors to be considered.
 
Well I think it is interesting.

Got a link for the jangling keys?

Questions:

Have you tried the tests on others eg yourself or a friend and with what result?

Have you tried any other interesting samples eg music at 24/96 vs same down sampled to 16/44?

What about 24/192 vs 24/96?

How does the listener describe the differences?

What is the setup, is it bit-perfect to the DAC and what DAC?

Tim
 
Incidentally there was a guy on hydrogenaudio who gave similar evidence that he could distinguish between a particular sample 16/44 and 24/96. He said he found it difficult though; I listened to the same files and was not sure that I would have succeeded. It is always hard to know what you are testing, eg is it the down sampling or something inherent to the resolution? I actually prefer the Meyer/Moran methodology which it seems to me has less to go wrong.

But in principle it it helpful to have a person who can hear a difference, then you can experiment further (and with different kit) to analyse further and also to find out the limits beyond which no difference is heard.

Tim
 
jkeny said:
Sure, these tests are blind tests of the audibility of 2 issues considered inaudible by many.

The first results show that inserting an A/D converter followed by D/A converter in an audio chain is audible when using Winer's Soundblaster sound card as A/D & D/A. Winer is often cited by evidence-based audio people as the yardstick by which to bust audio myths (you probably have linked to him, yourself, Max?)
Click to expand...

Thanks. Yes, he posts on the AVI Forum too,

Have a look at what he says about this at about 43:39 into this video https://www.youtube.com/watch?v=BYTlN6wjcvQ "Audio Myths Busted"
Click to expand...

If I'm understanding this correctly, the point is that the sound is incremantely degraded as the signal is put through more and more conversion?

I think he mentioned on another vid something along the lines of one component might be audibly transparent but when a signal passes through several such components the end result may be not be audible transparency, is this the point or am I getting it wrong?

The second set of results show that 16/32 & 16/44 are audibly distinguishable from 24/96.

The importance of these results is that there's no level matching issues, no room issues (done on headphones), no group psychology (done solo).

But there are other factors to be considered.
Click to expand...

Given what I've read before and many such tests that I believe found the opposite, at least between 16/44 and 24/96, this surprises me.
 
maxflinn said:
Thanks. Yes, he posts on the AVI Forum too,
Click to expand...
Ok, so you know of him
If I'm understanding this correctly, the point is that the sound is incremantely degraded as the signal is put through more and more conversion?

I think he mentioned on another vid something along the lines of one component might be audibly transparent but when a signal passes through several such components the end result may be not be audibly transparent, is this the point or am I getting it wrong?
Click to expand...
Can you tell me how something can be transparent but degrade the sound, please? The results here prove that it is audibly "not transparent" after 1 pass.

Given what I've read before and many such tests that I believe found the opposite, this surprises me.
Click to expand...
Ah, so are you faith based, Max? I think we all know the answer, though - just wonder if you do?
 
jkeny said:
Ok, so you know of him
Click to expand...

Yes.

Can you tell me how something can be transparent but degrade the sound, please? The results here prove that it is audibly "not transparent" after 1 pass.
Click to expand...

John, I'm sitting here chilling out and thinking right, let's give John a chance to explain this and see what's what.

No need to be defensive or abrupt. I'd like you to discuss and explain this without an argument.


Ah, so are you faith based, Max? I think we all know the answer, though - just wonder if you do?
Click to expand...

See above.

Thanks.
 
Status
Not open for further replies.


advertisement


Back
Top