advertisement


Audiophile Network Switches for Streaming ... really ?

It would be a poorly-designed double-blind test that did rely on aural memory, but you’re right, and as I posted when this thread was still in its infancy, a double-blind test capable of ending this never-ending argument is not going to happen.
I know this is an area in which you have particular expertise, so I'm curious as to the bolded bit in the above.

My understanding of the methodology, in general terms, is this:

  1. Play extract of music on Device A.
  2. Play same extract of music on Device B.
  3. Play same extract of music on either Device A or Device B without attendees being aware which is in use. Ask attendees to choose whether it is A or B which is playing.
  4. Repeat 3) until desired level of statistical significance is achievable.
It seems to me that aural memory is engaged in a) remembering the sound of A, of B, and of A vs B and what differences (if any) were detected; and b) recalling these when listening to the device during the repeated stages at line 3. Plus, there may be time taken to disconnect A and reconnect B, even if this is only scant seconds. We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

It is misgivings in the above that lead me to ask advocates of blind testing whether they have ever run a control test for their methodology (ie, to show whether it can reliably detect known audible differences, and how sensitive it can be - how gross do the differences have to be in order to be reliably detected). I've yet to hear that this has been done, still less to any statistically significant extent.
 
I know this is an area in which you have particular expertise, so I'm curious as to the bolded bit in the above.

My understanding of the methodology, in general terms, is this:

  1. Play extract of music on Device A.
  2. Play same extract of music on Device B.
  3. Play same extract of music on either Device A or Device B without attendees being aware which is in use. Ask attendees to choose whether it is A or B which is playing.
  4. Repeat 3) until desired level of statistical significance is achievable.
It seems to me that aural memory is engaged in a) remembering the sound of A, of B, and of A vs B and what differences (if any) were detected; and b) recalling these when listening to the device during the repeated stages at line 3. Plus, there may be time taken to disconnect A and reconnect B, even if this is only scant seconds. We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

It is misgivings in the above that lead me to ask advocates of blind testing whether they have ever run a control test for their methodology (ie, to show whether it can reliably detect known audible differences, and how sensitive it can be - how gross do the differences have to be in order to be reliably detected). I've yet to hear that this has been done, still less to any statistically significant extent.

I now realise I’m applying unpublished proprietary methodology from assessing MRI images here, but I think it is relevant. It’ll need a bit of thought and a longer message than I’m prepared to type on a tablet. I’ll be back!
 
I know this is an area in which you have particular expertise, so I'm curious as to the bolded bit in the above.

My understanding of the methodology, in general terms, is this:

  1. Play extract of music on Device A.
  2. Play same extract of music on Device B.
  3. Play same extract of music on either Device A or Device B without attendees being aware which is in use. Ask attendees to choose whether it is A or B which is playing.
  4. Repeat 3) until desired level of statistical significance is achievable.
It seems to me that aural memory is engaged in a) remembering the sound of A, of B, and of A vs B and what differences (if any) were detected; and b) recalling these when listening to the device during the repeated stages at line 3. Plus, there may be time taken to disconnect A and reconnect B, even if this is only scant seconds. We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

It is misgivings in the above that lead me to ask advocates of blind testing whether they have ever run a control test for their methodology (ie, to show whether it can reliably detect known audible differences, and how sensitive it can be - how gross do the differences have to be in order to be reliably detected). I've yet to hear that this has been done, still less to any statistically significant extent.
Hence the role of a null tester, real time, no memory required, assessment of the differences....
 
We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

Agree on the unreliability of audio memory in making comparisons (at least, where there is not a major difference). But this is an even stronger reason to distrust the reliability of claims for any improvements due to changing/inserting a network switch.
 
It seems to follow from this then that there is no reason to take hobbyist testimonials into serious consideration. Pity indeed!
“Hobbyist testimonials” eh, interesting terminology, having another dig in your very obvious crusade? Perhaps you could declare your professional credentials; apologies if you have already done so.

The point is that the evidence of Nigel’s (the flash) ears and others is all we have but it is evidence, albeit not of the sort you appear to respect although I do think it needs to be treated with caution particularly as we are all using different partnering equipment and have different hearing and minds that are sensitive to different audio qualities. Given that your need for a properly conducted blind test of sufficient proportions isn’t going to happen either makes your mention of it rather pointless.

In my view Nigel’s idea are of interest and may be worth trying by those with an interest but clearly not by those who haven’t which doesn’t seem to stop them contributing countless posts of negativity. I do wonder if some “contributors” (I use the word loosely) might be happier cheering on at ASR, or perhaps that’s where they’ve come from. Surely, unless some new technical point is being made, the same old absolute bollox types of posts only have to be made once. AFAIK Nigel isn’t trying to part anyone with large amounts of money so I fail to see why the need to crusade or make silly comments such as the one quoted.

Many thanks to @hc25036 and @John Phillips for bringing some interesting and worthwhile points to the discussion, for I’m not saying that anyone who doesn’t wholeheartedly agree with Nigel is on a crusade!
 
Agree on the unreliability of audio memory in making comparisons (at least, where there is not a major difference). But this is an even stronger reason to distrust the reliability of claims for any improvements due to changing/inserting a network switch.
I think it is a reason to be cautious rather than distrust and perhaps, if feasible, accept Nigel’s invitation to his place to hear his ideas for oneself.
 
Agree on the unreliability of audio memory in making comparisons (at least, where there is not a major difference). But this is an even stronger reason to distrust the reliability of claims for any improvements due to changing/inserting a network switch.
Perhaps, but you are here assuming that the claims for improvements come from aural memory-based assessments. IME, while there may be an immediate and perceptibly audible difference, it takes a little time to decide whether that is an improvement. That may be moments, it may be days, or weeks. Sometimes, the most reliable arbiter is whether you spend more, or less time listening to music; or whether your music tastes become wider, or more narrowly focussed. These are largely subliminal cues, and emerge over time.
 
Perhaps, but you are here assuming that the claims for improvements come from aural memory-based assessments. IME, while there may be an immediate and perceptibly audible difference, it takes a little time to decide whether that is an improvement. That may be moments, it may be days, or weeks. Sometimes, the most reliable arbiter is whether you spend more, or less time listening to music; or whether your music tastes become wider, or more narrowly focussed. These are largely subliminal cues, and emerge over time.
Indeed. But perceived differences - even if they emerge over time - may not necessarily be real. It is perhaps useful to realize that whatever problems various blind test arrangements may have, it does not make sighted tests any more reliable. They are inherently unreliable.
 
I know this is an area in which you have particular expertise, so I'm curious as to the bolded bit in the above.

My understanding of the methodology, in general terms, is this:

  1. Play extract of music on Device A.
  2. Play same extract of music on Device B.
  3. Play same extract of music on either Device A or Device B without attendees being aware which is in use. Ask attendees to choose whether it is A or B which is playing.
  4. Repeat 3) until desired level of statistical significance is achievable.
It seems to me that aural memory is engaged in a) remembering the sound of A, of B, and of A vs B and what differences (if any) were detected; and b) recalling these when listening to the device during the repeated stages at line 3. Plus, there may be time taken to disconnect A and reconnect B, even if this is only scant seconds. We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

It is misgivings in the above that lead me to ask advocates of blind testing whether they have ever run a control test for their methodology (ie, to show whether it can reliably detect known audible differences, and how sensitive it can be - how gross do the differences have to be in order to be reliably detected). I've yet to hear that this has been done, still less to any statistically significant extent.
If you don’t trust your aural memory when listening blind, why should we trust your aural memory when you listen sighted? When you compare two things sighted are there not still the same issues of remembering what the first thing sounded like, disconnecting and reconnecting etc.? So if aural memory is unreliable, there’s no value in comparing two things successively, whether or not you know which is which. In which case all of your claims to hear differences under sighted conditions are worthless.
 
I know this is an area in which you have particular expertise, so I'm curious as to the bolded bit in the above.

My understanding of the methodology, in general terms, is this:

  1. Play extract of music on Device A.
  2. Play same extract of music on Device B.
  3. Play same extract of music on either Device A or Device B without attendees being aware which is in use. Ask attendees to choose whether it is A or B which is playing.
  4. Repeat 3) until desired level of statistical significance is achievable.
It seems to me that aural memory is engaged in a) remembering the sound of A, of B, and of A vs B and what differences (if any) were detected; and b) recalling these when listening to the device during the repeated stages at line 3. Plus, there may be time taken to disconnect A and reconnect B, even if this is only scant seconds. We have been told, numerous times, that aural memory is only reliable for scant seconds. And if aural memory is unreliable, this is a mechanism by which confusion can set in between the original A and B, over time (eg during 4, above, depending on number of repetitions).

It is misgivings in the above that lead me to ask advocates of blind testing whether they have ever run a control test for their methodology (ie, to show whether it can reliably detect known audible differences, and how sensitive it can be - how gross do the differences have to be in order to be reliably detected). I've yet to hear that this has been done, still less to any statistically significant extent.

Some background - I worked in medical imaging, mainly testing to see if adding something in to MRI scanning increased and/or made more accurate the information that a Radiologist gains from the images. For this discussion we have to ignore the hardest question (how do you know that the lump on the image is really there) and just deal with getting a bunch of Radiologists to look at the images and give you unbiased evaluations of what they see. First thing is to remove all evidence of the sex of the patient (some lesions have are more common in one sex or the other), second to remove evidence of age (same reason), the hospital where the image was obtained (hospitals tend to see more of one disease type) and of the country (due to the Radiologists knowing the population bias for different diseases - eg more primary liver cancer in Japan, but more rarely liver metastases). Just taking out those cues reduced the accuracy of top-level radiologists from 90+% to 60+%. That's important because it means that in the study the only thing leading to a diagnosis is what's in the images rather than all those possible biases which will be at different levels across the Radiologists.

To try to get to the point, very early on we did simple studies, 3 or more Radiologists in a room (on separate workstations) looking at images and writing down number of lesions, location of lesions, size of lesions. etc etc, plus a diagnosis. Took a couple of weekends, give the data to statisticians, home and hosed. But not. First up we learned about dominant participant bias, where over time the less senior Radiologists started to fit in with the most senior guy, especially after breaks where they could talk together. All future studies were one Radiologist at a time.

The big surprise though was visual memory (at last getting to the audio comparison). Although visual memory is thought to be longer than audio memory, it is still fairly short (just ask a detective looking for witnesses). We learned that some/many experienced Radiologists can accurately recall images from a case they saw a couple of months (and hundreds of patients) ago and use that information to inform decisions about an image in front of them. Bad news for us as we now had to increase the period between seeing different images from the same patient from a couple of weeks to 3 months plus. The point with audio is that I'd bet 50p that 'golden ears' is actually at least partly a manifestation of extended audio memory.

How did we deal with that? The last, most complex and costly we designed specifically asked the Radiologist not to make a diagnosis (at least not until the very end) but to describe what they saw in the image, in detail. That was then compared to reality (this was a liver disease study where the liver was to be removed from the patient and examined in detail by a Pathologist which was the 'truth' that the Radiologist's assessment was compared to). The study took 18 months of weekends and worked at a level we couldn't have dreamed for.

So, if I was designing a study (no I won't!) to compare two things that are going to be very similar using people with a huge interest in what is being assessed and some of whom may have enhanced memory/discrimination skills, I would not get them to do something they do all the time where the biases could have maximum effect.

I'd suggest a test that doesn't rely on listening to the music in the normal way, which gives a greater chance of getting some discrimination:

1. Get agreement on the music to be played - duration probably doesn't matter

2. Get some agreed experts to listen to the music using one of the systems (no need to be blind) and to describe what they hear as being exemplary, most noticeable and whatever else (within reason). Could be the snare drum being just left of centre, can hear the type of drumstick (not my area of expertise), violin note decays realistically - anything.

3. Get your victims doing the test (but not together to avoid dominant participant bias), but get them just to score the items identified before, and obviously blind to what version of the system is in use. Repeat randomly with the systems changing and also with the same system being used twice/3 times in succession. These guys need to be different from the people in part 2.

Never going to happen and will take months to do if someone is daft enough to try, but I think this would remove/reduce the problem with the standard methodology described by Steve in post #1781. Clearly I had the advantage of 8-figure budgets and willing participants prepared to give up time to have their knowledge and ability tested in evil ways (I loved working with Radiologists because they were so keen to learn and improve in what is a difficult speciality).

Not sure if that helps or hinders, but I hope it answered the question!
 
Perhaps, but you are here assuming that the claims for improvements come from aural memory-based assessments. IME, while there may be an immediate and perceptibly audible difference, it takes a little time to decide whether that is an improvement. That may be moments, it may be days, or weeks. Sometimes, the most reliable arbiter is whether you spend more, or less time listening to music; or whether your music tastes become wider, or more narrowly focussed. These are largely subliminal cues, and emerge over time.

But surely any conclusions drawn on this are subject to the far greater variability of other factors that affect our enjoyment. I don't know about you, but on some days or occasions I am really drawn into the music, and marvel at the sound quality of the hi-fi, but on others days I'm much less moved. This could be due to all sorts of biological factors that might affect the physiology of our hearing, but also on some occasions listening to music seems much more appealing than on others. For some reason I enjoy music more at night-time than during the day. I'm sure some will try to tell me that this is because the mains-supply is quieter, but I think its simply down to (my) human preference for the best time to listen. Also it is possible to spend too much time listening I think - I often enjoy music more when I've had a few days break away from listening. I have also had the experience of thinking some change has made a major improvement - then reverting back some days later to find no real difference.

So I'm with John Phillips' comment above to the effect that to me it only makes sense to invest the time and effort to assess something if I think there is a real reason it could make any difference.
 
That's silly. How many non-audiophile switches are there between the download source (?) and your socket? A hundred? Possibly much more... I can't imagine the data loss.
And there might even be radio transmission into the chain. The horror! Data lost in the ether, oh dear.
Let's be serious for a minute.

Data loss would mean drop-outs or clicks. This has nothing to do with data (loss), it's noise affecting the clocking and D/A conversion.
 
Perhaps, but you are here assuming that the claims for improvements come from aural memory-based assessments. IME, while there may be an immediate and perceptibly audible difference, it takes a little time to decide whether that is an improvement. That may be moments, it may be days, or weeks. Sometimes, the most reliable arbiter is whether you spend more, or less time listening to music; or whether your music tastes become wider, or more narrowly focussed. These are largely subliminal cues, and emerge over time.
Good points and therein lies a problem in that we are not necessarily talking about massive and obvious differences to the sound but the smallest things can affect our enjoyment and comprehension of a piece of music.

When I listened to Wave cables in my system, another item that deals with “noise”, the difference was quite obvious in that delicate brushwork on cymbals on one track became much clearer as to what the percussionist was doing, whilst going back to my normal bnc interconnects and the clarity of the brushwork became less distinct, relatively speaking more of a general shsss. On the other hand I think, but can’t be entirely sure on comparison, that violins are improved. The only way to find out would be to return to the original cables, although as I don’t have a need to prove a point I doubt if I’ll bother!

Which brings us back to the latest sideshow of this thread and that is the motives of those who claim that Nigel’s tweaks are bollox or, much worse, that audiophiles are mental if they report their experience of improvements (thankfully the worst of those posts was removed). Reasoned arguments are fine but it would be interesting to know the motives of the multi post naysayers, assuming they even know themselves.
 
But surely any conclusions drawn on this are subject to the far greater variability of other factors that affect our enjoyment. I don't know about you, but on some days or occasions I am really drawn into the music, and marvel at the sound quality of the hi-fi, but on others days I'm much less moved. This could be due to all sorts of biological factors that might affect the physiology of our hearing, but also on some occasions listening to music seems much more appealing than on others. For some reason I enjoy music more at night-time than during the day. I'm sure some will try to tell me that this is because the mains-supply is quieter, but I think its simply down to (my) human preference for the best time to listen. Also it is possible to spend too much time listening I think - I often enjoy music more when I've had a few days break away from listening. I have also had the experience of thinking some change has made a major improvement - then reverting back some days later to find no real difference.

So I'm with John Phillips' comment above to the effect that to me it only makes sense to invest the time and effort to assess something if I think there is a real reason it could make any difference.
You make some good points which is why I personally take weeks or more to come to any firm conclusions as to whether a change to my system is worthwhile. In some ways we have to get to the stage where we forget we’ve implemented a change and just enjoy some music over a long enough period to balance (average?) out the day to day variations in our perception. In the grand scheme of things small changes probably don’t individually matter that much but small insignificant things can add up for the better, and of course, sometimes for the worse. I suppose the aim is getting the system to sound as good as possible so that we can then just enjoy the music without any distractions.
 
Indeed. But perceived differences - even if they emerge over time - may not necessarily be real. It is perhaps useful to realize that whatever problems various blind test arrangements may have, it does not make sighted tests any more reliable. They are inherently unreliable.

If you don’t trust your aural memory when listening blind, why should we trust your aural memory when you listen sighted? When you compare two things sighted are there not still the same issues of remembering what the first thing sounded like, disconnecting and reconnecting etc.? So if aural memory is unreliable, there’s no value in comparing two things successively, whether or not you know which is which. In which case all of your claims to hear differences under sighted conditions are worthless.
These are reasonable points, on their own terms, but they do divert from my underlying point that the blind test, as advocated on here and often used to refute various subjective observations, is itself a basically flawed thing*. Therefore the detractors of the subjective approach are themselves coming from a similarly flawed position, but may be less aware of it. What does that suggest about the relative biases which may be in play?

My point is that the subconscious mind will tell us plenty, by the way we respond to the system over an extended period. That's bypassing, to a large degree, any conscious subjective assessment.

But more fundamentally than that, if aural memory is not to be trusted, nor subjective impressions (conscious or subconscious) then why are we even bothering with hifi at all?

*Just look at the lengths hc25036 would have us go to to address the flaws.
 
My point is that the subconscious mind will tell us plenty, by the way we respond to the system over an extended period.

So you are saying that the subconscious mind can make comparisons over an extended period, whereas the conscious mind can’t even manage them over a few seconds?
 
But surely any conclusions drawn on this are subject to the far greater variability of other factors that affect our enjoyment. I don't know about you, but on some days or occasions I am really drawn into the music, and marvel at the sound quality of the hi-fi, but on others days I'm much less moved. This could be due to all sorts of biological factors that might affect the physiology of our hearing, but also on some occasions listening to music seems much more appealing than on others. For some reason I enjoy music more at night-time than during the day. I'm sure some will try to tell me that this is because the mains-supply is quieter, but I think its simply down to (my) human preference for the best time to listen. Also it is possible to spend too much time listening I think - I often enjoy music more when I've had a few days break away from listening. I have also had the experience of thinking some change has made a major improvement - then reverting back some days later to find no real difference.
Yes, I agree that my enjoyment will vary from day to day, and I'm sure mood is just one factor in that. On your last point: this is, I think, one of the more telling arguments. I often find that the biggest differences only emerge after a period using a new device, when I revert to the original device. Most times, the old device is now disappointing because I've got used to what the new device does and miss that when it's absent; but sometimes there is no meaningful difference, or it is swings and roundabouts. It's an interesting and necessary part of the process. Blind test that, if you can!
 
So you are saying that the subconscious mind can make comparisons over an extended period, whereas the conscious mind can’t even manage them over a few seconds?
It's nothing to do with comparisons. Perhaps I haven't made my point clearly enough. The subconscious reaction is in terms of:

Am I listening more frequently, or less than I did before the change? Am I listening for longer? Am I listening to a wider variety of music, or a narrower one? Does a listening session range over my entire collection, or am I likely to just home in on one genre? This sort of thing.

The comparison there is to previous habits, not to sound. It is, if you like, a measure of music as a mood-altering substance, and how compelling we find it. So, how much we choose to indulge in it. That is, surely, what we're all looking for?
 
These are reasonable points, on their own terms, but they do divert from my underlying point that the blind test, as advocated on here and often used to refute various subjective observations, is itself a basically flawed thing*.
Any specific examples? And what are there flaws you are referring to in the tests you are referring to?
My point is that the subconscious mind will tell us plenty, by the way we respond to the system over an extended period. That's bypassing, to a large degree, any conscious subjective assessment.
This is a beautiful idea, but sadly not supported by evidence.
 


advertisement


Back
Top