Jump to content

Why you can't trust audio measurements


Recommended Posts


5 hours ago, rocky500 said:

 

Interesting video I came across tonight. 

 

 

 


Thank you for linking this video, it is very informative to say the least. Interesting how he explains how some manufacturers, especially some DAC manufacturers, are gaming reviews.

 

cheers,

Terry

Link to comment
Share on other sites



38 minutes ago, Cloth Ears said:

Nothing new under the sun.

True, but conveniently not often broadcast and many sites are not transparent in how they go about their testing.  

Link to comment
Share on other sites

41 minutes ago, Cloth Ears said:

Nothing new under the sun.

True, but it may give a bit of insight to the less technically minded, as to why measurements can be misleading, especially if you're measuring the wrong thing, or the wrong way.

Link to comment
Share on other sites


32 minutes ago, frankn said:

True, but conveniently not often broadcast and many sites are not transparent in how they go about their testing.  

 

29 minutes ago, bob_m_54 said:

True, but it may give a bit of insight to the less technically minded, as to why measurements can be misleading, especially if you're measuring the wrong thing, or the wrong way.

Yes, but measurements of the kinds being shown (and the older ones going back into the 90's) have always only provided an indication. It's the same when someone says to you "these are the best speakers ever", or "this interconnect will change your life", or "this amp is unbeatable". A balanced approach is always best - once you like what you've heard...

Link to comment
Share on other sites

Why would you trust Audio measurements in the first place?

Common sense will tell you that one can not hear the difference between a SNR of -100dB and one of -120dB, and if one can, it might be a 10 year old with sensitive hearing!

Same goes for THD+N ratio. Heck, there are so many of us that enjoy the sound of tubes and yet some bicker over a THD of 0.00002%.

How many of us will buy an amp for its silly performance figures?

I listen with my ears! Full stop.

I heard to many top performing amps sounding average to give a hoot about these tests.

 

Interesting video never the less!

Edited by Ihearmusic
  • Like 10
Link to comment
Share on other sites

4 hours ago, Cloth Ears said:

 

Yes, but measurements of the kinds being shown (and the older ones going back into the 90's) have always only provided an indication. It's the same when someone says to you "these are the best speakers ever", or "this interconnect will change your life", or "this amp is unbeatable". A balanced approach is always best - once you like what you've heard...

You (along with others it seems) missed the point of the video.. What is being pointed out is that if you do look at measurements to compare devices, make sure the measurements are valid, and taken on a level playing field..

 

There was no emphasis from the video to say that measurements are what you should use as a deciding factor

Edited by bob_m_54
  • Like 5
Link to comment
Share on other sites

30 minutes ago, bob_m_54 said:

You (along with others it seems) missed the point of the video.. What is being pointed out is that if you do look at measurements to compare devices, make sure the measurements are valid, and taken on a level playing field..

 

There was no emphasis from the video to say that measurements are what you should use as a deciding factor

 

Yep, I just knew some people were going to jump in and start up the old bandwagon without bothering to understand what was being said.  The intro to the video explains what it is about. 

  • Like 3
Link to comment
Share on other sites

2 minutes ago, aussievintage said:

 

Yep, I just knew some people were going to jump in and start up the old bandwagon without bothering to understand what was being said.  The intro to the video explains what it is about. 

Probably would have picked that up, listening with their ears. 🤣

  • Like 1
  • Haha 1
Link to comment
Share on other sites

15 hours ago, bob_m_54 said:

You (along with others it seems) missed the point of the video.. What is being pointed out is that if you do look at measurements to compare devices, make sure the measurements are valid, and taken on a level playing field..

 

There was no emphasis from the video to say that measurements are what you should use as a deciding factor

Sigh!

I didn't miss the point. You are saying to me what I posted (in different words). Maybe I should not have used the word 'But' at the beginning.

And hopefully it will allow for some of the less technically minded...

 

14 hours ago, aussievintage said:

 

Yep, I just knew some people were going to jump in and start up the old bandwagon without bothering to understand what was being said.  The intro to the video explains what it is about. 

So, I can't agree with the video? Hm.

 

14 hours ago, bob_m_54 said:

Probably would have picked that up, listening with their ears. 🤣

Nice!

  • Like 1
Link to comment
Share on other sites

On 27/04/2022 at 2:19 PM, Ihearmusic said:

Common sense will tell you that one can not hear the difference between a SNR of -100dB and one of -120dB, and if one can, it might be a 10 year old with sensitive hearing!

Same goes for THD+N ratio. Heck, there are so many of us that enjoy the sound of tubes and yet some bicker over a THD of 0.00002%.

How many of us will buy an amp for its silly performance figures?

 

Yes, I think that buying amp A rather than amp B because A has a higher SNR and lower distortion when both A and B are ultra-quiet and ultra-low in distortion makes no sense in terms of audible sound quality.  It may perhaps provide a feeling of luxury.

Actually that consideration takes a lot of the wind out of the sails of the video for me.   Even if deliberate "gaming" is engaged in or or "valid measurement options" exercised, these days devices tend to be so good that a resulting say 10% difference in a range of parameters would not be audible anyway, even if it really were a true reflection of differences in performance. The video remains interesting from an academic viewpoint, for example its demonstration that certain measurements fluctuate in real time and a snapshot at one instant can be more favourable (or unfavourable) than a snapshot at another instant.

 

I'd mention that an important reason why measurements stopped being reported by many hi-fi magazines towards the end of the 20th century was that ruler flat frequency responses, very high SNRs and very  low THD + N figures  were not providing information particularly useful to readers. In earlier decades when many hi-fi power amps were noisy, high in distortion, and limited in output power and frequency response, measurements could be of great utility to readers and actually help with purchase decisions.

Edited by MLXXX
  • Like 1
Link to comment
Share on other sites

The real problem he correctly identified is the ranking mentality. Even if Amir at ASR says all of this is below the limits of audibility, everyone gets hung up on the ranking and buys accordingly from the chifi brands, leading to an arms race and gaming of results in a cut throat low profit market. If we were to assume that everything below a certain threshold is below the limits of audibility, then one should only concentrate on weaknesses and flaws in the electronics instead. I do see Amir try to do this in the text, but the ranking graphs are celebrated almost like a religion. If I had an apx55 (which I've been tempted to buy) I would go looking for the things that do affect sound only - one such as intersample overload mentioned in that video can have absolutely huge audible effects which is infinitely more important than even a SINAD of 60dB. Clipping, current limits, behaviour into complex loads, output impedance, handling of varying input voltages and distortion, handling of ultrasonic products, ground loop and ground plane noise proneness, etc. etc. etc. would be infinitely more useful. These are the things high end audio should do well that discriminate it from cheap fi, but we're still stuck on pushing measurements from the 1970s to the extreme when these are mostly solved problems.

  • Like 3
Link to comment
Share on other sites

1 hour ago, Cloth Ears said:

Sigh!

I didn't miss the point. You are saying to me what I posted (in different words). Maybe I should not have used the word 'But' at the beginning.

And hopefully it will allow for some of the less technically minded...

OK, I misinterpreted what you were saying, the word "indication" threw me a bit, because I thought you were inferring that the video was about measurements being an indicator of whether a device sounds better or not. Apologies.

 

I should have read with my eyes eh? 🧐

 

1 hour ago, Cloth Ears said:

So, I can't agree with the video? Hm.

I know that wasn't directed to me, but, you don't agree that measurement data can be inadvertently or purposely fudged?

 

To be honest, I'd most likely totally disregard any quoted data, from a manufacturer, on their advertising blurb. It's simply not of any use, when there is no indication of how the tests were set up and measured.

Link to comment
Share on other sites

  • 2 weeks later...

The point of the equipment measurements on asr isn’t to find the absolute best performing product, it’s to sort well engineered products from those that aren’t in regards to price. The idea is that something that measures well requires a level of attention to detail in engineering that one would hope is seen throughout the rest of the product. Thus you discover $1000 products that are found to be better performing than $20000 products, saving gullible consumers of getting fleeced $19000 by con men / corporations.
 

It is also a fairly self regulating process, in that certain cheap products are bought en mass when proven to measure well. This causes flaws in other areas of said products to be discovered and made visible by the new user userbase, resulting in pressure publicly placed to fix these flaws via firmware updates or in new models.

 

This is a completely deregulated luxury industry where a large percentage of products are provably terrible and sold for made up sums through appeals to emotion, nostalgia, aspiration and FOMO. 
 

But, at the same time, it’s all about the music and you are into what you are into. If you don’t like the measurements don’t pay attention to them.

  • Like 4
  • Love 1
Link to comment
Share on other sites

the funny thing is distortion can and some do make a system sound good.

 

ive found that ASR forum members all they care is about low distortion and if you can hear a difference its your mind playing tricks.

 

Amir has indicated various items (components or which ever he was reviewing at the time) he heard a difference. so he kept on listening till in his head there was no difference as measurements indicated no difference. 

  • Like 1
  • Love 1
Link to comment
Share on other sites

All true... but at the end of the day, mainly FUD (especially in context of the click bait title).

 

Simple take home for almost everyone is that small differences differences in measurements (especially non linear distortion) can't be relied on to be at all relevant.  

 

Can you "trust" audio measurements .... yes, sure .... can you trust them to mean what you think they mean, or what you're told they mean?   LOL.

 

Should you think a device with 105 SINAD is definitely worse than one with 118?..... probably not.

 

It's just "THD" or similar, all over again.    Almost worthless in the context of what is important.

  • Like 1
Link to comment
Share on other sites

The Topping dacs are a good example of what is being discussed here …

 

Amir and his ASR crowd love these things for their excellent measurements and no doubt they do sound very good. While I own one and have heard them in situ in people’s homes, I get the impression that they do not compete at the very highest level of sound performance. I mean that last layer of sonic bliss achieved by very transparent (and expressive) HiFi components.

 

I currently have in my possession a Philips TDA-1541 Killerdac with Miflex KPCU capacitors in the output stage; a Musical Fidelity Trivista  21 dac; a PS Audio Perfectwave dac and a Perpetual Technology P3A dac. Also owned the superb Holo Audio Level 4 dac from Wildism Audio …all of these dacs sound slightly better (more transparent) than the Topping units that I have heard.

 

Cheers,

 

Steve.

  • Like 3
Link to comment
Share on other sites

2 hours ago, Steve M said:

The Topping dacs are a good example of what is being discussed here …

 

Amir and his ASR crowd love these things for their excellent measurements and no doubt they do sound very good. While I own one and have heard them in situ in people’s homes, I get the impression that they do not compete at the very highest level of sound performance. I mean that last layer of sonic bliss achieved by very transparent (and expressive) HiFi components.

 

I currently have in my possession a Philips TDA-1541 Killerdac with Miflex KPCU capacitors in the output stage; a Musical Fidelity Trivista  21 dac; a PS Audio Perfectwave dac and a Perpetual Technology P3A dac. Also owned the superb Holo Audio Level 4 dac from Wildism Audio …all of these dacs sound slightly better (more transparent) than the Topping units that I have heard.

 

Cheers,

 

Steve.

 

the question remains is does it sound better for the right or wrong reasons,  only full measurements will tell you what you're hearing especially when you have a smorgasbord of dacs and the measure equipment on hand.  i haven't sample the list of dacs on your list but many others that were the flavour of the month here on this site, and i can tell you that some of the dacs that Amir has measured i have heard and compared it to others that he's measured with decent figures and i can tell you im glad i dont own those dacs  But that's really not the topic here, the topic is set the measuring equipment up so its a level playing field when measurements are performed.

Edited by Addicted to music
Link to comment
Share on other sites

5 hours ago, genkifd said:

ive found that ASR forum members all they care is about low distortion and if you can hear a difference its your mind playing tricks.

 

Amir has indicated various items (components or which ever he was reviewing at the time) he heard a difference. so he kept on listening till in his head there was no difference as measurements indicated no difference. 

Wow! So much for science 😅

Link to comment
Share on other sites

 

 

Standards!  The reason why measurements  are often untrustworthy is that they do not  adhere to national/ international standards , or they do not fully disclose all the parameters  of the test that can affect the outcome .  In Japan there is JIS standards,  USA  has IEC,  but vary rarely do you see measurements published  that were recorded in tests that conform to these standards.  

 

Companies like Accuphase use JIS and guarantee their measurements.   Not all manufacturers are sticklers for conservative claims for measurements but there are still some. Then you get those that in the area of S/N ratio and THD have made some outrageous claims which  are not really based on noise measurements  of actual equipment.

 

Back in the 70's I recall that there was a German proposal to make it illegal to publish measurements of electronic equipment that did not conform to internationally recognised standards.  Pity that the legislators didn't  proceed with the plan.

 

 

  • Like 1
Link to comment
Share on other sites

6 hours ago, genkifd said:

Amir has indicated various items (components or which ever he was reviewing at the time) he heard a difference. so he kept on listening till in his head there was no difference as measurements indicated no difference. 

 

That sounds odd. Could you point us to an instance where Amir made such a comment, please? I'd be interested in how he expressed himself.

Link to comment
Share on other sites


 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.




×
×
  • Create New...
To Top