Why you can't trust audio measurements

[THOMO]

1 hour ago, aussievintage said:

 

a lot of modern problems and exacerbated by group think, and social media has multiplied the effect disastrously.

Indeed.

Within such a context I quite like this quote from another forum.

 

"Scientific measuring for the sake of ascertaining something is useful and needed but not for every damn claim made by someone who hears a difference. The fanaticism exhibited here by some in the objectivist camp are akin to religious extremism.

Every conceivable excuse as to why you can't hear what you're hearing is right up there with being born with original sin. That you can't hear correctly and are a fool to trust your senses, is quite an asinine position to take for a harmless and simple hobby.

Saving you from yourself (emotionally and economically) is yet another parallel aspect of this religious like zealotry. So is having a flock of sheep followers and acolytes sent out to other lands (websites) to combat (think Crusades), proselytize and convert.

If god were never invented or ASR never existed, we'd all be right where we're at, enjoying our hobby and sharing our experiences, as if none of this ever arrived on our doorstep in a flaming paper bag."

[rocky500]

Can we get a amen for that one.

 

[Stereophilus]

9 hours ago, DrSK said:

 

Hi Grant

 

Assuming this post was a bit of a summary in response to my question about the purpose of this thread.

 

These threads keep on running, because knowledge isn't widespread. Witchcraft and voodoo keeps being repeated as fact. And most people can't tell the difference or don't actually care.

 

Understanding requires enthusiasts to understand niche parts of niche professions within niche professions, where there are professionals who have pulled together broad knowledge within that discipline to make sense of audio.

 

This isn't a simple task, to get to a level of understanding in other professions. And explaining stuff takes a lot of time.

 

Much of what people want to know, is already explained in this thread to various levels of detail.

 

I entered the hifi realm with an open mind as I enjoy music. And have put professional effort into trying to resolve voodoo from actual physical and psychological effects.

 

I've come out the other side very cynical of much of the industry as there is much commercial reliance on voodoo, where many of the commercial interests don't seem to know the difference either and honestly believe what they say.

 

It is entirely feasible to sell $100k speaker pairs without understanding how to relate measurements fully to human perception.

 

The science of relating physical measurements could be improved, I'm fairly sure just using existing psychoacoustic models to test significance of results would help. And I'm not certain, some manufacturers aren't doing this. However, there isn't any commercial motivation to publish this IP.

 

For example, I own IP and have patented stuff, it could advance current knowledge if I published the IP that drives the accuracy of the patent. But why undermine my competitive edge in the way I do things?

 

Even what has been published out of Harmon et al, it's what they were happy to release. What are they holding onto?

 

And I doubt there is enough public interest in the subtleties of what we care about in this thread for millions to be spent on it using public funding. If there was, I'm sure the work would be done by now.

 

I think the industry broadly has us where they want us, and they are the only ones likely to fund it.

That’s an interesting perspective.  I would seek to have you clarify a few points to ensure I have understood your thinking.

 

 - the knowledge/IP of (re)creating certain facets of music in a home hifi are researched by hifi companies.

 - the companies hold this knowledge and do not release it, as it would be detrimental to their business model to share their knowledge.

 - the companies deliberately obfuscate to avoid encroachment on their IP, and maintain profits.

 - this obfuscation results in the debates we see on on audiophile forums.

 

Do I have it correct?  If so, I would agree that as you said, it does come across as cynical.  The main issue in your post seems to be that you despise capitalism.  I know plenty of drug companies doing exactly the same thing.  They pay to research a drug through clinical trials, market it, then sell it for as much as they think the market can bear.  We live in a capitalist society, so it kinda goes with the territory.

 

The deliberate obfuscation is the interesting bit… how widespread do you believe this to be? Ie, is it all companies, or just a few?  Is there any evidence for this, or is it speculative based on your experience?

[POV]

13 minutes ago, rocky500 said:

Can we get a amen for that one.

 

what?  Yet another unnecessarily divisive post referencing this mysterious and unnamed band of nefarious objectivist zealots and referencing ASR forum.  Seriously!?

[MLXXX]

33 minutes ago, THOMO said:

or ASR never existed, we'd all be right where we're at, enjoying our hobby and sharing our experiences, as if none of this ever arrived on our doorstep in a flaming paper bag.

 

I think that the benefits of crosschecking/confirming our raw aural perceptions by objective means were recognized before the founder of ASR was born! And will continue to be recognized centuries from now.

 

I also note that the term "hobby" (which I often see mentioned on this forum) may create an impression of mere fooling around or playing a game. For those who want to listen to music with high fidelty, it can be more than a mere game or pastime. It can be a serious endeavour. Measurements can be important tools and/or sources of information for that endeavour.

 

And please readers of this post, do not assume that what I've written here rules out listening subjectively. For selection of speakers for example it is an important factor in a purchasing decision: weighing up how much better certain speakers sound subjectively, against how much extra they would cost.

[rocky500]

14 minutes ago, Stereophilus said:

The deliberate obfuscation is the interesting bit… how widespread do you believe this to be? Ie, is it all companies, or just a few?  Is there any evidence for this, or is it speculative based on your experience?

There is this mention from a question to this from the audience at 1:01:30

 

 

[Stereophilus]

1 hour ago, Stereophilus said:

The deliberate obfuscation is the interesting bit… how widespread do you believe this to be? Ie, is it all companies, or just a few?  Is there any evidence for this, or is it speculative based on your experience?

I didn’t intend for the questions to be rhetorical, but I have realised there is a glaring example of this behaviour in the MQA product.  The evidence from GoldenSound and subsequent scrambled responses by MQA execs is definitely convincing of this behaviour being present in the hifi industry.  Shame. Shame. Shame.

 

However, this neatly draws us back to the OP, referencing another GoldenSound exposition, in that measurements (from the manufacturer, or from commentators) can also be tools of deliberate obfuscation.  Ranking DACs by SINAD (alone) is a great example of this.  Shame. Shame. Shame.

 

When I look at the industry and the forums in this light, I understand the cynicism.

 

And yet, I listen to my system and I am instantly content. Perhaps I am better off being a happy fool?

[Guest deanB]

2 hours ago, rocky500 said:

Can we get a amen for that one.

 

I'll see your amen and raise you a hallelujah.

[LHC]

13 hours ago, DrSK said:

Even what has been published out of Harmon et al, it's what they were happy to release. What are they holding onto?

 

That is a good question. The narrative we were told was that the Harman facility was offered to Toole and Olive to conduct science research and not for exclusive benefit to the company. This way they can claim there were no conflict of interest. The predictive model parameters found by Olive was patented for years and therefore not available to the public. Interestingly (in another forum thread) I discovered there was another data plot diagram in that patent application that was excluded from Toole's book. That second plot results were a lot worse than the related diagram published in the book. And to your point what other information may have been withheld and not published? 

[frednork]

6 hours ago, LHC said:

 

That is a good question. The narrative we were told was that the Harman facility was offered to Toole and Olive to conduct science research and not for exclusive benefit to the company. This way they can claim there were no conflict of interest. The predictive model parameters found by Olive was patented for years and therefore not available to the public. Interestingly (in another forum thread) I discovered there was another data plot diagram in that patent application that was excluded from Toole's book. That second plot results were a lot worse than the related diagram published in the book. And to your point what other information may have been withheld and not published? 

With the research I was involved  with which  had partial funding from public and commercial entities, any research delivering potential ip discoveries  was "ringfenced"  and that aspect paused until an appropriate assessment was made and if necessary then ip lawyers brought in and then that particular bit of the research was hived off to ensure it could be protected and commercialized and not in the public domain either through patent or just confidentiality agreements and limiting information spread. 

The trick was ensuring that nothing definitive had occurred prior to ringfencing and information hadnt already got out and of course the initial research agreements which had the ip ringfencing clauses and how things would happen.

 

In the above case it would be simple to fund Olive and Toole for "foundational" research which formally characterises what was basically known by experts in the industry but not laid out in the public domain and anything novel (we used to call it "interesting") that may have popped up through this work was hived off and a separate private project setup with more funding for Toole or Olive but no publishing.  It would be extremely rare for a commercial entity to just fund public research without some secondary benefit at least.  It can work pretty well and be a win win for companies and researchers. 

 

Edited October 10, 2022 by frednork

[Grant Slack]

Hi LHC,

 

12 hours ago, LHC said:

...The narrative we were told was that the Harman facility was offered to Toole and Olive to conduct science research and not for exclusive benefit to the company. This way they can claim there were no conflict of interest.

Dr Toole was already a preeminent researcher in the field when he joined Harman in 1991, with 25 years of non-commercial audio research in public research institution(s) in Canada. He made it a condition of his accepting Dr Sidney Harman's offer of a head of research position, that he can publish his work undertaken in that position.

 

You make it sound like Dr Toole and Dr Harman conspired to do secret research for Harman's interest, while putting a misleading narrative to the public that it was all philanthropy and public domain. "...the narrative we were told...", "This way they can claim...."

 

Quote

The predictive model parameters found by Olive was patented for years and therefore not available to the public.

I think you are confused. Firstly, patents are available to the public. Secondly, you dated the patent at 2012, yet the 'suppressed' predictive model parameters were published by Olive in 2004. A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model: Sean E Olive, AES Fellow.

 

Quote

Interestingly (in another forum thread) I discovered there was another data plot diagram in that patent application that was excluded from Toole's book.

You are giving the wrong impression. The second data set was openly discussed in Toole's book. Secondly, the 'suppressed' graph was published by Olive in 2004, see above, and referenced by Toole in his book.
 

Quote

That second plot results were a lot worse than the related diagram published in the book. 

And for good reasons: Toole believes the larger set of speaker tests probably has an artificially high range of test results, as the tests were fragmented, and long-term, and subject to less strict organisational control than a single experiment, so IMHO that is probably why he published extracts and graphs from the smaller test instead, ie, it was higher-quality work. Nevertheless, he still describes the larger test's result as "impressive". In fact, he wrote that the impressive result from the larger test, despite having more opportunities for variation, means "that the listeners themselves were highly stable "measuring instruments"".

 

Quote

And to your point what other information may have been withheld and not published? 

Nothing was withheld; you seem to have gone on a fishing expedition to smear mud on some of the best research out there, and not for the first time, either. Toole's book was 550 pages and covered about 225 topics, but you think the reason that a lower-quality graph wasn't published in it as well as the higher-quality work, even though it did get discussed and full reference given to a paper containing it, was a deliberate attempt to mislead readers, plus a sign that there is so much more information being withheld. 

 

Regards,

Grant

Edited October 11, 2022 by Grant Slack

[MLXXX]

15 minutes ago, Grant Slack said:

Dr Toole was already a preeminent researcher in the field when he joined Harman in 1991, with 25 years of non-commercial audio research in public research institution(s) in Canada. He made it a condition of his accepting Dr Sidney Harman's offer of a head of research position, that he can publish his work undertaken in that position.    ...

 

Thanks Grant for clarifying a number of issues. (I did find the adverse allegations a bit odd, but you have now dispelled them.)

[LHC]

5 hours ago, Grant Slack said:

You make it sound like Dr Toole and Dr Harman conspired to do secret research for Harman's interest, while putting a misleading narrative to the public that it was all philanthropy and public domain. "...the narrative we were told...", "This way they can claim...."

 

It was research (a fact) and they can indeed claim there was no conflict of interest (a fact). Nothing wrong with what I wrote and it was not leading. 

[LHC]

5 hours ago, Grant Slack said:

I think you are confused. Firstly, patents are available to the public. Secondly, you dated the patent at 2012, yet the 'suppressed' predictive model parameters were published by Olive in 2004. A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model: Sean E Olive, AES Fellow.

 

For clarity the patent application was filed on 2 March 2005, and granted on 13 Nov 2012. I don't believe one could apply for patent if the results were already published in 2004 as you said. There were several related patent applications made by them earlier in 2004. 

 

Now prompted to re-read the patent again I realised my memory failed me about the model parameters statement, so I will admit I made the mistake of claiming them to have been suppressed. The model and parameters is/was protected by the patent. 

[LHC]

6 hours ago, Grant Slack said:

Nothing was withheld; you seem to have gone on a fishing expedition to smear mud on some of the best research out there, and not for the first time, either. Toole's book was 550 pages and covered about 225 topics, but you think the reason that a lower-quality graph wasn't published in it as well as the higher-quality work, even though it did get discussed and full reference given to a paper containing it, was a deliberate attempt to mislead readers, plus a sign that there is so much more information being withheld. 

 

That is silly and revisionist writing. Recall an older thread on SNA here where you talked about the small test set graph with the nice clean result, and you said at the time you don't know why the second graph was not in the book. You claimed then that second graph must have been very similar - a claimed I refuted by finding the lower-quality graph from the patent. So I wasn't the one been misleading in this forum. 

 

Recall also in that same older thread I pointed out that Sean Olive wrote several times in his blog that the prediction from his model has an 'accuracy' of 86%, when in fact that was the correlation value (therefore a lower level of accuracy). Another member verified this with Toole. Olive must have gotten word of this because sequently he correctly referred to this as correlation (see the third dot point in the slide at the 48.30 min mark of the video below). So I am not about misleading people, but instead about seeking the truth. 

 

 

[LHC]

12 hours ago, frednork said:

In the above case it would be simple to fund Olive and Toole for "foundational" research which formally characterises what was basically known by experts in the industry but not laid out in the public domain and anything novel (we used to call it "interesting") that may have popped up through this work was hived off and a separate private project setup with more funding for Toole or Olive but no publishing.  It would be extremely rare for a commercial entity to just fund public research without some secondary benefit at least.  It can work pretty well and be a win win for companies and researchers.

 

It is a possibility, but I do not know of any evidence to support such a scenario in the case of Toole and Olive's work.

 

What is intriguing is kind-of in the opposite direction:

• There doesn't appear to be many speakers designed directly from their research results (I think Toole admitted something to this effect on ASR if I am not mistaken);
• One Harman speaker designer (I recalled it was for JBL?) wrote in an interview article (after he left the company) that speakers conforming to the research results tend to be very smooth but not very dynamic. But he also stated he did not use blind testing with his design ;
• Another famed speaker designer (not from Harman) respected the work of Toole and Olive, but said that their principles can cause design difficulties in other areas (I am paraphrasing from memory)
• I previously tried to search the literature but really couldn't find any peer-reviewed replication of their research work, so I don't know if it has ever been scientifically and independently verified outside of Harman?
• Harman no longer fund that research and Olive's model has been abandoned (as noted in the video above)

So I feel we still don't have the 'full' story yet and hence my post yesterday. It does not mean I disrespect their body of work which I regarded as excellent. 

 

[Assisi]

Tonight, on the ABC 7.30 Brian Cox the renowned cosmologist was interviewed.  In response to an interesting question what does it mean to be human he said:

 

 

“The science is about the intellectual, philosophical and the emotional”

 

 

He was talking about the universe.  Nevertheless to me , the word Audio could be substituted for Science.  This thread encapsulates all three of the above.  Take your pick of one or all of the above.

John

[MLXXX]

17 hours ago, LHC said:

So I feel we still don't have the 'full' story yet and hence my post yesterday. It does not mean I disrespect their body of work which I regarded as excellent. 

Thanks, LHC, for the various clarifications.

 

My understanding is that Toole and Olive were pioneers in investigating, identifying and formalising the importance of certain attributes of loudspeaker system performance, for human beings listening to recordings.  Their studies provided a useful and solid theoretical foundation.  

 

I'm aware that high-performance bookshelf-sized speaker cabinets housing relatively small diameter drivers are now feasible because of various improvements (e.g. materials used for diaphragms and surrounds; stronger magnets).   However, these developments can be seen as refinements of the same basic form of technology ("speaker drivers in a cabinet").

 

There is significant use today of active speakers, that is to say speaker enclosures incorporating one or more power amplifiers (which may incorporate negative feedback derived from movement of a driver).   And, importantly in audiophile circles, we now have fairly common use of room correction via DSP, used to control the signal delivered by power amplifiers.

 

For myself looking broadly at the developments over the last few decades in loudspeaker systems  ---   and I may be making a big leap of logic based on rather scant personal knowledge ---  I find it hard to imagine that anything really important in terms of theoretical findings about speakers and human hearing has been hived away for use by only one or two specialist manufacturers. That's just my gut feeling.

 

Edited October 12, 2022 by MLXXX

[sir sanders zingmore]

15 minutes ago, MLXXX said:

I find it hard to imagine that anything really important in terms of theoretical findings about speakers and human hearing has been hived away for use by only on or two specialist manufacturers.

If that were the case, logically one might expect that Harman would consistently produce speakers that leave the rest miles in their wake because of their secret sauce. 

By all account their speakers are good but not that good to suggest they have some super duper edge that no other manufacturer knows about. 

[davewantsmoore]

5 hours ago, MLXXX said:

I find it hard to imagine that anything really important in terms of theoretical findings about speakers and human hearing has been hived away for use by only one or two specialist manufacturers. That's just my gut feeling.

 

100%

 

There's nothing really that special going on except "reproduce the input signal without distortion"..... which is essentially the finding of Toole/Olive.    (Gasp).

 

 

 

[davewantsmoore]

5 hours ago, sir sanders zingmore said:

If that were the case, logically one might expect that Harman would consistently produce speakers that leave the rest miles in their wake because of their secret sauce. 

By all account their speakers are good but not that good to suggest they have some super duper edge that no other manufacturer knows about. 

Indeed.

 

The "super duper edge" (nice!) is not in knowing that you should not distort the signal .....its finding ways to avoid distorting it.

 

The size and shape of the transducers and cabinet, are the things which contribute the most to the speaker distortion.... and so in terms of gross distortion, all 1" + 5" + 5" speakers that are 25cm wide and 1m high, are all approximately the same.... so in many ways, there's little edge to be had (once all competitors are calibrated to a common-ish on-axis amplitude response)

 

 

Toole found that if you're going to have this unavoidable distortion, then it's best to make it as smooth and even as possible.... which was already pretty well known, but they actually did the work to say "we checked very carefully, and people definitely notice".

 

[rocky500]

Just watched this video and found myself nodding in agreement with a lot of this.

What I also like is they don't tend to say they are 100% right and this is the way it is. Just what they might observe from their listening.

 

 

Edited October 15, 2022 by rocky500

[Geoff Millar]

As a practical example of 'measurements v listening', have a look at the Dayton DC160 'Classic' range 6" woofer. 

 

Quoting from Zaph Audio's web page:

 

"This woofer does not perform very well, with horrible tall order harmonic distortion right smack in the midrange. The response curve between 1 and 2kHz is pretty rough also". 

 

That's true and Zaph really knows his stuff, but when that woofer is used in a well designed speaker such as Paul Carmody's Classix II or Dennis Murphy's "Affordable Accuracy Monitor", it sounds fine and they're really enjoyable speakers.

 

Geoff

[BugPowderDust]

I was reading this old Stereophile article (interview with John Dunlavy) earlier today and it made me want to revisit this thread and put some quotes in for reading: https://www.stereophile.com/interviews/163/index.html 

 

His approach to speaker design was very heavily tilted in the direction of only measurements before any listening occurs.

Quote

John Dunlavy: Oh, no. Listening comes later. Because if you stop to think about it, no loudspeaker can sound more accurate than it measures. It may sound worse, or it may sound sweeter, prettier, but if we're talking about absolute accuracy—the ability of the speaker to reproduce as perfectly as possible whatever's fed to it—such a system can never sound more accurate than it first measures. So we try to get the greatest accuracy we can achieve from measurements. Then we begin doing what some people might call "voicing," because the best set of measurements are still open to interpretation. 

 

You could have a speaker that makes ±1dB, for example. And you say, "Gosh, how can you better that?" But what if it's +1dB over an octave and a half, say from 1-2.5kHz? And then it suddenly jumps down to -1dB, for a total change of 2dB, for the next octave? You're going to hear a spectral imbalance. The specs might look great—golly, here's a speaker that measures ±1dB—but it's not going to sound nearly as good as a speaker that is up and down 1dB every third of an octave. In the real world we're used to hearing that—reflections from the walls of the room cause similar variations—and we tune that out. 

 

It may come as a surprise—this is giving away a trade secret—but when I design a loudspeaker, I first design it by looking at the step response. I find that by playing around with the crossover network while observing the step response in real time, any change I make is immediately available. When I get the step response right, everything else goes along. It's implicit. It goes along for the ride.

 

[frednork]

5 hours ago, BugPowderDust said:

I was reading this old Stereophile article (interview with John Dunlavy) earlier today and it made me want to revisit this thread and put some quotes in for reading: https://www.stereophile.com/interviews/163/index.html 

 

His approach to speaker design was very heavily tilted in the direction of only measurements before any listening occurs.

 

Its a fantastic interview and shows just how far ahead of the game Dunlavy was in terms of the technical understanding and how it impacted upon listening. Supported by lots of listening (and a very good understanding of the technology!!.)

 

Its a shame we never got to hear the dsp active speaker called Magnus he was designing.

 

It makes me wonder are there any designers who are out and proud in their belief that listening is not required? (Speakers or otherwise)