Dustin Posted February 16, 2012 Posted February 16, 2012 . . . . if your believing some lack of understanding in this regard from my part, thats a failure on your part. not something I can do anything about Hi : ) al, You're right and I was wrong. I misread your post . . my apologies. But I might take you up on this point: if anyone is involved with any rigid testing involving human senses they would know they infact embrace the subject instead. understanding how important a factor that is and give the subject training and understanding to make a subjective and informed conclusions. and it will always remain person dependant the result. no matter how rigorous or controlled you make it. Of course it will be person dependent. How could it be anything else. We don't want it to be anything else. But after you've put a hundred people through the same test, you're likely to get a result. It might be a positive result or a null result but either way, you'll be better informed than you were before.
betty boop Posted February 16, 2012 Posted February 16, 2012 Hi : ) al,You're right and I was wrong. I misread your post . . my apologies. But I might take you up on this point: Of course it will be person dependent. How could it be anything else. We don't want it to be anything else. But after you've put a hundred people through the same test, you're likely to get a result. It might be a positive result or a null result but either way, you'll be better informed than you were before. no worries dustin, it very much depends on the persons going through the same test. since we were talking about beverages, I think might have posted this before, but have a watch of this youtube of this myth busters episode "vodka top shelf filteration" "Through a double-blind taste test, the cheap vodka seemed to taste better with every subsequent filtration, although the top-shelf vodka beat them all. However, a chemical analysis showed no actual difference between the filtered and unfiltered cheap vodka." although not abx and not a full DBT probably one of the best illustrations of how your knowledge, training, senses can influence the result. In your 100 people example for instance if you had every one of the 100 with trained and calibrated senses eg the vodka expert you are going to get a completely different result picking it every time in perfect order to say people off the street in the kari's and jamies... In industry a lot of work goes in with taste panels. With beverages for instance its all about education and training for both palette and nose and your brain to understand and pick various aspects. flavours aromas, taste and smell, colour. And even with all the training and regular retraining and calibration there are some very good at picking some things than others. Alcohol companies will have trained and calibrated panels. They acknowledge within that some will have greater resolution and accuracy in their senses. They also utilise general public in their research and testing and acknowledge they are untrained and uncalibrated. In situations for instance they will do a mix of testing all full ABX DBT and between the their panel and public. and find their top testers for particular aspect will pick everytime whereas general public clueless to the issue. Thats the human element and acknowledging it. Its why I said cannot divorce the person from the test. and why "it will always remain person dependant the result no matter how rigorous or controlled you make it"
Jake Posted February 16, 2012 Posted February 16, 2012 Michael, there are two basic questions that need to be separated, yet somehow they often get mixed together in this kind of discussion:1. Is controlled testing the most reliable way to determine sonic differences in isolation of all other factors? I would say an emphatic YES! I don't see a comparable alternative that will reach the same goal. 2. Is controlled testing the best way to make audio choices? Again an emphatic answer - NO! The thing that makes it effective for its intended purpose makes it ineffective as the sole basis for purchase decisions. When you buy, there are many other factors involved. Sound quality is just one of them, but aesthetics, warranty and perceived reliability, brand recognition and pride of purchase, features among other things are an important part of the choice. So this brings up the question 'how useful is controlled testing in the larger context of making a purchase decision?' That is up to the individual to decide. Good post Paul. End of story really.
proftournesol Posted February 16, 2012 Posted February 16, 2012 OK, so I can see how it's a useful technique for some DIYers, have any SNAers used this technique to buy any commercially made equipment?
cheekyboy Posted February 16, 2012 Posted February 16, 2012 OK, so I can see how it's a useful technique for some DIYers, have any SNAers used this technique to buy any commercially made equipment? I would stand to be corrected, but I'm sure Dr X said he bought all his equipment after evaluation by controlled testing/listening. For me, I'm the exact opposite and I've purchased zero equipment on this basis. Cheers, Keith
GFuNK Posted February 16, 2012 Posted February 16, 2012 OK, so I can see how it's a useful technique for some DIYers, have any SNAers used this technique to buy any commercially made equipment? I've used it to decide whether to sell equipment .
Dr X Posted February 16, 2012 Posted February 16, 2012 I would stand to be corrected, but I'm sure Dr X said he bought all his equipment after evaluation by controlled testing/listening. I was kidding mate! All of my stuff I've purchsed completely unheard.
Paul Spencer Posted February 17, 2012 Posted February 17, 2012 OK, so I can see how it's a useful technique for some DIYers, have any SNAers used this technique to buy any commercially made equipment? As mentioned, I've used it to confirm a choice already made. A more recent test has also shown the advantage of a more advanced crossover. The difference was shown to be in what it can do, rather than simply sound quality.
proftournesol Posted February 17, 2012 Posted February 17, 2012 Hmmm... given the vehemence of some strongly heard views on this topic, I thought that this thread would be flooded by responses from people that have bought their whole systems using nothing but controlled listening!
Drizt Posted February 17, 2012 Posted February 17, 2012 Why would you think that? I get the distinct feeling that some people still don't understand the concepts being discussed and in what context they are relevant. Paul's posts have been brilliant. People went a bit silent after he posted.
RoHo Posted February 17, 2012 Posted February 17, 2012 (edited) Paul very neatly described what he looks for when doing a controlled test: location of instruments, bass quality etc. He's obviously an experienced, educated listener. I'm thinking that to have a successful objective test these qualities MUST be present in the "testee". Is there a risk of false outcomes with "inadequate" or inappropriate subjects? And a couple more quick and genuine questions. Is the goal of the test to choose a preference or merely a difference between units under test? And does the result of any one test have meaning outside of its original setting? If you change the testee then all bets are off, right? Don't get the idea I'm a total naysayer. I've used similat techniques described by Paul to choose between capacitors or resistors for my DIY projects and the results surprised me. Edited February 17, 2012 by RoHo
proftournesol Posted February 17, 2012 Posted February 17, 2012 Drizt don't get me wrong, I can certainly see the value in it and I'm a supporter of it, especially in the examples that Paul gives, in fact it'd be hard to evaluate the changes without some sort of controlled listening, yet nobody seems to do it to buy equipment despite freely acknowledging how poor we are at doing accurate comparisons without it.
ArthurDent Posted February 17, 2012 Posted February 17, 2012 I'm thinking that to have a successful objective test these qualities MUST be present in the "testee". Is there a risk of false outcomes with "inadequate" or inappropriate subjects? Of course there is. Imagine if a panel of only beer drinkers were asked to judge at a wine show. Mind you if the "inadequate" or inappropriate subjects were in fact the ones who were judging the gear for their own use and whilst their opinions might and probably wouldn't agree with a panel of experienced testers, isn't their opinion then valid.
Drizt Posted February 17, 2012 Posted February 17, 2012 Drizt don't get me wrong, I can certainly see the value in it and I'm a supporter of it, especially in the examples that Paul gives, in fact it'd be hard to evaluate the changes without some sort of controlled listening, yet nobody seems to do it to buy equipment despite freely acknowledging how poor we are at doing accurate comparisons without it. It is just a tool for ascertaining if an audible difference is 'real' or 'imagined'. Nothing more really. If you don't care if you are imagining the change there would be no need to do an objective test. But if you want to 'prove' a difference is 'real' (actually audible and reliably detectable) then you have to do objective testing. If people could even acknowledge that I'd be happy. Yet some still cling to the hope that subjective listening tests can ever be as accurate. It is bewildering.
Drizt Posted February 17, 2012 Posted February 17, 2012 I'm thinking that to have a successful objective test these qualities MUST be present in the "testee". Is there a risk of false outcomes with "inadequate" or inappropriate subjects? If we forget about trying to extrapolate results to other circumstances and other people then the answer is no. Lets just make it as simple as possible... 1). We do the objective testing for our own benefit..... - If someone claims they hear a difference during subjective testing then they should be able to also hear that same difference during objective testing.
rantan Posted February 17, 2012 Posted February 17, 2012 "The chances of anything coming from Mars are a million to one he said" ..............and still they come.
RoHo Posted February 17, 2012 Posted February 17, 2012 No, Drizt, what's bewildering is your refusal to undestand that some people disagree that the test itself can PROVE (your word) anything at all. Until we have a way of calibrating the measuring instrument ie. subjet then the result can never be objective proof. OF COURSE it can be an interesting, if not illuminating experiment.
Drizt Posted February 17, 2012 Posted February 17, 2012 No, Drizt, what's bewildering is your refusal to undestand that some people disagree that the test itself can PROVE (your word) anything at all.Until we have a way of calibrating the measuring instrument ie. subjet then the result can never be objective proof. OF COURSE it can be an interesting, if not illuminating experiment. How can it 'not' prove a positive? People get too hung up on the objective tests trying to 'disprove' that a difference exists (i.e. prove it doesn't exist - which is actually impossible to do). If you can '100/100' in an 'objective' test would you consider that to be 'proof' that the difference exists? Conversely .... If you can '100/100' in a 'subjective' test would you consider that to be 'proof' that the difference exists? I wish people could look at this objectively rather than trying to rationalise what they want to be true.
Drizt Posted February 17, 2012 Posted February 17, 2012 Some people need to see and / or know what is playing to hear a difference. I think that is pretty telling!
Drizt Posted February 17, 2012 Posted February 17, 2012 People should forget about preference when they are trying to understand objective testing. The aim is.... Can you reliably and repeatedly hear an actual audible 'difference' under controlled conditions. I think people feel that it is in some way negating the subjective preference side of audio. It clearly is not. It is a tool we can use to confirm 'differences'. Preference is still going to be subjective.
cheekyboy Posted February 17, 2012 Posted February 17, 2012 I was kidding mate! All of my stuff I've purchsed completely unheard. Thanks Doc, I didn't actually believe it, just that I was sure I'd read it. Cheers, Keith
RoHo Posted February 17, 2012 Posted February 17, 2012 How can it 'not' prove a positive? People get too hung up on the objective tests trying to 'disprove' that a difference exists (i.e. prove it doesn't exist - which is actually impossible to do).If you can '100/100' in an 'objective' test would you consider that to be 'proof' that the difference exists? Conversely .... If you can '100/100' in a 'subjective' test would you consider that to be 'proof' that the difference exists? I wish people could look at this objectively rather than trying to rationalise what they want to be true. My problem is that I don't see how one person's result becomes "proof" of anything, except to themselves. Remember, you're the one whose going on about being objective and scientific methodologies. If you want to test whatever and tell us the results, then great. I'll be interested. But I'm a sceptic, I'll never take someones opinion as being my unfallible proof. And that really the downfall, as I see it, of listening tests. In the end it's just someone's opinion, we're not reading a voltages or measuring a temperature. Of course a 100/100 result is comprehensive and in the real world would be very meaningful but it's an extreme example. But if you got that result then I would believe that there was an audible difference.
Drizt Posted February 17, 2012 Posted February 17, 2012 It is proof that the one person in that one test was able to hear a difference. That is all that is required. Forget about extrapolating the results for any other purpose.
Once was an audiophile Posted February 17, 2012 Posted February 17, 2012 I was kidding mate! All of my stuff I've purchsed completely unheard. I honestly thought at least one control listening preacher would practice what they preach...........
Drizt Posted February 17, 2012 Posted February 17, 2012 I honestly thought at least one control listening preacher would practice what they preach........... kajak, you are trolling. It has been explained to you many times on how that line of questioning is irrelevant to the topic. Please at least try to add something useful to the conversation.
Recommended Posts