Jump to content

Blind Test GTG #1 - CD Digital Transports


Drizt
 Share

Recommended Posts

DISCLAIMER: Blind testing has proven to be a controversial topic, but hopefully maturity levels are high enough to be able to discuss topics without them degenerating into a flame fest. So PLEASE if your not interested in blind testing please look away now :).

After many efforts to organise a blind test (which failed miserably) we were finally able to muster enough interest to get one of these babies off the ground. The big THANK YOU has go to the man of the moment, our very own LuckyDog, for offering up his house and system up for us. Very much appreciated mate.

After LuckyDog had generously volunteered his house and system for the blind test GTG I decided to put together a bit of a plan for how I thought things would go. I then circulated this plan to the other guys to look over and put forth their opinions regarding a few of the options I had laid out. Well... in the end we didn't really follow the plan, but we did find our own way on the day that seemed to work well. Sure it wasn't perfect, and theres a few things we would do differently if we had our time again. Ill come back to that later.

Stepping into LuckyDogs dedicated HIFI/HT never fails to bring a smile to my face as its an equisite setup. Lovely high end gear, very well set out and comfortable furniture, and well thoughout and executed room accoustic treatments of different varieties. LuckyDog has made quite a few new additions to his system since the last time I was there. There has been the addition of his new Dynaudio C4's that replaced the VAF I93's (which were extremely enjoyable themselves). There has also been the addition of the the Behringer CX2310 crossover to releave the mains of bass duties (115Hz and below) and redirect the bass to the DD15 sub. Not to mention the new addition of his very serious looking TT setup. Lets just say the system as a whole is awe inspiring.

I arrived about 20 mins late after missing a turn (stupid road signs! - it wasn't my fault I swear). Upon arrival only LuckyDog and Spearmint were present and I got the sinking feeling that the other guys would not be able to make it as the forum was down and they would not be able to get the address or contact details of anyone else at the GTG to get said details. Luckily the guys were able to get the details from the email copies of the PM's... crisis overted :eek: It was great to be able to meet Earl for the first time and catch up with the other guys again after what seems like a very long time between drinks. Each of the guys were great characters, and were very easy going and made the day extremely enjoyable. The list of attendees are below.

The Attendees:

LuckyDog

Spearmint

ehtcom

junkmail

Drizt

To start the day we did some casual listing of LuckyDogs system as it was, and then decided to setup the system in preparation for the blind testing. Below is a list of how we hooked it up.

System Setup:

CEC CD transport connected to digital coax input 1 of Stello DP200

Marantz SACDP connected to digital coax input 2 of Stello DP200

Stello DP200 used as both a DAC & Preamp

Stello DP200 XLR outputs connected to the Behringer CX2310

Behringer CX2310 to the DD15 sub for below 115Hz duties

Behringer CX2310 to the Rowland Poweramp for above 115Hz duties.

Rowland Poweramp connected to Dynaudio Confidence C4 Loudspeakers

Casual Sited Testing:

We decided to test out the various upsampling modes before we did any real listening. Theres (Bypass) 44.1kHz, 48kHz, 96kHz and 192kHz upsampling. This proved to be quite difficult. As on each change there seemed to be an obvious difference in the first second or two of switching, but then it settled down and sounded pretty much the same. No one put forth a strong opinion / preference for any of the modes and as such we stuck with the 192kHz option. This really could have been its own blind test topic, and something I would be interested in doing / reading about later on.

We then proceeded to roughly sync up two burns of the same CD in each transport. For quite some time (over different CD's and Differenent tracks) we flicked between the two options. To cut a long story short, between the five of us 2 preferred the Marantz and 3 prefered the CEC used as transports. Ill leave it to the other guys (if they want to) to put forth what they heard during the sited listening tests.

Before we started to do the real blind testing we decided to test the volume levels just to make sure everything was as it should be. Given that we are testing digital inputs there should be absolutely no difference in volume but for the sake of completeness we decided to do it. Well.... this didn't get off to a great start. For some reason we were detecting a 1.5dB difference in volume (using the same Test Tone Disk on both transports) - WTF ??? To get a volume difference there would have to be some manipulation of the digital values (DSP) that are transmitted before they reached the DAC. Each of us started to wonder what the hell was going on, but we decided to compensate for the difference in volume and proceed with the first suite of blind tests. After the first round of tests (stopped at 9) it became pretty obvious that there was a volume difference (7/9) correct choices (method used is described in the following). It was decided that we should measure the SPL again to make sure we had it right. Surprise surprise we had made a mistake. After the second lot of SPL measures we came back to the same volume measurements for each transport.... problem solved. Now on to the next set of blind tests.

There was much discussion before we started on what would be the best method. In the end we decided upon choice (:D below as we thought it would give us the best chance of detecting a difference if it existed.

Method:

Test Type Choices:

Choice (a) - Blind ‘ABX’ testing.

- Transport ‘A’ will be played, followed by Transport ‘B’. Transport ‘X’ (One of Transport ‘A’ or Transport ‘B’) will be played and the testee is to nominate if Transport ‘X’ is Transport ‘A’ or Transport ‘B’.

Choice (:D – Blind ‘Is it different?’ (?? – Not sure what that’s called?) testing.

- Transport ‘X’ will be played (One of Transport ‘A’ or Transport ‘B’) followed by Transport ‘Y’ (One of Transport ‘A’ or Transport ‘B’). The testee is to nominate if Transport ‘X’ and Transport ‘Y’ are the same Transport or different Transports.

Then we had to decide on what method we would use. In the end we decided to 'Sync' the two CD's and then allow the person being tested to raise their hand when they wanted to switch between the two transports.

Test Method Choices:

Choice (a) – Start the track at the same time on both transports.

Choice (B) – Play the track on transport ‘A’ for a period of time, then play the track on transport ‘B’ for the same period of time.

Choice © – The tester changes between the transports at set intervals.

Choice (d) – The testee nominates when to change between transports.

Description of the testing:

We had:

1). One person performing the testings.

2). Everyone was present in the room, but nobody was allowed to talk or make much movement. Of course this is not perfect, but everyone agreed to it and it worked fine.

3). One person tested at a time.

4). The person testing would select an input that was not connected so that there was silence.

5). The tester would select either 'A' or 'B' to play.

6). The testee would raise their hand (in their own time) when they wanted to switch.

7). The tester would then again select an input that would produce silence

8). After a period of time the tester would then select either 'A' or 'B' to play.

9). The tester would listen for a period of time (their own choice) until they would say that what they were listening to was the same as the first or different to the first.

10). The tester would write down the actual combination, the testee's selection and whether they were right or wrong.

11). Repeat until the testee had had enough.

RESULTS:

After only one suite of tests we decided to call it a day and start on the beer :D. The result of 10 tests was 50%.

Now there are MANY MANY deficiencies with our testing, but I think I speak for everyone in that it got you thinking about what level of differences really do exist if they exist at all. I hope the guys got something out of the experience even if they don't believe in blind testing.

THINGS TO NOTE:

1). Any volume difference completely skews the results. If you are to do blind testing, lots of care must be taken to make sure that the volume differences are accounted for.

2). You have to be very careful if you do testing where you 'SYNC' up two CD's. Timing of the music is a dead give away that the inputs have changed. If 'A' is out of time with 'B' then if you keep the beat of 'A' in mind while the switching occurs, then if 'B' does not match up with that beat then you know they have changed. Also if you hear a bit of music you had already heard on the previous one then its very obvious that it has changed.

3). Lots of planning and rigour is required to make the results as meaningful as possible. As this was the first Blind Test GTG we were not to worry about the protocols used as to my mind we were using this as a guinie pig so to speak.

4). If you are 'SYNC'ing up the CD's sometimes when you switch between the inputs if its not during a constant piece of music it can be very hard to compare them. On a few occasions we swapped, and as the music started a new section I found it almost impossible to compare them. It might be better to Play one whole song through, and then play the same song again all the way through before stating your answer. Not sure.. possibly needs further investigation. I think the 'SYNC'ing works well if you choose to switch when there is a pretty constant part in the music playing.

After the Blind Testing:

We cracked open the beers, talked freely, enjoyed the lovely food and hospitality and generally had a great time.

We then swapped the Digital transports and compared a DVD Player to the CEC transport. Again I dont think anyone could pick the difference. Guys, any extra thoughts on this one? I certainly couldn't tell the difference.

My favourite combination on the day was when we introduced the Marantz SACD (Analogue) output into Spearmints buffer and then into the Stello DP200 (purely as a Pre) through the system playing "Closer to the music". ABSOLUTELY MAGIC!

To finish off the day we listened to quite a bit of vinyl which was great to listen to. There was an increase in the noise floor and the occasional snap crackle and pop, but while the music was playing it was extremely laid back, enjoyable, easy listening with a massive sound stage.

I think thats enough writing for now.... Ill just post this and come back to it over time.

Cheers.

Guys that were present... feel free to add anything you want, make suggestions on what your thought worked / didnt work and correct any mistakes I have made in this report.

EDIT: FORGOT TO MENTION.... We got a chance to listen to one of Earls integrated valve amps. Was very impressive, with an extremely low noise floor (almost dead silent even with an ear to the drivers). It sounded very musical and was able to get the speakers upto very high SPL (measured 96dB average I think it was.. from memory). It had run out of head room by that volume though, so it wouldn't be recommended to listen at those levels with the sensitivity of the speakers we were listening to. Hopefully the others will add their thoughts about this lovely unit.

Regards,

Drizt.

Edited by Drizt
Spelling, Grammar, the usual..
Link to comment
Share on other sites
  • Replies 41
  • Created
  • Last Reply

Top Posters In This Topic

Congratulations on your first (of many I hope) DBT! And a great report too Drizt, I enjoyed the read....just need some pics though....:eek:

I am so pleased this has finally taken place I have to say, and from your report I reckon you have done even better than I was expecting for a first go.

It sounds like a DBT is actually a lot of hard work, rather than sitting around just drinking piss :D

Great stuff, looking forward to many more!

I do have a couple of tiny points though....(apparently if I grow my hair thicker no-one will notice :)) But these arent criticisms just observations.

1. When you talk of upsampling modes it is obvious to me that you are talking about the Stello DP200. But I doubt that anyone else would know that, unless they have a first hand experience with the Stellos or are thoughtful enough in their reading to work it out.

2. When the Marantz SACD player was unleashed are you sure the buffer was used? I am pretty sure a buffer is only for a digital signal, and as SACDPs can only output analogue then the buffer would be superfluous.

Cheers,

Jake

Link to comment
Share on other sites

3). Lots of planning and rigour is required to make the results as meaningful as possible. As this was the first Blind Test GTG we were not to worry about the protocols used as to my mind we were using this as a guinie pig so to speak.

Drizt.

Lots of planning and liquor is required to make the results as meaningful as possible.

Fixed your typo drizt.

well done guys. in my case not so much for the results, but more that some were willing to make it happen! as a start excellent, but it's main use (perhaps) is that of as a test case to sort protocol for future efforts.

My only 'beef' with it is simply that it was not a particularly 'startling' result because it may not have been a particularly 'startling' test. What I mean is (to me) why test two 'top end' cdps and use them as a transport only, would not it gave been more illuminating to compare an expensive cdp vs a very cheap cdp as a transport only?

Having said that, of course I am aware of the 'atmosphere' surrounding you when getting this of the ground, so it makes perfect sense to start with something completely uncontroversial.

It is the start that is important, well done everyone.

Link to comment
Share on other sites

Lots of planning and liquor is required to make the results as meaningful as possible.

Fixed your typo drizt.

well done guys. in my case not so much for the results, but more that some were willing to make it happen! as a start excellent, but it's main use (perhaps) is that of as a test case to sort protocol for future efforts.

My only 'beef' with it is simply that it was not a particularly 'startling' result because it may not have been a particularly 'startling' test. What I mean is (to me) why test two 'top end' cdps and use them as a transport only, would not it gave been more illuminating to compare an expensive cdp vs a very cheap cdp as a transport only?

Having said that, of course I am aware of the 'atmosphere' surrounding you when getting this of the ground, so it makes perfect sense to start with something completely uncontroversial.

It is the start that is important, well done everyone.

Thanks mate. Yeah I was just happy to get it off the ground. We did compare the CEC against LuckyDogs DVD player (although its a pretty expensive player itself) with similar results, no difference reliably detected (although we did not blind test them)

Excellent! I admire your determination to see this through, and the rigour with which you approached.

And good on you for presenting the results in such a straight-forward manner.

Thanks mate. Im glad you enjoyed it. :)

Link to comment
Share on other sites

Great write up Drizt, I enjoyed the read.

Just to put things straight, I was never going to be part of the blind testing as I find them rather strenuous. My involvement from the start was to deliver the CEC transport for comparison purposes, plus of course deliver it to its new owner.

I’m glad I hung around though and enjoyed both the company and the equipment being played, especially the Vinyl and the integrated valve amp. Both of these would make me happy with the quality of their reproduction.

The results were interesting and opened my eyes, and IMO goes to show how much impact a DAC has with regards to CD playback.

2. When the Marantz SACD player was unleashed are you sure the buffer was used? I am pretty sure a buffer is only for a digital signal, and as SACDPs can only output analogue then the buffer would be superfluous.

Cheers,

Jake

Hi Jake,

The buffer we used is an Audio-GD Buffer3 which is line level.

B31s.JPG

Link to comment
Share on other sites

Thank's for the write up of the day Drizt an well done on getting the event happening.

Sounds like a fair bit was discovered on the day . as in how you thought it might go and the discoveries on the day about how it needed to go . next time should be easier.

With the different volume levels between the transports , I wonder if they had different mains voltage requirements as in a 220volt unit could output more voltage on the spdif than a 240 volt unit . Or could it be a ploy by the manufacturer of the louder transport so that it stands out more when comparing different units in a demo ??

Good to hear you enjoyed listening to some records . They sure have their benefits. Interesting that Spearmints SACD- Buffer was the best sound on the day. the buffer must really help.

Wow another Valve amp visits LuckyDogs , Amazing stuff . Rarer than a ford in the driveway.

Cheers

Moondog

Link to comment
Share on other sites

Hi Mndie.

I think our first readings were affected by our own movement within the room. This became apparent when one of us moved during the second attempt and it threw the meter off by a few db.

We then moved the meter closer to the speakers and on this setup both transports were set at the same level.

Cheers, Earle.

Link to comment
Share on other sites

Great work Drizt. That is a very well written description of the day.

I was most impressed with how all participants attended with an open mind and prepared to give the session/test the best chance possible to succeed.

I was also impressed, albeit in a slightly different way, when I returned from helping my wife with our crying son for 20 min's or so, to return and find 4 lunatics ripping apart my system to hook up a valve integrated, interconnect cables flying through the air, as were power cords, transports & solid state amps being discarded all over the shop :)! Nice :eek:

BTW Drizt, you have left your SPL meter here.

Link to comment
Share on other sites
Nice writeup on an interesting GTG, did you get to the bottom of the 1.5dB difference between the two transports?

Cheers.

Mondie

Hi Mndie.

I think our first readings were affected by our own movement within the room. This became apparent when one of us moved during the second attempt and it threw the meter off by a few db.

We then moved the meter closer to the speakers and on this setup both transports were set at the same level.

Cheers, Earle.

Thank's for the write up of the day Drizt an well done on getting the event happening.

Sounds like a fair bit was discovered on the day . as in how you thought it might go and the discoveries on the day about how it needed to go . next time should be easier.

With the different volume levels between the transports , I wonder if they had different mains voltage requirements as in a 220volt unit could output more voltage on the spdif than a 240 volt unit . Or could it be a ploy by the manufacturer of the louder transport so that it stands out more when comparing different units in a demo ??

Good to hear you enjoyed listening to some records . They sure have their benefits. Interesting that Spearmints SACD- Buffer was the best sound on the day. the buffer must really help.

Wow another Valve amp visits LuckyDogs , Amazing stuff . Rarer than a ford in the driveway.

Cheers

Moondog

Thanks guys. Earl is right, the level differences were only due to the measurements we took being affected by bodies moving in the room. Digital transports can not effect the volume unless the digital '0's and '1's are manipulated (DSP) before hitting the DAC.

Link to comment
Share on other sites

Great work Drizt. That is a very well written description of the day.

I was most impressed with how all participants attended with an open mind and prepared to give the session/test the best chance possible to succeed.

I was also impressed, albeit in a slightly different way, when I returned from helping my wife with our crying son for 20 min's or so, to return and find 4 lunatics ripping apart my system to hook up a valve integrated, interconnect cables flying through the air, as were power cords, transports & solid state amps being discarded all over the shop :)! Nice :eek:

BTW Drizt, you have left your SPL meter here.

Thanks again for having us lunatics over mate. What did you think of the blind testing? Oh, and what did the remaining lunatics get up to after I had to depart? Ill grab the SPL meter some other time mate, thanks for letting me know.

Link to comment
Share on other sites

Thanks for the kind words Spearmint. And thanks heaps for bringing the CEC transport around. Im glad you were there, it was great to catch up again.

Regards,

Drizt.

Great write up Drizt, I enjoyed the read.

Just to put things straight, I was never going to be part of the blind testing as I find them rather strenuous. My involvement from the start was to deliver the CEC transport for comparison purposes, plus of course deliver it to its new owner.

I’m glad I hung around though and enjoyed both the company and the equipment being played, especially the Vinyl and the integrated valve amp. Both of these would make me happy with the quality of their reproduction.

The results were interesting and opened my eyes, and IMO goes to show how much impact a DAC has with regards to CD playback.

Hi Jake,

The buffer we used is an Audio-GD Buffer3 which is line level.

B31s.JPG

Link to comment
Share on other sites

I forgot to thank the fellow blind testers for bringing their goodies over. Drizt for bringing his Stello, a snake pit of cables and duplicated music, Junkmail for his power cables, which looked great, Spearmint for bringing his CEC transport and audio buffer and Earle for bringing his toob integrated, which sounded great.

Link to comment
Share on other sites
Great work Drizt. That is a very well written description of the day.

I was most impressed with how all participants attended with an open mind and prepared to give the session/test the best chance possible to succeed.

I was also impressed, albeit in a slightly different way, when I returned from helping my wife with our crying son for 20 min's or so, to return and find 4 lunatics ripping apart my system to hook up a valve integrated, interconnect cables flying through the air, as were power cords, transports & solid state amps being discarded all over the shop :)! Nice :D

BTW Drizt, you have left your SPL meter here.

Oh da bastards snuck the valve amp in when you wern't looking :eek: Lunatics hehe

Link to comment
Share on other sites

There's been a lot of talk . . .

Look it's not really my cup of tea but I am somewhat amazed you actually got a blind test up and running. Congrats on getting one of these off the drawing board and into action.

BTW (sorry, just can't help myself :) :eek: ) did you enjoy the vinyl Drizt? Did it make you want to forget the blind test and the DACs and just listen to music? You do realize I can now quote Drizt as stating "we listened to quite a bit of vinyl which was great to listen to." It's a slippery slope mate, you're wavering I can feel it :D

OK, ribbing over, hope you all had a good time and, probably surprisingly, I will look forward to your further results. Maybe one day I might find the time to come to a blind test, we could compare TTs or even phono stages (actually that would be a bloody good blind test).

DS

Link to comment
Share on other sites

Glad you got this going Drizt, I much prefer reading reading results and how you got to them than the usual banter that is thrown around.

Congrats to all involved.

Does anyone think that by doing this they have changed there outlook on hifi and how they would purchase it in the future?

Link to comment
Share on other sites

nice work to all it looks like.. as much as there is much to be said about the merits of the test, I heartily encourage people actually trying it for themselves - the worst that can happen is that no-one learns anything and doesn't have any fun, and that certainly doesn't sound like that was the case at all! :)

I look forward to more commentary if/when the next one occurs :eek:

Link to comment
Share on other sites

Like Terry i am not surprised by the result considering the quality of the players used.

Either way, good on you guys for doing it and posting your findings

Link to comment
Share on other sites

It was a great day and i enjoyed meeting up with everyone .

Phil thank you for opening up your home and i have to say listening to your equipment was a treat - loved the vinyl

Dritz thank you for organizing the day and your write up has described the day and results perfectly . I to have to say that setting the volume was not easy and the sync of the music was hard to set. And that changes in the flow or pace fof the music made the job a lot harder .

If i was to do the day again i would prefer to listen to each transport with the music to be used and then do the A B test , i found that after listening the same music over long time period that it became harder to distinguish between transports .

Earle your amplifier was also a treat to listen to i was impressed

Link to comment
Share on other sites

Thanks guys for adding your well wishes and support :)

Im just so very glad that we got it off the ground and that it wasn't too boring or tiresome.

I think that although blind testing is hard, it can be a very useful tool.

Of the guys that attended, what are your thoughts on blind testing now that you have experienced it? Is it something you would do in the future from time to time? Is it something you think is valuable? Or just a nice to experience but not for you?

Link to comment
Share on other sites
Like Terry i am not surprised by the result considering the quality of the players used.

Either way, good on you guys for doing it and posting your findings

Thanks JohnA :) Your right, that the two units tested were of high quality and price to start with. What was interesting though, was that during the sighted testing just about everyone was leaning towards one transport or the other stating various reasons why. But under the blind test conditions these differences could not be reliably detected. Then again with the DVD player against the CEC nobody was confident that they could tell a difference reliably.

Link to comment
Share on other sites

G'day,

After only one suite of tests we decided to call it a day and start on the beer :). The result of 10 tests was 50%.

What does this mean? The correct result was picked 50% of the time (equivalent to chance) ? Or 50% of respondents picked the correct result?

--Geoff

Link to comment
Share on other sites
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.




×
×
  • Create New...
To Top