More on Blind Testing

Lately there’s been a lot of discussion on these pages about performance measures and blind testing. If Helen of Troy had the face that launched a thousand ships, Robert Harley’s coming out against blind listening in The Abso!ute Sound (Issue l83) may prove to be the op-ed that spurred a thousand rebuttals. Here’s one more.

I think the reason we at the SoundStage! Network have a hard time letting go of this issue is that Harley’s assertion (that blind listening is unreliable junk science) flies in the face of common sense. If for no other reason, Harley’s argument is absurd because it concludes that while blind testing is a valid experimental protocol in every area of science, it doesn’t apply to the physics of audio reproduction.

Twenty-five years ago Dr. Floyd Toole, then with the physics department at Canada’s National Research Council, published his groundbreaking work on the physics and psychoacoustics of loudspeaker testing. That work led to, among other things, the establishment of such prominent speaker companies as Paradigm, PSB, Energy, and Mirage, and it later secured Dr. Toole as the head of research at Harman International. About a year ago Dr. Toole published the sum of his work in Sound Reproduction, a book that Amazon.com has trouble keeping in stock. And the point of this short summary? It’s this: many of the study’s results followed from the use of blind listening panels. So if blind testing is really the intellectual dog Harley claims it is, does this mean that Toole’s research is invalid?

Readers might ask why, if blind listening is so powerful a tool, we don’t use it for reviews. The answer is simple: blind tests are a logistical nightmare. In the case of loudspeakers, to get the best experimental result each speaker would have to be placed in the exact same listening position without the listener being aware of the change. Harman has a dedicated room with a mechanism for just such a purpose, but unfortunately not everyone has the budget or the space to replicate that setup. Then there’s the issue of impedance and level matching speakers to the test amplifier, on the fly, which would have to be done by a non-listener. Finally, a reviewer would need to have at least three or four different speakers on hand to conduct a reliable test.

But it’s one thing to admit blind listening tests are difficult, and quite another to claim they’re invalid. Robert Harley’s argument against blind listening implies that we can’t hear the absolute sound if we can’t see the equipment producing it. Following this logic, we can never hear the best in audio reproduction in a totally dark room, and it’s something the blind can never experience. Mr. Harley writes of blind listening tests leading to "patently absurd conclusions." There is certainly something absurd in all of this, but it’s not blind testing.

. . . Colin Smith
editor@goodsound.com