Write a comment

Almost all audio product reviews share the same conceit: the idea that the opinions of the person doing the review will correspond with yours. That’s because almost all evaluations of audio products are performed by a single reviewer, with negligible, if any, solicitation of or mention of others’ opinions. The premise has always been that because the writer is an expert, he (or in very rare cases, she) will understand the product well enough to predict whether or not an audio product will be a good choice for you -- or at least he or she will be able to describe the product’s characteristics accurately enough for you to get a good idea of whether or not you’ll like it.

In my 28 years of writing and editing audio product reviews for at least 16 different publications, I’ve worked closely with well over 100 writers and editors. I’ve also had a chance to evaluate the work of a couple hundred more, through reading their reviews of products I’d also tested. In that time, I’ve seen how much writers’ opinions of an audio product can vary -- and, sometimes, miss the mark entirely.

Brent Butterworth

I’ve seen writers overlook obvious performance flaws that measurements or user reviews easily uncovered. I’ve seen them praise products that had severe tonal colorations -- which, again, were found in other reviews or in measurements. I’ve seen them proclaim one product clearly superior to another when the two were, in fact, nearly identical. I’ve seen them trash products that were very successful in the market and, by general consensus, skillfully and thoughtfully engineered.

In short, while I often find the opinions of audio writers interesting, I see no reason to trust them as a sole gauge of audio product performance. Sadly, that goes for my own opinions, too.

I’m fortunate in that early in my career, a wise and experienced person guided me toward a better method. That person was Lance Braithwaite, former technical editor of Video magazine (since absorbed into Sound & Vision), where I began working in 1989. One of my first assignments at Video was to write up a comparison test of VCRs that Lance had organized. He used a calibrated monitor and a professional-grade switcher, and brought in all the Video editors as well as a few freelance writers to give their opinions without knowing which VCRs were in use. This way, he could get multiple perspectives on what differences the participants saw, and how important those differences were to them -- and by extension, how important these differences would likely be to readers. We later used this same structure for countless multiproduct reviews of TVs, video sources, camcorders, and audio systems.


The benefits of this approach are obvious. If, say, four out of five panelists share a viewpoint about a particular product, you can make a pretty solid case for or against it. But if all you know is that a single person likes or dislikes that product, how do you know if he’s one of the four in the majority . . . or the outlier? You don’t.

Sadly, the reasons most publications don’t use multi-listener panels are just as evident.

The logistics alone can be daunting, especially in a time when audio publications make just a small fraction of what they did when I started in this biz. Writers or editors have to spend somewhere between four and 16 hours putting together each test, then enlist (and probably pay for) the services of nearby people who are interested enough in audio to spend an hour or two in the demanding, focused work of listening tests. It’s possible to ship small audio products around to gather multiple opinions, but again, it involves hours of the writer’s time and incurs significant costs.

More troublesome, though, is that the writer and editor, if they’re honest, lose some control over the review process. I’ve conducted and published somewhere around 100 panel tests (easily more than all other active North American audio writers combined), and I’d estimate that in about 75 percent of them, the results are different from what they’d be if only I had done the test. It always pains me to present those results -- but I always present them, knowing that the group opinion will better predict what you’ll think of the product.

So while I wish audio publications would do more multi-listener tests, I don’t blame them.

Fortunately, though, because headphones are small and easy to deal with, and because I live in a city with a large population of music and audio professionals, we’ve figured out a way for me to bring multiple listeners into our testing at SoundStage! Solo. In most of our tests going forward (beginning with last month’s review of the Monoprice Monolith M650 headphones), I’ll be gathering the opinions of two additional qualified listeners. I may be using different listeners from month to month, but all will share a passion for music and audio, and I’m hoping they’ll all be younger than me (so their high-frequency hearing is better) and that every test will include at least one female listener.

Ideally, I’d love to do these tests with a panel of a dozen outside listeners, but that would chew up far more time than I can afford, and more money than SoundStage! would prefer to spend, without any clear economic benefit to either of us. I’d love to do them under blind conditions, but with headphones, that’s extremely complicated and time-consuming, and the listeners can still be biased by the weight and feel of the headphones.

My “gold standard” for audio product reviews has always been double-blind tests with a dozen listeners, accompanied by measurements. Unfortunately, I’ve never had the resources to accomplish that, but I think this is a step in the right direction, and I hope you enjoy it!

. . . Brent Butterworth
This email address is being protected from spambots. You need JavaScript enabled to view it.

Say something here...
Log in with ( Sign Up ? )
or post as a guest
Loading comment... The comment will be refreshed after 00:00.

Be the first to comment.

Latest Comments

Ragav 2 days ago Sennheiser IE 300 Earphones
@Robin28Thank you so much! Much appreciated!
How many decibels do you think these put out with an iphone or android phone?
Brent Butterworth 7 days ago Phiaton 900 Legacy Bluetooth Headphones
@BenThe Phiaton measurements are correct. We'll work on fixing the Technics measurements. I think just ...
Robin28 13 days ago Sennheiser IE 300 Earphones
@RagavBrent wrote more about the HD 600 in an older version of that Wirecutter guide ...
Brent Butterworth 17 days ago Phiaton 900 Legacy Bluetooth Headphones
@BenThanks for pointing that out, Ben, we'll take a look. We did a full review ...
Hi Brent Just wanted to point out that your first measurement graph for the Phiaton's ...
Giacomo Traballoni 24 days ago Beyerdynamic Free Byrd True Wireless Earphones
I've replaced my Status Between Pro (great product overall except for reliability) after a failure ...
Brent Butterworth 28 days ago Phiaton 900 Legacy Bluetooth Headphones
@JorgeI think they sound better. I actually prefer the Phiaton's noise canceling because it doesn't ...
I read the review and want to ask directly if these are better than the ...
Dave Angelo Obico 1 months ago Are There Valid Objections to the Harman Curve?
Heres my take. The etymotic target for iems sounds better for me with a boost ...