Guess what. Everyone’s biased. The people who say they aren’t biased are definitely biased. The key to navigating the world is trying to understand what someone’s bias is, perhaps determining why, and keeping that in mind when you’re evaluating their opinion. This could definitely be about politics (you are not immune to propaganda), but instead I’ll stay in my wheelhouse and talk about how this impacts audio reviews.
While everyone is biased, people are not biased equally, nor does their bias necessarily negate their opinion. I’ll give you a personal example. I am biased toward headphones that offer a bit more bass than most people prefer. I mention this in nearly every review. I’m aware of my bias because I’ve been doing this a long time. It’s also my job to be as objective as possible, a laudable but impossible goal when all reviews are subjective.
I think, or at least hope, this is something you already know about product reviews. Where I want to dive a bit deeper is an aspect of reviews that I think is misunderstood: specifically, where reviews come from and the inherent biases of platforms. Maybe this is a dry topic, but I’d argue it is not too “inside baseball.” If anything, this is something important to talk about. There are countless places to find reviews right now. They can also get classified into a few categories. There are strong opinions out there, even among my fellow SoundStagers, about which of these categories is the least biased. All have pros and cons.
Having participated in all of the options below, I believe I have a perspective that, while not unique, is at least somewhat informed from the inside.
Traditional media with ads
Examples: Websites like this one, magazines, newspapers
Pros: Professional reviewers (theoretically)
Cons: Appearance of bias toward advertisers (best case), actual bias toward advertisers (worst case)
This is the original method for generating reviews of products. A magazine or website publishes reviews, and the people writing those reviews are paid by the publication. The publication gets money from advertisers. Some would say this creates an inherent bias toward advertisers. I would argue it creates the appearance of bias, but not necessarily an actual bias. In the best cases, there’s a barrier between the people selling the ads and the people writing the reviews.
At some publications, this barrier is more porous than at others. Believe it or not, in my career I have never been asked to give an advertiser a good review. The bias usually shows up in other ways, like getting coverage at all. I couldn’t tell you who advertises on SoundStage! sites. I’ve never looked. Why would I care? Owner and publisher Doug Schneider lets me review whatever I want. To be honest, I don’t think we’ve ever discussed any brand, advertiser or not.
So at a decent publication run by honest people, the potential for bias is low. Figuring out what’s a decent publication is of course the trick. Some people won’t look, or won’t ever believe that any “separation of church and state” is possible. The truth is more complex.
Also, while magazines historically had a subscription fee, that usually barely covered the cost of delivery. Website subscriptions I’ll cover in a moment.
Affiliate links
Examples: Wirecutter, basically every other review website now (sorry)
Pros: Less bias toward brands
Cons: More bias toward selling anything
I was one of the first people Brian Lam brought on board at Wirecutter. Back then, we were just a handful of lunatics doing something different. At the time, the idea of using affiliate links (where the site gets a commission if you buy through their link) to run a successful business was considered madness. People thought he, and later we, were crazy. Our reviews were 10,000+ words. We did in-depth, ridiculous testing that took months, and yet somehow we were making money? Traditional media assumed we were hemorrhaging cash until the New York Times bought the site for many millions of dollars. Now, nearly every review site has either a direct ripoff of the Wirecutter idea or, at the very least, affiliate links in its content.
There are two ways of looking at the potential for bias in this method. The original pitch for Wirecutter was that our motivation was to find the best product for the most people because if you didn’t like it, or worse, returned it, we’d make less or no money. So we weren’t biased toward a specific brand, because we made our money on the sale, not because of a specific product. For what it’s worth, we were never told what made money either, and in the early years we’d often review things that weren’t monetized.
The potential for bias is, of course, just wanting you to buy anything. The point is less about finding you the right thing and more just showing you lots of things and hoping you click on one. Also, depending on how the site is run, there could be managerial influence to cover certain categories that sell over those that don’t. Some companies also offer special commissions to sites to recommend their products, but I’d be surprised if that info trickles down to influence a reviewer. With Wirecutter, any affiliate links were established after the review was finished.
So the appearance of bias here is different from ad-based media, but once again, it does rely on the culture of the publication.
Subscriptions
Example: Consumer Reports, Rtings.com
Pros: Less bias toward advertisers (assuming there aren’t ads), less bias toward selling (assuming there are no affiliate links)
Cons: Significantly reduced usability for most people, yet another subscription
When it comes to potential bias, or the appearance of bias, a subscription model is very interesting. Consumer Reports has long been held in high regard by many because it doesn’t accept advertising. The New York Times led the way with paywalled news, which was considered folly at the time but is now widely seen as genius. It will be interesting to see if Rtings.com has success doing something similar.
Potentially, a subscription model is ideal for removing the perception of bias, both from advertisers or to just “sell” with affiliate links. I say “potentially” because even though CR and Rtings.com lock all or part of their content behind paywalls, both are still getting affiliate income. So to me, this is kind of “having cake and eating it too.” Having been in the Big Chair for a publication before, I won’t fault anyone for trying to keep their company afloat.
The paywall means you’re making significantly less money on affiliate links because you have lower overall traffic. Both CR and Rtings.com put their buy buttons in front of the paywall, though it varies how much you can see of the specific reviews and ratings to determine what they’re recommending. If you don’t get enough people deciding your site offers content good enough to literally pay for, you’re stuck with less affiliate money and less subscription money, which means fewer reviews, which means fewer links and less coverage, which means fewer subscriptions. That kind of death spiral would keep me up at night. Thankfully, I don’t need to worry about that.
So in theory there’s potentially less bias, but in practice it’s not quite as pure as it seems. The money’s got to come from somewhere.
Monetized videos and podcasts
Example: YouTube
Pros: Potentially less bias toward advertisers (varies)
Cons: Potentially a lot more bias toward advertisers
Google makes more money from YouTube ads than the major studios do in ticket sales. It’s easy to see how and why it has grown. It’s basically free to make content, everyone has a phone, and if you find a niche, you can eventually monetize your videos and make money. Make no mistake, anyone making a livable wage on YouTube is treating it like a full-time job. It’s a grind.
It’s also rare that successful YouTubers only make money on advertising revenue. More likely, they’re making money through Patreon, donations, affiliate links (see above), and sometimes sponsorships. So the idea that YouTubers offer a “purer” look at a product is insane. They have the same potential biases as traditional media. I’d argue it’s potentially worse because there’s zero oversight. At least with websites and magazines, there’s a potential wall between ad sales and editorial. If a YouTuber is scraping by and they get offered to do an ad read for Brand X, how do you think that affects their review of Brand Y? A lot? A little? It can’t be none. That’s not possible. We’re all human.
So the trick with this method is finding someone who is knowledgeable and honest enough not to accept money directly from the brands they cover. These people exist, of course, but YouTube’s a big platform.
While YouTube is far bigger, these same pros and cons also apply to podcasts with ad reads. Podcasts without ads are not perfect either, but potential biases aren’t directly related to their income since, theoretically anyway, there isn’t any. I’d still want to know what their bona fides are before I trust their opinions, however.
User reviews
Examples: Reddit, forums, comment sections
Pros: The appearance of no bias
Cons: Is it even a real person? What do they know?
I’ve certainly read plenty of “hot takes” from people who insist that user reviews are the only way to get an unbiased take on a product. The idea is that the person reviewing actually purchased the product, so how could they be biased? They spent their own money!
Well, once again there’s still bias. First, especially in this age, is that comment even from a real person? Do you know how easy it is to prompt an LLM to generate a positive review? Seconds later, a company can post that “review” with a few “totally authentic” user photos using an anonymous account.
Even without AI chatbots, there’s still inherent bias in user reviews that can’t and shouldn’t be ignored. First, there’s the inherent bias of money spent. Anyone who has spent days or weeks researching something is not neutral to the outcome. Most people hate being wrong, so they’ll gloss over issues with the product. Or they’ll be so anxious about their purchase they’ll accentuate otherwise minor issues. Also, what’s this person’s experience with the category? If this is their first pair of high-end headphones, of course they’re going to sound amazing compared to the earbuds that came with their phone. With user reviews of any kind, including restaurants, hotels, etc., people are far more likely to post about something they don’t like than to go out of their way to post about something that was fine but not amazing.
This isn’t to say user reviews are never accurate, nor is it to say that if a lot of people feel strongly about a product, such consensus isn’t worth considering. Not knowing a stranger’s bias is, I would argue, worse than a professional who knows the category, has lots of experience reviewing and being as objective as possible, but gets paid to do reviews.
But then again, I’m biased.
. . . Geoffrey Morrison
