Reviewing bike reviews

Rain cycling

Regular readers will know I don’t usually feature product reviews on here. I did a book review last week but that’s about it and I’m not sure I’ll be doing many more. It’s much harder than I thought.

As background, I’ve been on a quest for a rain jacket. You’ve probably seen the pictures of riders with deep section wheels and frames designed in a wind-tunnel. Yet the aerodynamic benefits dissolve in the rain. Because as soon as there’s a downpour riders reach for their rain bags and out comes the jacket. This billows and flaps, a visual and sometimes even audible indication of just how un-aero this garment can be. Obviously the need to stay relatively dry wins but some jackets balloon up big time. It’s not like rain is a freak event and I’ve been trying to find a jacket that doesn’t slow you in a race or a ride.

Hincapie Sports must have read something about my jacket quest and got in touch, asking if I’d be interested in testing their rain jackets especially since they claimed to have the answer. I was interested in the product but also thought it would be an interesting exercise to try and review something. I’ve even put a piece on the blog asking for what readers looked for in a review. There were some helpful comments and some of you even said “don’t do it” as this blog isn’t really about tech reviews which I understand.

But the whole process of review is much harder than I imagined. I thought I’d go for a few rides and then write about the product but it’s just not that simple.

  • It’s one thing to ride with something but another to write a critical assessment. Since I know others will be spending money on the product there’s a duty of responsibility. I try not to be careless on the blog but if I make a mistake about the altitude of a col or the history of a race then you won’t be dropping money on this: a review is different. It feels like every detail has to be covered.
  • Next there’s the issue of bias. I’ve covered angles on this before but this time when a box arrives with several pieces of gear to use then I feel flattered. Even if I am aware of this and point it out to you, plenty of evidence suggests experts get swayed in reviews and I’m only a novice reviewer.

Note the tension between these two points. One side of my brain feels a duty to the reader to highlight every detail from a sceptical point of view; another is saying “aren’t you compromised by the free gear?

I’d been chatting on Twitter about reviews and discovered this morning that the cyclingtips blog has covered things very well. I’d just add that if some say “those magazines have ads to sell” then in some cases the person doing the tech reviews is distant from the sales or accounting people. But here I’m writer, editor, tech reviewer, sales and accounting all in one meaning I’m potentially less independent. But if I started shill reviews you’d call it in the comments.

One idea is to make things clear up front that if I review a rainjacket then I got it free or if I got a book then I paid for it. But even this isn’t so obvious. Studies show that people who buy items rate them higher, obviously the act of buying something suggests you already rate it so highly that you want to hand over money for it.

Tour Magazin
Can you spot the difference?

In fact the more you look into the subject, the harder it gets. As humans we’re all prone to superstition, biased, confusion and other psychological frailties. It makes life interesting but neutrality is hard. To avoid this some magazines test a frame in the lab to measure flex under load, a good idea… but there are tales of bikes being designed to score high in the EFBe test used by some magazines just as cars can be designed to meet the exercises imposed by regulatory crash tests ahead of offering all round protection. And to revert to the original idea of a rain jacket, what if it fits the reviewer but you’re a different shape?

Simple solutions
Enough ifs and buts. Reviews help and more reviews help more so read as many as possible and ask cycling friends for their views before you spend big. Better still, be instinctive because if there’s any bull in a review you can probably smell it.

Finally, for all the examination and introspection let’s note Hincapie Sports presumably sent the gear here to get it featured on screen. They’re not after an analysis of their kit in the context of bias.

Conclusion
Writing a review is a lot harder than I thought it would be. You might be able to describe how your bike rides but turning this into a critical and independent evaluation is something else. The more you examine the methodology and psychology of the review process the more fraught it becomes. But all the more reason to keep a review simple and to give a personal take on the experience, to say what I got from it rather than extrapolate what you’re going to like.

I don’t think I’ll be doing many reviews on here but will follow up with the Hincapie gear reviews soon. After a long period of autumn sunshine the weather has been miserable recently, ideal conditions to test rain jackets. Perhaps riding down a mountain in a downpour is easier than evaluating bike kit?

38 thoughts on “Reviewing bike reviews”

  1. If the opinion of movie-review-blogger can help you: say what you honestly think, and try to be entertaining in doing so. The rest is the reader’s responsibility. You shoudn’t refrain from expressing opinions that you think can be interesting, or care if people suspect you of bias: you are your own, one and only, judge.

  2. I used to always be amazed at how ‘un-aero’ so many of the rain jackets used in the pro peloton were, and consequently how much energy much be being ‘wasted’. Surely with all those suppliers and budgets something ‘much better’ could be designed?

    However I’ve now come to the conclusion that on a horrible wet day some riders need the comfort of a warmer more effective rain jacket to maintain their morale and performance. This would typically be less aero than another riders’ jacket who wasn’t so psychologically/physically affected by rain and cold.

    In summary the performance of a jacket (water resistance, areo qualities) can be objectively tested, but the riders’ own attitudes to rain and cold vary and add a purely subjective element.

  3. Reviews are by definition biased. A review is someones findings of a product. You shouldn’t really look for objectivity in a review. If that’s what you want, you shouldbe prepared to study scientific results.

    dpreview.com does this very well I think. They test every camera in the same studio under the same settings and with the same basis for results. The test shots are there for the geeks to study themselves. In addition they offer their “expert” opinion on the camera for the rest of us, who don’t want to get tricky with all the details.

    I know you can’t apply this method to every product, but I think some kind of benchmarking is beneficial.

    If nothing else, by putting the raw findings out there, it gives the review some kind of honesty because it’s harder to hide anything. Even if you where paid to review a product, by laying out the raw scientific results for everyone to see, your opinion would be less important.

    Nevertheless by putting a piece of kit through a standard proces, it gives some sort of benchmark that, even though not exactly precise, makes one able to compare on an equal basis.

  4. The cheapest $20US clear, flapping rain jacket with velcro strips instead of a zipper, is the only thing that really works for warmth and comfort.

    Hope your review proves me wrong.

  5. I would caution against reviewing products UNLESS you pay full-price for them. This way you have no reason to be biased, you either like the product or you don’t – no influence from the nice guys who gave it to you at no cost, no incentive for others to give you their stuff in order to gain a good review. If the company receiving a good review later decides to buy ad space on your blog, sell it to them! If a company decides to yank their ads after a bad review, too bad for them. But keep the policy that you (like Consumer Reports) purchase the products at retail. This gives your reviews a lot more credibility – if you insist on adding product reviews, which I highly suggest you avoid in any case. The result will likely be a whole lot of “well, you should have tried MY favorite product, it’s way better than the one you reviewed” baloney. There’s plenty of advertorial stuff out there already for us to read.

  6. I liked your review of Mountain High, and if it helped someone check out this rather sumptuous tome, then so much the better.

    You touched on it in the text; this site is your house, so you can set the rules in ways that are impossible to the print journalist. I’m sure the companies that sponsor you would prefer honest constructive feedback rather than praise for the sake of praise if it came to reviewing their products.

  7. @ Kasper – That would be quite a challenge, especially with new kit coming in all the time throughout the year. The nearest I’ve seen is the big light tests that some mags and website do (go to http://www.road.cc to check out ours).

    @ Larry T – People who have already paid for Product X tend to be quite protective and defensive about their choice. It doesn’t make for an objective review and if you are in the position of being given lots of free stuff that source of possible bias vanishes.

    I write for http://www.road.cc and I’d like to think we have a good rep for clear and unbiased reviews. All you can do is provide as much information as you would like if you were looking to purchase an item. That’s what I aim for, but the reader still has to interpret that information and decide how useful or relevant it is to their situation. I’ll tell you if something isn’t cut very well for my shape, but I’ll also tell you that I’m short and slightly chubby.

    As for reviews on Inner Ring – I’d suggest that they won’t add much to what is an excellent blog. The only exception is book reviews, I think those would sit very well with the tone of the blog and your style of writing.

  8. As you noted, writing a review isn’t as easy as it first appears and getting the gear for free makes that even trickier on a number of levels. After reviewing apparel for a year, I have learned a few things. One, a review in isolation is less relevant. You can like or dislike a bib short, but you can’t put it in a meaningful context unless you’ve tried a dozen other new bibs. Then you can start to write from a position of authority as opposed to just staying hey, I like this bib. That’s even more the case in reviewing a bike — have to ride a dozen other to gain the knowledge to speak about “road feel” and handling. The other issue is objectivity and bias and anyone who really expects that is foolish. Even big magazines and test sites have “relationships” with companies that supply them. Plenty of news about companies playing people to write positive reviews on Amazon. As always, let the buyer beware. Personally, I don’t look at reviews as a definite test, just an opinion and I like to think of the reviews as a acting as a brand evangelist for things I like. I don’t write negative reviews because I don’t test junk that I don’t want in the first place. After that, all companies know you have an obligation to strive from some balance — otherwise the reader knows you’re simply a shill.

  9. Anon: I’ve seen the Gabba and remember a Twitter exchange with Highroad’s Craig Lewis who said he thought Garmin had the best rain gear with this jersey and the armwarmers. But is it something that you can put in your pocket or is it more for starting a ride?

    Gear Ratio: another factor is putting it on and taking it off. A big jacket is just easy to pull on and off, if it is very fitted then taking it off at 40-50km/h gets harder.

    Larry T: there’s some evidence that suggests people who buy cars given them higher reviews. It’s because they’ve spent so much that they don’t want to admit they bought a bad car in case it makes them look stupid. Only if I could buy things to review, eg buy 20 rain jackets, could my testing be independent. And that I’m probably not going to do.

    Starr: there’s something in that but I’ll try and explain a bit more. Remember many guys today still take newspaper at the top of a climb. All those modern fabrics and design and the world’s best are deploying paper sheets.

    To everyone else, yes I won’t be doing many reviews but sometimes just like to start writing a piece on here as a way to think through a subject. I first thought riding in a jacket and writing about it couldn’t be easier but it’s not so straightforward.

  10. The level of concern you have for doing a proper review of a rain jacket is impressive. I can not imagine the mental gymnastics that you would have to go through to determine how to do an honest unbiased review of a complete bicycle.
    I am sure your readers are familiar with the standard review that describes bike x as “horizontally rigid yet vertically compliant”. The reality of choosing a bike is the same as choosing a nice suit, that is: how much money do you have to spend and which colour do you like? Silly reviews in magazines showing the bike climbing the Poggio or ripping down a descent of Mt. Baldy are just there to sell copy and generate revenue.
    An discussion of the format and tone of bicycle and related product reviews across the current media landscape is more interesting than the reviews, but I do like looking at the photos in the reviews.

  11. Twisted Spoke – “Even big magazines and test sites have “relationships” with companies that supply them. Plenty of news about companies playing people to write positive reviews on Amazon. As always, let the buyer beware. ”

    User reviews on Amazon/Wiggle etc can be taken with a pinch of salt because you know next to nothing about the reviewer. At least on mags and websites you can usually be pretty sure that the reviewer is a genuine bike geek. I’d also be interested to hear any actual, solid, evidence of this stuff going on in bike kit land (Inner Ring, an investigative piece perhaps?). People are expensive, even if they are only students throwing out +tive reviews on Amazon, and accusations of reviews being bought are cheap. As for “relationships” that’s a pretty grand word for a bunch of distributor contacts.

    “I don’t write negative reviews because I don’t test junk that I don’t want in the first place. ”
    How can you be objective if you’ve already made a decision on the product before you review it?

  12. @Rob Simmonds
    True it is almost impossible. And I don’t think it’s a thing to be striving for. I don’t think we should go to extreme lengths to make reviews unbiased. Rather we should treat them as what they are, and appreciate that. As long as you know what you are getting, the reader can make their own assumptions.

    I think bloggers tend to be good at that. They often put a disclaimer in their post, or say something like they have no connection to the company or similar. Reviews can be as biased as anything and still provide valid infomration, as long as it’s stated up front.

  13. It would help if you already have a good rain jacket. Then the new jacket would have to prove itself. If it’s better than what you have, it’ll get a great review. If it isn’t better, then you know where it falls short.

  14. Your readers are happy to call you out on things they feel you’ve got wrong, and these comments form part of the review. So, not only do the reviewees get your opinion but the wisdom of the debate it sparks. And if the consensus outweighs your view you can always go back and edit your review to say “I though… but others think…”

    And, even a subjective review is helpful to lots of people. And readers soon learn which areas they feel your reviews are accurate for them and which are not. And really no one should be taking the advice of a single reviewer when buying something. Not all the responsibility is yours, the the reader must take responsibility for their choices and understand you’re offering an opinion. Caveat emptor.

  15. Before I bought my current bike, a Giant TCR, I looked at reviews about the bike, liked what I had read, and bought the bike! Within a month or two I had put the bike throught ride/race tests that had been described in the reviews, and found myself ‘confirming’ what the reviewers had said! Whether the reviewers had described the bikes qualities accurately, or I had simply assimiliated their impressions and replicated them, does not matter to me! I still love the bike! Maybe I have a propensity for being impressed indefinitely and am still being influenced by their reviews after 18 months!!! The power of suggestion, in this case, has led to positive results, which is what I was looking for! If the review is a means to an end then it has achieved its purpose in this case! There are other times, however, when I have made a choice based on a review and later regretted it! Maybe in the end it will all balance out (reviews that helped vs reviews that didnt)! After all, even scientists know that objectivity is more a concept than a reality (observer effect, for example), and we all have a basket of complex filters influencing our experience of ‘reality’! At the end of the day, or ride/race, it all depends on how you feel about the experience, in your gut, not in your head! And each day is different!!!

  16. Regarding the freebie dilemma, have the company lend you the garment rather than give it to you. When you’re done with it you can return it and let them donate it somewhere.

  17. I like zurich07’s idea, but will add to it. You can conclude your review with ‘happy to return it’, or go the Victor Kiam and be ‘so happy that I bought it’ (with your own hard-earned).

  18. Consumer Reports BUYS all the products they test. If everything is paid for credibility is improved though the “give it back when you’re done” angle might work too. I’m well aware of the concept of those who pay a lot of dough for something quite often are defensive about their choice but that’s NOT a review, rather it’s a purchase justification. Overall, the entire review concept is fraught with conflict-of-interest issues and I’d rather have the ‘ring stay out of them. There are already thousands of review sites available, do we really need more? I’d rather keep the focus more on the DOING rather than the BUYING — too much of cycling these days is a “competitive shopping” exercise anyway.

  19. Larry T – “Consumer Reports BUYS all the products they test. If everything is paid for credibility is improved though the “give it back when you’re done” angle might work too.”

    Plenty of our stuff goes back already and if we had to buy all of our test kit – bingo, no reviews because we simply don’t have the money. Credibility comes from writing honest and detailed reviews, not from chest beating about how you paid for the kit yourself.

  20. Rob – I guess the “give it back when you’re done” angle CAN work then? Looked at your website and since you guys have plenty of reviews there I see no reason for the ‘ring to get involved in this type of thing. If I care what you guys think about a product I can click over there, otherwise stay here for posts about doing things rather than more stuff to buy. Thanks!
    PS–ALL of your stuff should go back to the source if credibility is important. Perhaps after the test period you purchase the products you truly like at a price break, since they are then used, test-models. This issue is of special importance to me as a bike tour operator. Reviewers getting free tours from our competitors are extremely unlikely to say anything less-than-glowing about a free vacation, right? If they say bad things about the tour it won’t be long before they don’t get invited to join any in the future. But rarely (if ever) do these reviewers mention the fact that their tour was entirely or mostly “comp’d” in exchange for the review/publicity. Beyond this, we’ve even been advised the potential reviewer required his own personal bar tab be picked up each evening by us! I hope you can understand the source of my skepticism about journo reviews if the products they’re reviewing are essentially GIFTS?

  21. I understand furnishing reviews can be difficult, and some have the knack for it whilst others should resist going down that road, but an “Editor’s Pick” stamp of authority for a product by a highly-respected voice or magazine in the industry, is something I tend to trust and indeed rely upon when looking to purchase new products. More than a product maker’s review though, I highly value the reviews of my peers in the comments section of a product’s write-up, so your “simple solutions” of asking friends holds good sway, imo.

    “Finally, for all the examination and introspection let’s note Hincapie Sports presumably sent the gear here to get it featured on screen. They’re not after an analysis of their kit in the context of bias.”

    Yes INRNG, perhaps you’re right, you certainly won’t know for sure unless you asked; but how shrewd that they would have done so in any case – says a lot about their thinking and time management, and if I was in business I would value this kind of aim from such a personable brand name. In any case, what I tend to value in this particular blog, is how you are diligently working these things out for yourself, it’s like you are teaching yourself ethics, very impressive!

    Cheers,
    Daniel Moszkowicz. (TKofC)

  22. @Larry T

    Ah, now I’m starting to understand why you have such a beef about this. I don’t think you can compare kit reviews with freebie holiday/tour packages. Kit reviews are essentially a production line but (on our site at least) tour packages are rare and get written up as articles, and the source of the freebie is always acknowledged. I can’t answer for your experience, but it’s unfair to simply assume that *all* reviews are essentially corrupt.

    As for your point that all review items should be returned if we want to be credible, I simply don’t agree. Am I supposed to return the lube I’ve spent the last few months testing? What about the bike wash and brush sets that I’ve reviewed? Is anyone going to want them back? Am I really going to write up a biased review because I didn’t pay for those electrolyte tablets? No-one is going to want a pair of well worn shorts returned, so should I then have to pay for them? Trust me, being a reviewer doesn’t mean that I treat every item that comes my way as a special treat that deserves a favour in return. If anything it’s the opposite, when yet another multi-tool appears or another bottle of lube or some more panniers my response is more likely to be ‘come on then, impress me’ rather than ‘free stuff! I will write nice things’. Of course that in itself is a form of bias, but wouldn’t you prefer that a reviewer gave an item a hard time? Our reviews are credible because they are written by and for bike geeks. If we returned everything we tested then the reviews wouldn’t change one bit.

  23. Appreciate the mental gymnastics, but I wouldn’t be too circumspect. B/c you have a blog that gets attention, you get free stuff to try. You review it, and given that it comes from a quality manufacturer, will probably like it. You mention some drawbacks to the product if you notice any, and it’s done. Everybody wins; perfectly symbiotic. And the readers know what’s going on, so nobody is hurt.

    I’d be concerned if you gave glowing reviews to crap gear from a crap manufacturer with the plan to maximize your haul of schwag. But this group of readers would hang you for it.

  24. Rob- rather than bore the others with this you might want to email me larry@cycleitalia.com. Obviously a bottle of chain lube’s not going to be returned, don’t be silly. But I’ve been around the bike and publishing biz far too long to believe much of what I read in reviews, period. Except for Consumer Reports which accepts no advertising and purchases the products at retail from normal outlets that do not know Consumer Reports is the purchaser. And bike tour reviews almost never mention the fact that the reviewer received the vacation package at no or low cost. These are the reasons I think the ‘ring should stay out of it, it doesn’t take too long before advertorial can creep in and once your integrity and credibility are gone…so are a lot of your readers. And for what? A free jacket or pair of shorts? That’s a fool’s bargain.

  25. Don’t angst over it so much. All blogs have bias – yours is no different. We readers know the score when we come here. As a reviewer, your job is simply to give your personal opinion of the product. As long as you are upfront and honest in your views, people will appreciate it. And if they disagree, they can do so in the comments.

  26. The relationship between the cycling media and the manufacturers is synergistic. The media bring us timely information informed by their knowledge and experience of the market. However, the public must understand the dynamics involved.

    Take a look at this behind the scenes video of the 2010 Trek Madone Launch: http://www.youtube.com/watch?v=V2JokAOz6Ew

    After a few months – or years – of such grand treatment, it becomes very difficult to bite the hand that feeds you.

  27. Its your blog(and its a great one) but don’t fool yourself that reviews would add any value for the readers, no matter how hard you try to be unbiased.. for me they would be the necessary evil at best.

Comments are closed.