Regular readers will know I don’t usually feature product reviews on here. I did a book review last week but that’s about it and I’m not sure I’ll be doing many more. It’s much harder than I thought.
As background, I’ve been on a quest for a rain jacket. You’ve probably seen the pictures of riders with deep section wheels and frames designed in a wind-tunnel. Yet the aerodynamic benefits dissolve in the rain. Because as soon as there’s a downpour riders reach for their rain bags and out comes the jacket. This billows and flaps, a visual and sometimes even audible indication of just how un-aero this garment can be. Obviously the need to stay relatively dry wins but some jackets balloon up big time. It’s not like rain is a freak event and I’ve been trying to find a jacket that doesn’t slow you in a race or a ride.
Hincapie Sports must have read something about my jacket quest and got in touch, asking if I’d be interested in testing their rain jackets especially since they claimed to have the answer. I was interested in the product but also thought it would be an interesting exercise to try and review something. I’ve even put a piece on the blog asking for what readers looked for in a review. There were some helpful comments and some of you even said “don’t do it” as this blog isn’t really about tech reviews which I understand.
But the whole process of review is much harder than I imagined. I thought I’d go for a few rides and then write about the product but it’s just not that simple.
- It’s one thing to ride with something but another to write a critical assessment. Since I know others will be spending money on the product there’s a duty of responsibility. I try not to be careless on the blog but if I make a mistake about the altitude of a col or the history of a race then you won’t be dropping money on this: a review is different. It feels like every detail has to be covered.
- Next there’s the issue of bias. I’ve covered angles on this before but this time when a box arrives with several pieces of gear to use then I feel flattered. Even if I am aware of this and point it out to you, plenty of evidence suggests experts get swayed in reviews and I’m only a novice reviewer.
Note the tension between these two points. One side of my brain feels a duty to the reader to highlight every detail from a sceptical point of view; another is saying “aren’t you compromised by the free gear?”
I’d been chatting on Twitter about reviews and discovered this morning that the cyclingtips blog has covered things very well. I’d just add that if some say “those magazines have ads to sell” then in some cases the person doing the tech reviews is distant from the sales or accounting people. But here I’m writer, editor, tech reviewer, sales and accounting all in one meaning I’m potentially less independent. But if I started shill reviews you’d call it in the comments.
One idea is to make things clear up front that if I review a rainjacket then I got it free or if I got a book then I paid for it. But even this isn’t so obvious. Studies show that people who buy items rate them higher, obviously the act of buying something suggests you already rate it so highly that you want to hand over money for it.
In fact the more you look into the subject, the harder it gets. As humans we’re all prone to superstition, biased, confusion and other psychological frailties. It makes life interesting but neutrality is hard. To avoid this some magazines test a frame in the lab to measure flex under load, a good idea… but there are tales of bikes being designed to score high in the EFBe test used by some magazines just as cars can be designed to meet the exercises imposed by regulatory crash tests ahead of offering all round protection. And to revert to the original idea of a rain jacket, what if it fits the reviewer but you’re a different shape?
Enough ifs and buts. Reviews help and more reviews help more so read as many as possible and ask cycling friends for their views before you spend big. Better still, be instinctive because if there’s any bull in a review you can probably smell it.
Finally, for all the examination and introspection let’s note Hincapie Sports presumably sent the gear here to get it featured on screen. They’re not after an analysis of their kit in the context of bias.
Writing a review is a lot harder than I thought it would be. You might be able to describe how your bike rides but turning this into a critical and independent evaluation is something else. The more you examine the methodology and psychology of the review process the more fraught it becomes. But all the more reason to keep a review simple and to give a personal take on the experience, to say what I got from it rather than extrapolate what you’re going to like.
I don’t think I’ll be doing many reviews on here but will follow up with the Hincapie gear reviews soon. After a long period of autumn sunshine the weather has been miserable recently, ideal conditions to test rain jackets. Perhaps riding down a mountain in a downpour is easier than evaluating bike kit?