How objective are bike reviews? Someone writing a review would say they know their stuff, that they can tell the difference between a good frame and a bad one, more they can get the measure of wheels, handlebars and tyres too.
Only there’s plenty of evidence from outside the cycling world that reviewing, testing and tasting is a complex matter where even the biggest experts can get confused. The next time you read a review saying “the frame felt stiff“, a host of variables come to play, for example the wheels, tyre pressure, cranks, pedals, seat post are all going to flex under pressure and isolating the frame is a complex issue. Now an expert reviewer should do a better job than most. But all the same it makes me wonder.
In vino veritas
Let’s step away from cycling and look at wine. Several studies have shown that blind tastings, where experts are not told about the wine they are drinking, achieve different results to a tasting when the drinkers are told what it is they are sampling. There have even been controlled studies involving pouring a moderate wine into the bottle of a grand cru. Put a cheap wine into a famous bottle even the experts get fooled by the label.
If fine wine tasters, with all their experience and even study, can be tricked, can us cyclists get fooled too? Would a frame labelled “Pinarello” get a better view than the same frame labelled as some anonymous Chinese producer? Evidence from the wine world says yes.
But there’s one big difference: we can measure how a bike performs, whereas taste is a more more subjective matter. French magazine Le Cycle puts frames to a simple lab test, applying weights to different areas of the frame in order to measure flex. Germany’s Tour Magazin goes further, hiring the EFBe lab to test the frames. It’s got to the point where the EFBe tests are incorporated by some manufacturers, obtaining a high score is one of the design goals.
Riding blind
One solution would be the cycling version of blind tasting. No, not blindfolding a rider but for a frame in disguise. Fit a frame with a standard set of components and then mask all the tubes, perhaps with pipe insulation. There are obstacles to masking a Cervélo P4 for review, it’s an idea that might only work on standard road frames. But until something like this is done, perhaps we should take any reviews with a pinch of salt. Or a glass of wine.
Relying on the experts
Tech reviews are incredibly important. There is so much choice of kit and the buyer is often desperate for advice. Bikes are a whole lot less subjective than wine, after all you can measure the weight in your hands. But a controlled test of frames where the components are always the same could be interesting.
Whatever works
Psychology comes into play in reviews and in real life. If you believe the bike works well, then that’s half the battle won as confidence in your machine is a big issue. Whether it’s basics like safety and relying on the brakes or knowing the frame will respond if you attack on that hill, belief that you are on the right machine matters.
you also need to consider the integrity & validity of these tests,
when they are conducted by magazines/websites with advertising to sell.
The testers themselves, are they :
1. enthusiasts who collect equipment ;
2. frustrated chimps with a new bike to build and no kit;
3. mates who wanted a free trip to Lanzarote, whilst testing the best bikes between 999 – 2499, etc;
4. objective analysts who actually know about equipment, frame geometry & design;
5. former racers, who work part time in bike shops & scribble for the associated kudos;
6. or the guy who got lumbered with the ‘test bike gig’ as its unpaid and no one else wanted it;
the tests or reviews are seldom worth much. One 5 minute blast around Mallorca, throw in a few cliche phrases “handles well” “stiff bracket transfer” “excellent for climbing” “feel the power” absolute pointless irrelevant bollocks in the mould of Clarkson Journalism.
I’ve encountered magazine bike/component tests, where the items were nicked (or bikes returned damaged – hammer dent in top tubes, purely for ‘asking; for them back) equipment never returned on the ‘unspoken’ understanding that a ‘good review’ costs the item in test….
If you want to test a bike, go along to a shop or frame builder and ask to ride whichever one that has your interest.
The most honest review you’ll find !!
Don’t know about frame reviews being biased- but the picture at the top certainly is. A Canyon leading the way? I think not!
Anonymous – damn right! Not as if Gilbert has won anything on one.
Or that they could even be a decent bike.
Cracking article. I like to think of myself as a purveyor/connoisseur or both bikes and wine. Also, I believe that these days, the price of the product is largely a function of branding (which is inferred in your last paragraph and the references you made to Pinarello by way of example).
I know I can serve you a glass from a £12 bottle of wine that will put many a fine 95+ point Bordeaux to shame. Equally, the same is true with bikes. As per my comments on the UCI approved frame article, frame technology is converging. If you do not believe this then you only have to look at the recent Canyon/Cervelo patent agreement to understand this point, or visit any decent factory in the Far East – you’d be amazed at the range of “brands” that literally roll off the same lines. However, this truth only reinforces the power of branding – get over that hurdle, and there is some amazing opportunities to be be experienced in both bikes and wine.
The one distinction that i would make between the two products, is that wine is what it is in the bottle – ie you do not dilute or mix it to experience it – just make sure you know how and when to serve it. However, a bike is very much a sum of it’s component parts. Wheels are critical for me – I can make a good frame a bad ride, and vice versa, depending on the wheels i use (and this also extend to the tyres/tubes used aswell). So, In summary I believe that not many reviewers have objective enough benchmarks (eg same/multiple wheels used) when doing reviews of multiple frames. Additionally, their independence is questionable (due to say, advertising revenues, and/or other considerations). Psychology at work again, but on a different level….
Great piece. I’ve been into cycling for over 10 years now but sometimes you make me think about the sport in new ways. Thanks, keep it up.
I find some magazine reviews of frames and components very dubious, you read a two or three page spread and then next you have a full page add from the relevant distributor of said products. Now who’s going to shoot themselves in the foot and say this thing weighs a ton and handles like an ocean liner with that sort of payment pending to the magazine ?
Also as I haven’t been in the position to receive test materials do they include a press release for the lazy journo so that if the week he has the bike it’s raining out, so all he does is rehashes the press release and bam he’s done ?
Anyway if you want a good read on how messed up testing/reviewing has gone, have a read of Chris Harris’s rant on Ferrari and their treatment of Press Cars and the Press
http://www.driver.ie/forum/showthread.php?t=88494
Makes a whole load of sense to be. Anonymity would be a very good way to do it, a huge step in the right direction.
Everyone has preconceived views / favourites with anything, whether bikes, clothing, whatever, so an unbiased review is going to be hard work.
As Flammecast so rightly points out, manby reviews are driven by advertising Euros / Pounds / Dollars and beev’s comment on using multiple wheels is essential.
TIME bicycles are my ride of choice.
flammecast – i must say i have always been rather indifferent towards chris harris before reading the link. i’m now warming to him….
Thing is, when you are moving between materials there is certainly a difference in ride characteristics. Which, like wine, can and should be written in to a review and is a variable external to that of wheel or groupset choice.
I’m not sure if I could trust any review. Even less are rider anecdotes from people who are in love with their bikes.
Though Robert Millar used to write cracking reviews of bikes.
About Canyon: They win almost all tests done by Tour magazine. Surprisingly, they also book a lot of ads in that magazine.
Anyone making a buying decision based on a review in a magazine deserves what they get. To me it’s obvious that magazines which rely mostly on advertising revenues from the cycling industry are going to struggle to print poor reviews. I don’t read them, they’re just low-quality content. Nice pictures usually though.
They are junkets at best.
I’d like to see someone write an article explaining what the cost is of making a frame and how a frame of $5000 is actually better then a frame of $500 direct from China.
I have this sneaking suspicion that we are being horribly ripped off by the carbon frame makers and I’d like to see an article that either proves or disproves this.. Nevermind the flaky reviews that just smell of marketing.
Excellent post. I agree entirely most bike reviews in the US cycling press are disguised infomercials… I wish more actual testing like Le Cycle does would be incorporated.
A good start for an unbiased review would be the disclosure of the precise site of manufacturing of the bicycle. I’ve read and heard many times that most carbon bikes came from the same manufacturers in China or Taiwan — if this is true why does no-one anyone name names? I understand that the commercial press is beholden to the companies who practice this kind of deception, but what about the Inner Ring or some other fearless blog? I’d love to know if Trek and Specialized are made in the same factory, or if Pinarello is made alongside store brands for Performance! Does anyone know facts about this rumor and is not afraid to speak? That would be a public service announcement of sorts for the average rider looking to purchase a bike.
I guess you chaps are mostly talking about high end road bikes but as someone who does test bikes, and other kit, for http://www.road.cc I can assure you that it’s always an unbiased writeup. That is to say, nothing depends on pleasing an advertiser and I’ve never been invited on an expensive junket, or a cheap one for that matter.
As far as I’m concerned a review is about treating the bike as an overall package, because that’s what is being sold, and trying to give a potential buyer as much useful info about the bike as possible.
As far as the expensive road frames are concerned I suspect that some of the comments above are confusing production cost with the entire cost of the frame. Most of the cost is likely to be in R&D, it’s not all in the factory build cost.
I read the Marcel Wurst reivews every month in Pro Cycling. He rarely has a bad word to say about anything, so you’re left the impression that all the new bikes are amazing. Well maybe they are I wouldn’t know but it doesn’t sound right
Then there is the matter of the correct sizing of the frame to the test rider. This review of a Pegoretti by a reviewer drew much hostility: http://www.velocipedesalon.com/forum/f7/bicycle-quarterly-reviews-love-3-a-15832.html
A step in the right direction, towards Le Cycle and Tour: http://velonews.competitor.com/2011/01/bikes-and-tech/velonews-rolls-out-lab-testing-for-bikes_157730
Lots of interesting comments and contributions, thanks guys.
Just one point I’d make, I’m not trying to blame junket-seeking reviews or suggest corruption, more to ask whether some brands can influence the tests, whether the prestige of a frame builder for example ensures a better review?
SOP for any magazine is to keep advertising and editorial departments separate. That doesn’t necessarily prevent interference, but I don’t think this is a particularly big problem with bike reviews. The issue really is that reviews are very subjective, and there is little, if anything, to distinguish one bicycle of the same type and price range from another, particularly at the high end. Bicycles are very simple machines, and most design issues affecting handling have been worked out. There are sometimes issues that crop up with speed wobble or other little issues, but these might not come up in a day or two of testing. There really just isn’t much of importance that a reviewer is going to be able totell you.
These days, anyway, the questions are, what is the part spec on the bike, what are the objective qualities I might be interested in, like frame weight and geometry, and is the price being asked for all of this within the range of what I am willing to pay? That’s not to say that reviews are necessarily useless, but very few are being put through a rigorous enough test to see if it really does handle confidently on 50+ mph/90 kph descents where you’re going to learn whether or not the handling is, in fact, good. And that’s to say nothing of how many race bikes are actually being tested in a race, which is where all of the intangibles will really come into play. Find a bike with the parts spec you want, at the price you want, in the color you want. A review isn’t going to take you much further than that.
I loved the reviews Chris Boardman did for Pro Cycling. He could be vicious at times. I never thought he was pulling punches to please advertisers.
What would be even more helpful from a review standpoint is to tell me this:
A) Who designed the frame. What I mean by this is: is the design proprietary to the manufacturer, or is the bike merely a sum of various design options put out by the factory (Giant, Merida, Kinesis etc…)
B) Who manufactured the bikes, and what other brands are manufactured in that facility?
C) What is the carbon lay-up, and what are the design benefits or downfalls to such a lay-up?
D) Provide independent lab scores on frame stiffness. Not just some reviewer’s feel about its acceleration.
Velonews appears to be going in this direction (at least with the lab). But their editorial decision making has to still be biased by the fact they review products from manufacturers who buy ads from them. Until cyclists are willing to pay a boatload of money for mag without ads, this will always be a problem.
What I have heard from the owners’ of a very respectable shop is that Bicycling’s reviews and picks are bought…. I am sure that some are honest and unbiased and others are not.
What I have always done is just asked my shop manager what I need. I never feel that I have been cheated in this system, but as I mentioned, it is a very respected shop. However, I also know exactly what his motives are: make a sale, keep a customer, and maintain the shop’s superb reputation. Most, if not all, of this shop’s business is done this way. I have no idea what makes a stiffer bike or if a stiffer bike is what I want, but my shop manager does.
What I would really like to see in a review is a cost benefit analysis of component groups over the long term? In my experience Campy and Shimano are much more reliable than SRAM (with proper servicing), but I would like to see quantitative data behind it. Between 7900 and Record which costs more per 1,000 miles over a 10,000 mile test. However, I doubt that this amount of honesty and objectivity would ever be seen in a review anytime soon.
Reviews are the whole grain of salt thing. Given that we all have different bodies, biology, psychological make-up and suseptibility to marketing bullshit, it’s tough to really draw any definitive help from a review. I’m always swayed by clever riding but that’s just me. However, I’d say beware: no matter how objective you think reviewers are, everybody is getting something in return.
Road.cc is a revelation in it’s honesty in terms of testing the testing of cross bikes appears to have been particularly honest a real revelation.
I have to say my latest road frame purchase has been based on user reviews of a similar product, looks, weight and craftsmanship.
In addition to brand bias, there is also the factor individual preference, as well as the suitability of a particular bike for a particular type of riding.
http://www.bikequarterly.com/vbqindex.html whose passion are randonneuring frames based on designs from the French Constructeurs of the 50s did an an interesting blind study by commissioning 3 frames of identical geometry, which were only differed by frame tubing. The frames were further disguised. The results were very interesting, although subjective.
You have two separate questions here.
1. Is it possible to determine which bike is better by just riding it? I believe the answer is Yes, provided you have ridden enough bikes and are able to see beyond the downtube sticker. When we suggested in Bicycle Quarterly that some bikes performed better because of their tubing selection, many didn’t believe us. In fact, we had doubts ourselves. We then designed the double-blind test that Touriste-Routier mentioned to see whether we really could tell the difference between frames made with tubes of different wall thickness or not. The answer was that we could. See http://janheine.wordpress.com/
Similarly, we test each bike first, then measure its geometry. These days, we get few surprises, meaning that handling can be determined in a blind test. Other things like weight and tire clearance are simple measurements, which are easy enough to do. Of course, all these things are just pieces of information – to say that one bike is better than another makes little sense, as it depends on your preferences.
2. Are journalists even trying to be objective? For the mainstream press, the answer is NO. Simply put, most magazines are supported by ads, the subscription fees are only a small part of the revenue. Whoever pays calls the shots – TOUR cannot afford to annoy Canyon. As I write for other magazines, too, I am sometimes surprised how far this goes. Once I suggested in an article on panniers that most Cordura panniers were not waterproof, and I got a call from the editor who wondered whether I could change this, as they were selling Cordura panniers in their online store!
The solution to 2. is simple: Finance the magazine by subscriptions, like Consumer Reports does. Bicycle Quarterly also is financed by readers. We can afford to annoy our biggest advertisers – we’ve lost quite a few of them. (It’s harder when we find bikes made by friends of ours to be lacking, or even bikes that we have helped design! That has happened, too, making me hope that we keep our inherent biases as small as possible.)
I think the main reason most testers like almost every bike they test is simple: That is the easiest review to write, and you won’t get much flak for it. If you say that a [insert famous Italian brand] bike made from oversize aluminum tubing rides harshly and is too stiff to perform well for the average rider, you’ll upset many of the tifosi. It is much easier to write about the “storied maker” and and throw in a few sentences on how the maker’s passion shines through as you “carve through corners”… and everybody is happy in the end.
Too many comments for me to respond individually but thanks for them all so far. It’s informative and useful.
I’ve sometimes feared the comments would get full of spam or worse, aggression, but the tone here is almost always great. Thanks!
@Rob wrote:
“As far as the expensive road frames are concerned I suspect that some of the comments above are confusing production cost with the entire cost of the frame. Most of the cost is likely to be in R&D, it’s not all in the factory build cost. ”
So if the $5000 frame really costs $200-300 to make in production costs, that’s a lotta R&D wouldn’t you say ? Those are some pretty hi-tech bikes !
I have another idea. I suspect instead of it going to R&D, it’s going to marketing. But again, an honest website that covers cycling would take a look at this and prove me or you wrong.
Of course, any website/magazine doing this is going to hit a hit in their R&D, er, I mean their advertising revenues….
J
Another angle to bear in mind when reviewing a new bike is the kind of aftersales support you’re likely to get – what kind of warranty comes with that fancy carbon frame? Will they happily replace the frame if you crash it? Some will, some won’t. You never hear about this aspect in bike reviews and I’m sure it would make buyers think twice.
@Grolby says it best. At the end of the day, most of us won’t recognize the subtle differences between frames in the same range.
Suggestions as above that reviews focus on lifetime value (see: car long-term tests) that include many riding conditions, maintenance frequency and cost, etc. would be more valuable. I also believe that wheels and tires make more of a difference than frames, and I think that the trend for everyone to have carbon rims which all have a depth within a few mm of each other would make blind wheel testing more feasible than with frames.
Newspaper live from advertising
Bike companies live from good reviews and marketing
Make 1 + 1 … 😉
Pick the bike you fell in love at first sight, don’t read any reviews!
A very interesting article, and some equally interesting feedback and comments.
Friends have often taken the p*ss out of me because I have a habit of buying frames without testing them – my response being that unless I can get the exact frame size I want, set it up with the kit I want on it with the tyres I’d plan to use and then ride it for a few days around roads I know well to give a direct comparison, how is a couple of hours on a bike that’s set up ‘not quite right’ going to tell me what it’s really like? I buy bikes that are never going to be ‘bad’ so why worry?
Likewise I’ve often seen reviewers riding bikes that are all over the shop from a sizing perspective – I just don’t see how that can result in genuine feedback.
I’ve reviewed stuff for my own site before, but that’s generally been stuff I’ve bought so my feedback has been genuine. Even so I’ll readily admit to buying some kit at a discount from a local bike clothing start-up company that I wanted to support – when the stuff wasn’t up to scratch I just didn’t post a review, and emailed them to tell them why I wasn’t going to.
I’ve recently reviewed my own bike after six solid months of riding in all conditions – sure, it’s a glowing report and it is my own bike, but at least I’ve really put it through it’s paces.
Perhaps more companies should speak to us bloggers about doing testing and prototyping?!
Interesting post and discussion, loving all the input. I think healthy cynicism in this area is a good thing.
For what it’s worth, Nick and I picked all 24 bikes we’ll be testing in the magazine this year while in driving in his car on the way to a USAC mechanic’s clinic we were covering earlier this year. No ad guys present. They live in the basement and we only go down to steal their coffee.
There is no “pay to play” here.
As a side note, our perceptions of stiffness in the latest review (April issue, aero road bikes) matched up perfectly with our torsional stiffness testing. But stiffer wasn’t always better… Nonetheless, the differences are tangible even between high-end bikes. Particularly when you ride as many as we do.
Cheers,
Caley Fretz
VeloNews tech
Here is an article from the magazine “Bycicle Guide” (Feb 1996) where
they did a blind test of seven road bycicles built by Mondonico using
then available different columbus tube sets. The bycicles had otherwise
the same components and wheels.
http://stahlrahmen-bikes.de/wp-content/uploads/Stahlrahmen_Vergleich.pdf
I won’t post a summary here, you have to read by yourself!
Next time a bike gets a good review, Pinarello or Specialized come to mind, g’head and check out how many pages of advertising that company has purchased in the magazine’s front six pages, etc.
Those are expensive pages. Those are expensive reviews. Of course they are going to be the the best, the lightest, the smoothest, the most comfortabl-est, the most lateraly stiff-est and vertically compliant-est.
If the big spenders had it their way, their bikes would also increase the size of your penis and make you a longer, harder, better lover.
It’s all SPAM.
The Bicycle Guide test mentioned by Eddi shows how not to do a test. Mondocino went out of their way to provide a large number of bikes, and then Bicycle Guide couldn’t even spare the time to have several testers ride them, multiple times!
If you ride each bike only once, and only for a short time, by the time you get to Bike 7, you barely remember how Bike 1 rode. Furthermore, it seems that the tester was worried more about how much the bike flexed than how it rode… A good bike tester shouldn’t think in terms of “secondary” attributes like flex, trail or weight, but in “primary” attributes like performance, handling and responsiveness. I don’t care whether my bike is flexible or stiff – I want it to perform well.
Thanks for this, it’s making me think quite a lot as I’m in the market for some new wheels. But reading the blog post and them all the comments only leaves me more confused. I’m close to a headache now.
I found this late but it reminds me of a story an engineer/product guy at a famous component maker told me, in confidence of course with nobody else around. He mentioned an in-house group of product testers they use. They give ’em stuff to test, sometimes telling them what’s new about it and how it’s supposedly improved and sometimes not telling them anything. He said invariably, when they were told the stuff was “stiffer” “smoother” or whatever, they always came back with glowing praise for the “improvement”. But when they were told nothing, it was VERY rare for them to actually report on increased “stiffness” “smoothness” etc.
Test riding bicycles is simply too subjective to be worth much. When the rider is also the engine there are too many variables. The VN guys try to eliminate a lot of them but it’s impossible to get rid of them all. The lab tests will certainly show which bike is stiffer, lighter, etc. but no lab test will tell you how you’ll like the response when you bend it into a corner or spend an hour climbing a steep hill on it. And they can’t tell you if the 52 cm frame they tested has qualities that will be duplicated in the 61 cm frame you buy. If it was all so easy, there would not be so many bike companies out there — the without-a-doubt, proven-by-science winner would simply drive the rivals out of business very quickly. Instead the market is driven by MARKETING whether it’s full-page ads in magazines or pro race teams using the “exact same bike you can buy” as they like to say, or reviews in the magazines. I won’t call anyone a whore or shill but think about this — how many times has the reviewer in a big (or not so big) bike mag moved on to a PR position in the industry, writing their ad and brochure text for them and schmoozing with the magazine tech editors? Do you think the industry guys are going to hire the guy who called their product a piece of crap?
“I won’t call anyone a whore or shill but think about this — how many times has the reviewer in a big (or not so big) bike mag moved on to a PR position in the industry, writing their ad and brochure text for them and schmoozing with the magazine tech editors?”
No idea. Could you tell me?
A degree of scepticism with regards to marketing claims is healthy, but cynicism is cheap, even more so on the internet. Here’s how it works at road.cc – a bike comes in, someone on the team gets to play with the bike for as long as they are allowed (usually it’s several weeks or more) then they write a review. I’ve never once had anyone, editorial or externally, try to give me a steer on a review. Not once.