Hardly a day goes by without the media reporting the results of recent scientific research into our health and well-being. Sometimes these news items (the three examples linked below are all taken from the recent issues of New Zealand Herald) appear perfectly reasonable:
Women aged 50-plus urged to do at least five hours of exercise a week.
sometimes they sounds a bit puzzling:
Our emotions affect flavour and can dull the sweetness of sugary foods, according to a study. The research also found sour foods taste even more sour when you’re feeling down.
and sometimes downright weird :
Drinking just two glasses of orange juice a day could increase your risk of getting the deadliest form of skin cancer.
The study reported in the screenshot below falls between puzzling and weird but otherwise does not look out of the ordinary:
So what – eating chocolate can aid weight loss and has a number of other health benefits. If drinking orange juice can cause skin cancer then the chocolate study findings do not strike as particularly suspect. This, however, was not your normal scientific study but rather a sting operation. It was carried out by a journalist John Bohannon who set out to prove that it is possible to publish a paper based on seriously flawed research in scientific journals and that mainstream media will report these findings as fact.
You might think that Mr Bohannon had falsified the results the paper was based on. It would be very difficult to detect outright fraud like this but this is not what had happened. The study had really been carried out and its results reported truthfully. The paper is a joke not because Mr Bohannon lied but because the statistical methods employed were flawed and no one picked it up:
Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
So they recorded 18 different physiological measurements but only reported the ones catchy enough to make the headlines and – because of the statistical variation caused by a small study population – they were always going to get some statistically significant results to report. Very clever.
According to the publisher of the Health News Review, Gary Schwitzer:
“[John Bohannon is] really only scratching the surface of a much broader, much deeper problem,” Schwitzer says. “We have examples of journalists reporting on a study that was never done. We have news releases from medical journals, academic institutions and industry that mislead journalists, who then mislead the public.” And the pressure to publish or perish, he says, can lead well-intentioned scientists to frame their work in ways that aren’t completely accurate or balanced or supported by the facts. “We are really mired in a mess, the boundaries of which few people really have a sense for,” says Schwitzer.