How 15 People and a Chocolate Bar Fooled the World’s Media

In spring 2015, newspapers around the world reported that eating chocolate helps you lose weight. The Daily Mail ran it. Germany’s Bild ran it. The claim came from a published study. There was little reason to doubt it. But the study was designed to mislead.

The study itself was real. Science journalist John Bohannon and a team of German filmmakers recruited 16 people. They split them into three groups. One was a control. One went on a low-carb diet. One went low-carb and also ate 1.5 ounces of dark chocolate a day (Bohannon 2015). The chocolate group lost weight faster. The result was “statistically significant.” Bohannon published it in the International Archives of Medicine under the name “Johannes Bohannon” (Bohannon et al. 2015).

The flaw was in the design. The team measured 18 different things. Weight, cholesterol, sleep quality, blood proteins, and more. All across just 15 people, since one subject dropped out. A short definition may help here. A p-value is the chance a result happened by luck. Scientists call a result “significant” when p is under 0.05, meaning under a 5 percent chance it is luck. With that many measurements on that few people, the team had about a 60 percent chance of finding something “significant” purely by luck (Bohannon 2015). This is called p-hacking. Run enough tests and one of them will cross the line by chance alone. It is the same problem behind the replication crisis. Ioannidis warned years earlier that most published findings in fast-moving fields turn out false, mostly because of small samples and flexible analysis (Ioannidis 2005).

But the statistics are only half of the story. The other half is science communication. The journal charged the authors around $650. It put the paper online within two weeks. There was almost no peer review (Bohannon 2015). Bohannon then wrote a press release that left out every problem with the study. Not one outlet that covered the story asked a difficult question. The pipeline from study to headline had no step where anyone actually read the paper.

This is where the two course topics connect. A p-hacked result on its own is only a weak paper in an obscure journal. Uncritical reporting on its own is only poor journalism. Together, they led millions of people to believe chocolate causes weight loss. That is what happened. Neither problem alone would have mattered much.

The solution is simple. Don’t only read the headline but also the study itself. Find out how many people were tested and how many outcomes were actually measured. Also keep in mind that “published” does not mean “true” but could simply mean the fee was paid.

References:

Bohannon, Johannes, Diana Koch, Peter Homm, and Alexander Driehaus. 2015. ‘Chocolate with High Cocoa Content as a Weight-Loss Accelerator’. International Archives of Medicine 8 (December). https://doi.org/10.3823/1654.

Bohannon, John. 2015. ‘I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How.’ Gizmodo, May 27. https://gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800.

Godoy, Maria. 2015. ‘Why A Journalist Scammed The Media Into Spreading Bad Chocolate Science’. Food For Thought. NPR, May 28. https://www.npr.org/sections/thesalt/2015/05/28/410313446/why-a-journalist-scammed-the-media-into-spreading-bad-chocolate-science.

Ioannidis, John P. A. 2005. ‘Why Most Published Research Findings Are False’. PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.