A friend claims to be psychic and correctly calls five coin tosses in a row. If he were only guessing, this would happen 1 in 32 times (3.1%). Since this fall under the typical p < 0.05 threshold, the result is “significant”, but would anyone believe he possesses psychic abilities? If he tried to convince a particle physicist, five calls would not come close. Physicists would want about 22 in a row before claiming a discovery (Lyons 2023), so why do different sciences set different standards for “surprising enough”?
The p value itself is defined using a null hypothesis, a scenario where nothing interesting happens. The p value is the probability of data at least this extreme if the null hypothesis is true, and this resulting nuanced definition is often misinterpreted. While the p value assumes the chance and questions the data, it does not give the likelihood that results are due to chance. Reversing a probability changes it, just as most pro basketball players are tall, but most tall individuals are not pro basketball players. Hence, the p value cannot determine the likelihood that a hypothesis is true (Wasserstein and Lazar 2016). It also cannot account for the number of tests run or the plausibility of the claim, as genome studies and particle physics show.
Genome wide studies compare DNA from individuals with and without a trait, such as diabetes. Each variant (a spot where people’s DNA spelling differs) is assigned its own significance test, totalling up to roughly a million independent tests (Pe’er et al. 2008). Although most variants have no link to the trait, a threshold of p < 0.05 would still produce roughly 50 000 false hits by chance. To mitigate this, the field responded by dividing the cutoff by the number of tests: 0.05 / 1 000 000 = 5 x 10⁻⁸ (Kanai, Tanaka, and Okada 2016). This reflects the first limitation, as the same p value weighs much less when it is only one of a million tests.
Particle physics follows uses a five-sigma standard, where results must lie at least five standard deviations from what the null hypothesis predicts (p ≈ 3 × 10⁻⁷). This mitigates against implausible claims and experimental errors that can deviate ordinary data to produce dramatic results (Lyons 2023). In 2011, OPERA timed neutrinos (nearly massless subatomic particles) travelling 730 km and found they arrived about 60 nanoseconds earlier than light, at six sigma, way beyond the five-sigma standard (OPERA Collaboration 2011). However, the result was absurdly implausible, contradicting relativity and previous supernova observations that had capped any neutrino speed difference at under a ten-thousandth of this size. The anomaly was later traced largely to a faulty fibre-optic connection (Physics World 2012). The p value only measures chance; it cannot assess whether a hypothesis is reasonable or account for experimental errors.
Genome studies tightened their threshold to account for a massive scale testing, while particle physics retains its cutoffs to mitigate implausible claims and hidden errors. A p value threshold is not an arbitrary boundary but a carefully derived value reflecting the risks a field’s questions face. It is easy to digest “statistically significant” as is, but readers should invest more time into questioning how many tests were run and how plausible the hypothesis was before jumping to a conclusion.
References
Kanai, Masahiro, Toshihiro Tanaka, and Yukinori Okada. 2016. “Empirical Estimation of Genome-Wide Significance Thresholds Based on the 1000 Genomes Project Data Set.” Journal of Human Genetics 61 (10): 861–66. https://doi.org/10.1038/jhg.2016.72.
Lyons, Louis. 2023. “Five Sigma Revisited.” CERN Courier, July 3, 2023. https://cerncourier.com/a/five-sigma-revisited/
OPERA Collaboration. 2011. “Measurement of the Neutrino Velocity with the OPERA Detector in the CNGS Beam.” arXiv:1109.4897v1. https://doi.org/10.48550/arXiv.1109.4897
Pe’er, Itsik, Roman Yelensky, David Altshuler, and Mark J. Daly. 2008. “Estimation of the Multiple Testing Burden for Genomewide Association Studies of Nearly All Common Variants.” Genetic Epidemiology 32 (4): 381–85. https://doi.org/10.1002/gepi.20303.
Physics World. 2012. “Faulty Fibre May Explain Faster-Than-Light Neutrinos.” February 2012. https://physicsworld.com/?p=16957.
Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA’s Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108.
Leave a Reply
You must be logged in to post a comment.