A p-value is considered statistically significant when it falls below a predetermined threshold, most commonly 0.05. This means there’s less than a 5% probability that the observed results would occur if there were truly no effect or no difference between the groups being compared. The 0.05 cutoff is the default in most scientific research, but it’s not a magic number, and understanding what it actually tells you (and what it doesn’t) matters more than memorizing the threshold.
What a P-Value Actually Measures
A p-value measures how compatible your data are with the assumption that nothing interesting is happening. More precisely, it’s the probability of seeing results as extreme as (or more extreme than) what was observed, assuming the “null hypothesis” is true. The null hypothesis is simply the default position that there’s no real effect or difference.
Say a study compares a new drug to a placebo and finds that patients in the drug group improved more. The p-value answers this question: if the drug actually did nothing, how likely would we be to see a difference this large just from random variation in the sample? A p-value of 0.03 means there’s a 3% chance of getting results this extreme by chance alone. Because 3% is below the 0.05 threshold, the result would be called statistically significant.
Where the 0.05 Threshold Came From
The 0.05 standard traces back to the British statistician Ronald Fisher, who wrote that a probability of 1 in 20 was “convenient to take as a limit in judging whether a deviation is to be considered significant or not.” Fisher noted that this corresponded to roughly two standard deviations from the mean, which made the math clean. The convention stuck, and by the 1950s it had become the dominant standard across biomedical and social science research.
It’s worth knowing that 0.05 was never meant to be a rigid, universal rule. Researchers can and do set their significance level at 1% (0.01) or 10% (0.10) depending on the context. In particle physics, the threshold for claiming a discovery is roughly 0.0000003 (one in 3.5 million). The choice depends on how costly a false alarm would be.
The Connection to False Positives
The significance threshold you choose is directly tied to your false positive rate. A false positive, called a Type I error, happens when you conclude there’s a real effect but there actually isn’t one. If you set your significance level at 0.05, you’re accepting a 5% chance of making this mistake when the null hypothesis is true. Set it at 0.01, and that risk drops to 1%.
This is why some researchers have pushed to lower the standard threshold from 0.05 to 0.005. A 2018 proposal backed by dozens of prominent statisticians argued that the evidence standards for claiming new scientific discoveries are “simply too low.” They pointed out that a p-value of 0.05 corresponds to only weak evidence in favor of a real effect when evaluated using alternative statistical frameworks. A p-value of 0.005, by contrast, falls in the range that most statisticians would consider strong or substantial evidence.
What a P-Value Does Not Tell You
This is where most confusion lives. A significant p-value does not tell you the probability that your hypothesis is true. It does not tell you the probability that your results happened by chance. And it does not tell you whether the effect you found is large, important, or meaningful in real life. The American Statistical Association released an unusual formal statement in 2016 spelling out these exact points, warning that “a p-value does not measure the size of an effect or the importance of a result.”
The distinction between statistical significance and practical significance is one of the most important concepts in interpreting research. With a large enough sample, a study will almost always produce a significant p-value for even trivially small differences. Imagine a weight loss study with 50,000 participants that finds people on a new diet lost 0.2 pounds more than the control group. The p-value might be 0.001, highly significant, but losing a fifth of a pound is meaningless in practice. As one widely cited paper in the Journal of Graduate Medical Education put it: “Statistical significance is the least interesting thing about the results.”
Effect Size Fills the Gap
Because p-values can’t tell you how big or meaningful an effect is, researchers also report effect sizes. An effect size quantifies the magnitude of the difference between groups, independent of sample size. A large study and a small study testing the same treatment should produce similar effect sizes if the treatment works, but the larger study will almost always produce a smaller p-value simply because it has more data.
Think of it this way: the p-value tells you whether an effect likely exists, and the effect size tells you whether it matters. Both pieces of information are essential. A study that reports only a p-value is giving you an incomplete picture.
Confidence Intervals Add Even More Context
Another tool that complements p-values is the confidence interval. While a p-value gives you a single number, a confidence interval gives you a range of plausible values for the true effect. A 95% confidence interval that runs from 2 to 15 tells you the effect is likely somewhere in that range, and because the entire range is above zero, the result is also statistically significant at the 0.05 level.
Confidence intervals have a practical advantage: they show the direction and strength of an effect at a glance. A p-value of 0.04 tells you the result is significant, but it doesn’t tell you whether the difference between groups was large or barely detectable. A confidence interval does both jobs at once, which is why many journals now require them alongside p-values.
P-Hacking and Why It’s a Problem
Because so much rides on crossing the 0.05 line, researchers face a temptation to nudge their results toward significance. This practice, called p-hacking or data dredging, involves running multiple analyses on the same data and selectively reporting the one that produces a significant result. Surveys indicate these behaviors are surprisingly common, even based on researchers’ own admissions.
P-hacking can take many forms: choosing which variables to analyze after seeing the data, removing outliers selectively, testing multiple subgroups, adding or dropping control variables, or gradually increasing sample size until significance appears. Each of these decisions on its own might seem reasonable, but when they’re made with the goal of getting below 0.05, they inflate the false positive rate far beyond the nominal 5%. One diagnostic sign of p-hacking in a body of research is a suspicious clustering of p-values just below 0.05, with very few results landing just above it.
How to Read a P-Value in Practice
When you encounter a p-value in a study or news article, ask yourself three questions. First, is the result below the stated significance threshold? If so, the researchers are claiming the effect is unlikely to be due to chance alone. Second, how large was the effect? A significant p-value with a tiny effect size often means the study simply had a lot of participants, not that the finding is practically important. Third, was the analysis planned in advance, or does it look like the researchers tested many possibilities and reported the one that worked?
A p-value of 0.04 and a p-value of 0.0001 are both “significant” at the 0.05 level, but they represent very different strengths of evidence. Treat a p-value near 0.05 as suggestive rather than definitive, especially if the study is small or the analysis wasn’t pre-registered. A single significant p-value from a single study is a starting point, not a final answer.