The significance level is a threshold you set before running a statistical test that defines how much risk of a wrong conclusion you’re willing to accept. Expressed as a probability (most commonly 0.05, or 5%), it represents the chance of concluding that an effect exists when it actually doesn’t. If your test produces a p-value smaller than this threshold, the result is considered statistically significant.
How the Significance Level Works
In hypothesis testing, you start with a “null hypothesis,” which is essentially the default assumption that nothing interesting is happening. A new drug has no effect. A teaching method doesn’t improve test scores. The significance level, represented by the Greek letter alpha (α), is the probability you’re willing to tolerate of incorrectly rejecting that default assumption.
Setting alpha at 0.05 means you’re accepting a 5% chance of being wrong when you declare a result significant. If you need to be more cautious, you might set it at 0.01 (1%) or even 0.001 (0.1%). The key point is that this threshold gets locked in before you look at the data, not after. It’s a decision about how much uncertainty you can live with, made in advance so the results don’t influence the standard you hold them to.
Alpha vs. P-Value
People frequently confuse the significance level with the p-value, but they play different roles. The significance level is your predetermined line in the sand. The p-value is what the data actually produce once you run the test. If the p-value falls below alpha, the result crosses your threshold and counts as statistically significant.
A p-value of 0.03, for example, means that if the null hypothesis were true (if there really were no effect), you’d see results this extreme in only 3 out of 100 random samples. It does not mean there’s a 3% chance the treatment doesn’t work. That’s a common misinterpretation. The p-value is a statement about the data given a specific assumption, not a statement about the probability that your hypothesis is correct.
Researchers should report exact p-values (like p = 0.034) rather than just stating whether a result cleared the threshold. The precise number gives readers more information to work with than a simple “significant” or “not significant” label.
Why 0.05 Is the Standard
The 0.05 threshold traces back to the statistician Ronald Fisher and has been the default in most scientific fields for decades. It struck a practical balance: strict enough to filter out many false alarms, lenient enough to detect real effects without requiring enormous studies. But there’s nothing magical about it. It’s a convention, not a law of nature.
A growing number of researchers argue the bar should be higher. A 2017 paper with 72 co-authors proposed shifting the default to 0.005, arguing this would substantially reduce the rate of false positives that have contributed to a reproducibility crisis across several scientific fields. Others have suggested 0.01 or 0.001 as alternatives. The debate is ongoing, but the trend is toward more stringent thresholds in fields where false discoveries carry serious consequences.
Regulatory bodies sometimes use different standards entirely. The FDA, for instance, requires clinical trials for new drugs to demonstrate significance at the 2.5% level or below, reflecting the higher stakes of approving a medication that might not actually work.
The Connection to False Positives
The significance level directly controls your risk of a Type I error, which is a false positive. This is the mistake of concluding something is real when it isn’t. If you set alpha at 0.05, you’re allowing a 5% probability of this kind of error across your testing procedure.
There’s a tradeoff involved. Lowering alpha to reduce false positives simultaneously increases your risk of the opposite mistake: a Type II error, or false negative. That’s when a real effect exists but your test fails to detect it. A study with alpha set at 0.001 is very unlikely to produce a false alarm, but it’s also more likely to miss a genuine finding, especially if the sample size is small or the effect is subtle.
This tradeoff is why researchers conduct power analyses before running a study. Statistical power is the probability of correctly detecting a real effect, and it depends on three things working together: the significance level, the sample size, and the size of the effect you’re trying to detect. Choosing a stricter alpha means you typically need a larger sample to maintain adequate power.
Confidence Levels and Alpha
If you’ve encountered confidence intervals (like “95% confidence interval”), you’ve already seen the significance level from a different angle. The confidence level and alpha are complementary: a 95% confidence level corresponds to an alpha of 0.05, because together they account for 100% of the probability. A 99% confidence interval corresponds to alpha = 0.01. The two concepts are simply different ways of framing the same tolerance for uncertainty.
Statistical Significance vs. Real-World Importance
Clearing the significance threshold tells you a result is unlikely to be pure noise. It doesn’t tell you the result matters in practice. This distinction between statistical significance and clinical (or practical) significance trips up even experienced researchers.
Consider two cancer drugs tested in separate trials, both producing p-values of 0.01 against an alpha of 0.005. Both are statistically significant. But Drug A extends survival by five years while Drug B extends it by five months. The statistical test can’t distinguish between these two scenarios. It only tells you that both effects are unlikely to be zero. Whether either effect justifies the cost, side effects, and disruption of treatment is a completely different question.
The American Statistical Association addressed this in a formal 2016 statement, emphasizing that a p-value alone “does not provide a good measure of evidence regarding a model or hypothesis.” Data analysis shouldn’t end with a significance test. Effect sizes, confidence intervals, study design, and the broader context of existing evidence all matter. A tiny but statistically significant difference in a massive study might be meaningless in practice, while a large but non-significant difference in a small study might hint at something worth investigating further.
Choosing the Right Alpha for Your Situation
The best significance level depends on the consequences of being wrong. In exploratory research where you’re generating ideas for future testing, 0.05 is often reasonable. In confirmatory studies where decisions hinge on the result, such as drug approvals or policy changes, a stricter threshold like 0.01 or 0.005 makes more sense. Fields like particle physics use an alpha equivalent to roughly 0.0000003 (the “5-sigma” standard) before claiming a discovery, because falsely announcing a new particle would be a major scientific embarrassment.
The significance level is ultimately a risk management tool. Setting it requires weighing the cost of a false positive against the cost of missing a real effect, then picking the threshold that best fits the stakes involved.