A t-test tells you whether the difference between two group averages is likely real or just due to random chance. It produces a p-value that quantifies how surprising your results would be if there were actually no difference between the groups. If that p-value falls below a chosen threshold (usually 0.05), you conclude the difference is statistically significant.
That’s the short answer, but understanding what the t-test actually does under the hood, what its numbers mean, and where it can mislead you will help you read research results (or run your own analysis) with much more confidence.
How the T-Test Works
At its core, a t-test compares a signal to noise. The “signal” is the difference between the averages you’re comparing. The “noise” is the variability in your data, captured by something called the standard error. The test divides the difference by the standard error to produce a single number: the t-value.
A larger t-value means the difference between groups is big relative to the scatter in your data. A t-value near zero means any difference you observed is small compared to how much the individual measurements bounce around. Think of it like trying to hear someone talking at a concert: the t-value tells you how loud the voice (signal) is compared to the background music (noise).
The test then converts that t-value into a p-value using a probability distribution. The p-value answers one specific question: if the two groups were actually identical (the “null hypothesis”), how often would you see a difference this large just from the randomness of sampling? A p-value of 0.03, for instance, means there’s only a 3% chance you’d get results this extreme if there were truly no difference.
What the P-Value Does and Doesn’t Tell You
Most researchers use 0.05 as the cutoff for significance, though 0.01 and 0.10 are also common depending on the field. If your p-value is below the cutoff, you reject the null hypothesis and conclude the difference is statistically significant. If it’s above the cutoff, you can’t rule out that the difference is just noise.
Here’s where people get tripped up. A significant p-value does not prove the groups are different. It tells you the data is inconsistent with the assumption that they’re the same, which is a subtle but important distinction. And a non-significant p-value doesn’t prove the groups are identical. It just means you didn’t find enough evidence to say otherwise, possibly because your sample was too small.
The p-value is also heavily influenced by sample size. With a large enough sample, even a tiny, meaningless difference between groups can produce a significant p-value. This is why statisticians stress that you should never rely on the p-value alone.
Why Effect Size Matters More Than You Think
The t-test tells you whether a difference exists, but not whether that difference is meaningful. That’s where effect size comes in. The most common measure for t-tests is Cohen’s d, which expresses the difference between group means in terms of standard deviations. A d of 0.2 is considered small, 0.5 is medium, and 0.8 or above is large.
A medium effect of 0.5, as the psychologist Jacob Cohen put it, is “visible to the naked eye of a careful observer.” A small effect of 0.2 is noticeably smaller but not trivial. To put it practically: if you’re testing whether a new teaching method improves test scores and you get a significant p-value but a Cohen’s d of 0.1, the method technically works but the improvement is so small it may not be worth the cost of implementing it. Both the p-value and the effect size are essential for understanding what your results actually mean.
Three Types of T-Tests
Which t-test you use depends on your data setup.
- One-sample t-test: Compares the average of a single group to a known or fixed value. For example, testing whether the average body temperature of your sample differs from 98.6°F.
- Independent (unpaired) t-test: Compares the averages of two separate groups that have no connection to each other. For example, comparing test scores between students who used method A versus students who used method B.
- Paired t-test: Compares two measurements taken from the same subjects. For example, measuring patients’ blood pressure before and after a treatment. Because the same people appear in both measurements, the test accounts for individual variation, making it more sensitive to real differences.
One-Tailed vs. Two-Tailed Tests
A two-tailed test checks whether the groups differ in either direction. It asks: “Is group A different from group B?” without specifying which one is higher. This is the default in most statistical software, and it splits your significance threshold evenly between both ends of the distribution. At a 0.05 level, that means 0.025 in each tail.
A one-tailed test checks for a difference in only one direction. It asks something like: “Is the new drug less effective than the current one?” This gives you more statistical power to detect an effect in that specific direction, because the entire 0.05 threshold is concentrated on one side. However, you completely ignore the possibility of an effect in the opposite direction. If there’s any chance a difference in the untested direction would matter to you, stick with two-tailed. Choosing a one-tailed test just because your two-tailed test didn’t reach significance is considered a misuse of the method.
Assumptions Your Data Needs to Meet
The t-test isn’t valid for all data. It requires several conditions to produce trustworthy results:
- Normal distribution: The data in each group should follow a roughly bell-shaped curve. This matters most with small samples. As your sample grows larger (generally above 30), the test becomes more forgiving of non-normal data.
- Equal variance: The spread of data in both groups should be roughly similar. When sample sizes differ between groups, unequal variance becomes an even bigger problem.
- Independence: Each observation should be independent of the others. One person’s result shouldn’t influence another’s. (The exception is the paired t-test, where dependence between paired observations is the whole point.)
- Continuous measurement: Your data should be measured on a scale with meaningful intervals, like height, weight, or test scores. The t-test doesn’t work for categories or rankings.
When these assumptions are violated, particularly with small samples, the results can be misleading. If your sample has fewer than 15 observations and the data is clearly skewed or contains outliers, a nonparametric alternative (like the Mann-Whitney U test) is a safer choice. Even with moderate sample sizes of 15 or more, severe outliers can distort the t-test enough to warrant switching methods.
Reading T-Test Results in Practice
When you see t-test results reported in a study, you’ll typically encounter three numbers: the t-value, the degrees of freedom, and the p-value. They’re often written in a compact format like t(48) = 2.31, p = 0.025. The number in parentheses (48) is the degrees of freedom, which roughly reflects your sample size minus the number of groups. Smaller degrees of freedom mean the test requires a larger t-value to reach significance, which is the test’s built-in correction for the extra uncertainty that comes with small samples.
Many studies also report a confidence interval for the difference between means. A 95% confidence interval gives you a range of plausible values for the true difference. If you’re comparing two groups and the confidence interval is 2.1 to 8.7, you can say you’re 95% confident the real difference falls somewhere in that range. If the interval includes zero, the difference isn’t significant at the 0.05 level, because zero (no difference) is still a plausible value. Confidence intervals are often more useful than p-values because they tell you not just whether a difference exists, but how big it plausibly is.
Common Misuses to Watch For
The t-test compares two groups. If a study compares three or more groups by running multiple t-tests (group A vs. B, A vs. C, B vs. C), the chance of a false positive inflates with each additional comparison. An ANOVA is the appropriate test for three or more groups.
Another common pitfall is ignoring effect size entirely and treating a significant p-value as proof that something is important. A study with 10,000 participants might find a statistically significant difference of half a point on a 100-point scale. That’s real in a statistical sense but irrelevant in a practical sense. Whenever you see a significant t-test result, look for the effect size or confidence interval to judge whether the finding actually matters.