t-tests and analysis of variance

What this method solves

You have a continuous outcome (birth weight, blood pressure, length of stay) and a grouping variable. The question is always the same sentence: is the gap between the groups too large to be sampling luck?

The t-test and analysis of variance (ANOVA) are the same idea at two scales. Both compute one ratio:

The denominator of a t-test is the standard error of the difference; the denominator of an ANOVA is the within-group variation. Once the numerator has been divided by it, what you hold is “how many units wide is this gap, measured with this dataset’s own noise as the ruler”. The p-value is only the last step that converts that number into a probability.

This page uses MASS::birthwt: 189 mothers recorded at a hospital in Springfield, Massachusetts in 1986, with newborn birth weight in grams as the outcome.

What the normality assumption actually assumes

The assumption is often compressed into “a t-test does not assume the data are normal”. It has three layers, and running them together is where the error creeps in:

1. The exact-inference layer really is an assumption about the data. The t statistic follows a t distribution exactly in a small sample only when the observations within each group — more precisely, the model’s errors — are exactly normal. Real data is at best approximately normal, so the distribution is approximate too. A paired t-test assumes that the paired differences are approximately normal, not that each of the two measurements is.
2. Welch relaxes a different assumption. What the Welch t-test drops is equality of the two variances; it leaves the normality assumption exactly where it was.
3. Only a large enough sample makes that layer matter less. When observations are independent and the distribution has finite variance, the central limit theorem (CLT) drives the sampling distribution of the mean (or of the mean difference) towards normal, so mild to moderate skewness in the raw data is usually not fatal. Small samples, extreme outliers, and very heavy tails still distort the inference.

So the question worth asking is not “are the data normal” but “given my n and the shape of this distribution, is the normal approximation good enough”. The third layer can be demonstrated: resample serum bilirubin from survival::pbc (skewness 2.7, with a very long right tail) and watch the distribution of the sample mean move towards normal:

So the working rule in practice is not “run a normality test and see whether it is significant”, but:

• Small n (a dozen or so per group) and a visibly skewed distribution → the p-value may be unreliable; consider a non-parametric method, or lead with the confidence interval for the difference.
• Moderate n or more (tens per group) → the t-test is usually quite robust even when the raw data are skewed.
• Extreme outliers → this is the genuinely dangerous case. The CLT converges far more slowly against outliers than against skewness, and the mean itself may no longer be the summary you wanted.

Welch should be the default

R’s t.test() defaults to var.equal = FALSE, that is, to the Welch t-test, and that default is right. Student’s t-test asks for one extra condition — equal variances in the two groups — and that condition:

• almost never holds exactly in real data;
• cannot safely be checked with a test (var.test(), Levene’s test) without walking into the trap described in the section on testing for normality first, below;
• distorts the type I error rate of the Student version when it fails, especially with unequal group sizes.

Welch does not need the condition. The price is fractional degrees of freedom and a negligible loss of power. Unequal variances are the normal state of affairs and equal variances the special case, so the default should match the normal state, not the special case.

With the data on this page the variance ratio is 1.3 and the two calculations reach the same conclusion:

The difference is tiny, and that is precisely the point: Welch costs you almost nothing and removes an assumption you would otherwise have to defend. There is no reason to take that assumption on to recover a sliver of power.

Running it yourself

library(MASS)
data(birthwt, package = "MASS")

birthwt$smoke_f <- factor(birthwt$smoke, levels = c(0, 1),
                          labels = c("Non-smoker", "Smoker"))
birthwt$race_f  <- factor(birthwt$race, levels = 1:3,
                          labels = c("White", "Black", "Other"))

# Two independent groups: Welch (the default)
t.test(bwt ~ smoke_f, data = birthwt)

# Paired: the same people measured twice
# Since R 4.4 the formula interface rejects paired=TRUE
# (error: cannot use 'paired' in formula method)
data(sleep)
t.test(Pair(sleep$extra[sleep$group == 2],
            sleep$extra[sleep$group == 1]) ~ 1)

# Three groups or more: one-way ANOVA + Tukey post hoc
fit <- aov(bwt ~ race_f, data = birthwt)
summary(fit)
TukeyHSD(fit)

# Non-parametric counterparts (badly skewed distribution and small n)
wilcox.test(bwt ~ smoke_f, data = birthwt)     # Mann-Whitney U
kruskal.test(bwt ~ race_f, data = birthwt)     # Kruskal-Wallis

import statsmodels.api as sm
from scipy import stats
import pingouin as pg

bw = sm.datasets.get_rdataset("birthwt", "MASS").data
g1 = bw.loc[bw["smoke"] == 0, "bwt"]
g2 = bw.loc[bw["smoke"] == 1, "bwt"]

# scipy's default is the opposite of R's — ask for Welch yourself
stats.ttest_ind(g1, g2, equal_var=False)

sleep = sm.datasets.get_rdataset("sleep").data
d1 = sleep.loc[sleep["group"] == 1, "extra"].to_numpy()
d2 = sleep.loc[sleep["group"] == 2, "extra"].to_numpy()
stats.ttest_rel(d2, d1)

groups = [g["bwt"].to_numpy() for _, g in bw.groupby("race")]
stats.f_oneway(*groups)
pg.pairwise_tukey(data=bw, dv="bwt", between="race")

The smoking group has 74 mothers with a mean of 2771.9 g (SD 659.6); the non-smoking group has 115 with a mean of 3055.7 g (SD 752.7). The difference is 283.8 g, 95% CI 79–489 g, p = 0.007, Cohen’s d = 0.4.

Paired or independent: the same numbers, two conclusions

What separates paired from independent data is not what the numbers look like, but whether the two columns are linked one to one: the same person before and after, the same person’s left and right eye, a matched case-control design. When the link exists you must use the paired version, because pairing removes person-to-person variation entirely and what remains is far quieter.

The soporific data Student analysed in 1908 (datasets::sleep, 10 subjects each trying both drugs, collected by Cushny and Peebles) shows it. Both analyses are fed exactly the same 20 numbers:

The point estimate is identical, yet the p-value moves from 0.003 to 0.079. The reason is that the two measurements correlate at 0.8 — people who sleep a lot anyway sleep a lot on either drug. The paired analysis subtracts that individual difference out and looks only at each person’s own change; the SD of the differences is just 1.23, against raw group SDs of 1.79 and 2.

Those lines on the left are what the paired test looks at. Of the 10 subjects, 9 slept longer on the second drug, 0 slept less, and 1 did not move at all. Vertically the lines occupy -1.6 to 5.5 hours, so people differ enormously from one another, yet each line runs almost the same way — and that direction only becomes visible once the two points are joined. The independent-samples analysis takes the two columns apart and sees two clouds of widely scattered points (SD 1.79 and 2), so the shared direction is counted as noise. The paired analysis collapses each line into one difference first (panel B); the differences run from 0 to 4.6, their scatter is down to an SD of 1.23, and the 95% confidence interval around their mean of 1.58 hours, 0.70–2.46, sits entirely above zero. The same direction is signal under one analysis and noise under the other.

ANOVA and the multiple comparisons that follow it

With three groups or more, running three pairwise t-tests lets type I error accumulate: across three independent comparisons the probability of at least one false positive is 14.3%, and across ten it is 40.1%. ANOVA puts an omnibus test in front, asking once whether all the group means are equal, and keeps the multiplicity outside the door.

For the three recorded race groups the result is F(2, 186) = 4.91, p = 0.008, $\eta^2$ = 0.05 — meaning this variable accounts for 5% of the total variation in birth weight.

Tukey HSD is the most common post hoc method, and what it controls is the family-wise error rate:

The gap between the last two columns is the price of the adjustment: for Black vs White the unadjusted p is 0.012 and the adjusted one is 0.043, sitting right against 0.05. A paper that reports only unadjusted pairwise p-values without saying so has quietly inflated its type I error rate.

A few practical points:

• Decide the post hoc comparisons in advance. “All pairs” and “each group against the control only” need different adjustments; the latter has more power with Dunnett’s test.
• Do not treat “only do pairwise tests if the ANOVA is significant” as a licence to skip adjustment (this is Fisher’s LSD). Beyond three groups it no longer protects the family-wise error rate.
• If the primary hypothesis was always about one specific pair, run that comparison directly and write it into the protocol; there is no need to pass through an ANOVA first.

Why “test for normality first, then decide” is a bad habit

A common workflow runs Shapiro-Wilk first, uses a t-test if p > 0.05 and switches to Wilcoxon if p < 0.05. It has three problems, and the first is fatal.

The power of a normality test varies with n in exactly the wrong direction. Draw repeated samples from one fixed skewed distribution (Gamma(shape = 2, rate = 1), skewness = 1.41), 2000 times each:

Same distribution throughout. At n = 20 Shapiro flags only about half the samples (52.4%), so close to half the time it waves you through to a t-test; at n = 500 it flags 100.0%, so you switch to a non-parametric method every time. Yet the Welch t-test’s actual type I error rate sits on the nominal 0.05 in both cases — the workflow waves you through when there is something to worry about and stops you when there is not.

The other two problems:

• A two-stage procedure distorts the final error rate. The p-value you report was computed conditional on the outcome of the first test, so its distribution is no longer the one you think it is.
• It hands the choice of method to the data. What analysis to run should follow from the design and the question, written down in advance, not picked after seeing the result.

The right approach: decide from the nature of the variable in advance — for quantities that are right-skewed by nature (length of stay, costs, biomarker concentrations), plan from the outset to report medians and use non-parametric methods or a transformation — then use a histogram and a Q-Q plot to confirm the data have no unexpected shape. Do not use a test as a switch.

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B1-02-t-test-anova.R

Watch next

▶醫學統計 EP10 t 檢定與非參數法

▶醫學統計 EP11 ANOVA

▶醫學統計 EP04 標準差與標準誤

▶生物統計學一 45.【型一與型二錯誤】

Report a problem on this page