Multiple comparisons and subgroup analyses

An alpha of 0.05 is a promise about one test, not about twenty

α = 0.05 means: if there is no effect, this one test has a 5% chance of claiming there is one. It promises nothing whatsoever about anything other than “this one”.

If twenty outcomes are independent and every one of them truly has no effect, the probability that at least one comes out “significant” is

That is arithmetic, not simulation. What follows is simulation: 5,000 fictional trials, each with 60 patients per arm and 20 outcome measures, and both arms of every measure are drawn from the same distribution in the code — all true effects are zero, so every “significant” finding is a false positive by construction, with nothing to infer.

• On average each trial produces 0.995 “significant” results (the theoretical value is exactly 20 × 0.05 = 1)
• 64.3% of trials have at least one
• The worst single trial produced 5

Three adjustment procedures, guaranteeing three different things

All three adjusted rows in the table above land near 5.06% and look interchangeable — and that is because every effect in this simulation is truly zero. When all hypotheses are null, false discovery rate control and family-wise error rate control coincide. As soon as some effects are real, Benjamini–Hochberg rejects more than Bonferroni does, at the price of tolerating a defined proportion of errors among the findings it declares positive.

The volcano plot in an omics paper (genomics, proteomics, metabolomics, PheWAS) is this same section drawn differently: effect size across, $-\log_{10}(p)$ up, so the smaller the p-value the higher the point and the more extreme the effect the further out it sits, and the two decision rules become two horizontal lines. The plot below uses no new data. It is trial 2 from the very simulation on this page, picked by a rule fixed in advance — the first trial whose count of unadjusted hits equals the expected count, 20 × 0.05 = 1, rather than the best-looking one — and all 20 of its endpoints are truly null. The number of points above the unadjusted line at 0.05 is 1; above the Benjamini–Hochberg line it is 0. That point above the red line is not a signal, it is the kind of false positive this whole page is about — its raw p is 0.029 and its BH-adjusted p is 0.57. Which is to say that “is anything floating above the line?” is a conclusion, not a property of the data: the same points give a different answer under a different line, and moving the line requires no reanalysis at all. The first thing to ask of one of these plots is therefore which kind of threshold that line is.

Where the green line sits needs one more sentence, because it is not a fixed height. Benjamini–Hochberg is a step-up procedure: sort the 20 p-values from smallest to largest and judge the i-th against its own rung of a ladder, α × i / 20. The largest p-value that still clears its own rung sets the line, and everything at or below it is declared positive — so the line moves with the data, and with real signal present it can sit far above where Bonferroni would. Bonferroni, by contrast, holds every p-value to the bottom rung and only to that rung. In this trial nothing cleared any rung, so no such p-value exists and the honest line to draw is the bottom rung itself, α / 20 = 0.0025 — numerically the same threshold as Bonferroni. That is not a contradiction of the table above: when nothing climbs the ladder at all, the two procedures meet at its foot.

Subgroup analyses: the problem is not the missing adjustment, it is the wrong question

The commonest multiplicity problem in a clinical trial is not twenty outcome measures; it is one subgroup forest plot. The simulation below has a single outcome and a single real effect, and the true effect is identical in every subgroup — no interaction exists at all. The total trial size is fixed and cut into g equal parts.

The middle column is where sentences like “this drug works in one group but not another” come from — and how much to believe one depends on how many subgroups were cut. A plain split by sex is a different case: at 2 subgroups the split rate is 11.3%, which the Callout below returns to. Cut into 6 subgroups, 96.7% of trials produce a split result — at least one subgroup significant and at least one not — while the truth is that every subgroup has exactly the same effect.

The right-hand column is the control: the correct question is not “which subgroup was significant” but “do the subgroup effects actually differ”, which is the test for interaction. Its false positive rate stays between 5.2% and 7.0% throughout — hugging the nominal 5%, with the upper end running slightly above it (with many subgroups each one is down to a few dozen people and the large-sample approximation starts to strain; see the end of this section).

Two honest caveats, because they are on the same table:

• The middle column reads 98.7% at 8 subgroups and then falls slightly to 97.5% at 10. It is not monotone: past a certain fineness every subgroup is too small, so “nothing is significant anywhere” starts to become common — and a result with nothing significant is not a split. A falling split rate does not mean the problem is shrinking; it means the information has been diluted until nothing is visible.
• The right-hand column reaches 7.0% at 10 subgroups, somewhat above its nominal 5%. That is the large-sample approximation straining when each subgroup is down to a few dozen people, not a failure of the interaction test as such — but it is a reminder that past a certain number of subgroups, even the correct test starts to become unreliable.

When adjustment is called for, and when it is not

What to look for when reading a paper

1. Which outcome the protocol or the registry entry names as primary. If the sentence in the abstract is about a different one, ask why first.
2. How many outcomes were measured in total. If the methods list fifteen and the results report three, ask where the other twelve went.
3. Whether subgroups were pre-specified or post hoc. CONSORT requires this to be stated; if it is not stated, treat them as post hoc.
4. Whether an interaction p-value is reported. Giving only each subgroup’s effect and each subgroup’s p-value is an invitation for the reader to make exactly the comparison that goes wrong.
5. Whether the adjustment method is named, and which error rate it controls. “Bonferroni-corrected” and “FDR < 0.05” are two different claims.
6. How much alpha the interim analyses spent. The final significance threshold is usually smaller than 0.05.

Run it yourself

set.seed(20260822)

# One trial: 20 endpoints, no effect anywhere
one_trial <- function(k = 20, n = 60) {
  replicate(k, t.test(rnorm(n), rnorm(n), var.equal = TRUE)$p.value)
}

p <- one_trial()
sum(p < 0.05)                          # how many "significant" findings from nothing

p.adjust(p, "bonferroni") < 0.05       # all FALSE, almost always
p.adjust(p, "holm")       < 0.05
p.adjust(p, "BH")         < 0.05

# The family-wise error rate, measured rather than asserted
mean(replicate(1000, any(one_trial() < 0.05)))                       # about 0.64
mean(replicate(1000, any(p.adjust(one_trial(), "holm") < 0.05)))     # about 0.05

import numpy as np
from scipy import stats
from statsmodels.stats.multitest import multipletests

rng = np.random.default_rng(20260822)

def one_trial(k=20, n=60):
    return np.array([stats.ttest_ind(rng.normal(size=n), rng.normal(size=n)).pvalue
                     for _ in range(k)])

p = one_trial()
print((p < 0.05).sum())

for method in ("bonferroni", "holm", "fdr_bh"):
    print(method, multipletests(p, alpha=0.05, method=method)[0].sum())

print(np.mean([(one_trial() < 0.05).any() for _ in range(1000)]))     # about 0.64

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B8-05-multiplicity.R

Report a problem on this page