Selection bias

It is not confounding, which is why adjustment cannot rescue it

Confounding means the two groups being compared were already different, and that difference affects the exposure and the outcome at once. It lives inside the data you collected, so as long as the variable was measured, putting it in the model handles it.

Selection bias means that who got into this dataset was itself determined by the exposure or by the outcome. It happens before the data exist. Every row you are holding has already been filtered. Put age, sex and every comorbidity into the model and it changes nothing about the people who were never selected, because they are not there.

Two diseases inside a hospital that look mutually exclusive but are not

Start with an example too clean to be misread. Two diseases exist in the population, with prevalences of 8% and 12%, and they are drawn independently in the code, so the true odds ratio between them is exactly 1. Neither affects the other, and they share no common cause.

The only thing that happens is this: having either disease raises your probability of being admitted. Someone with neither disease is admitted with probability 2.0%; someone with disease A alone with probability 36.3%, someone with disease B alone with probability 31.4%, and someone with both with probability 55.4%. That last cell is the engine of the whole thing: people who have both diseases are the easiest of all to get admitted, so inside the inpatient population “has both” is relatively magnified while “has neither” is all but emptied out — and most of what remains has one disease or the other.

Among inpatients the two diseases look strongly mutually exclusive. This is Berkson’s bias, also called admission rate bias. The mechanism is not mysterious: most people in hospital have at least one of the diseases, so within that group “does not have A” almost amounts to “has B”. The association was manufactured by the act of selection; it was never in the population.

The curve in the left panel deserves a second look, because it is continuous. A hospital is not a switch that is either biased or unbiased; the more specialised it is and the higher its admission threshold, the stronger the association it manufactures. At the far left, an admission probability of 0.5% pushes an odds ratio whose true value is 1 down to 0.025. At the far right, where everybody is “admitted”, the estimate returns to 0.990 — that cell simply is the whole population.

Four rules for who is missing from the analysis, with completely different costs

Berkson’s bias is the extreme case. The commoner problem is loss to follow-up in a cohort: some people are not followed to the end. Below, the same cohort — 400,000 people, of whom 40% are exposed — is analysed five times. The true risk ratio is 1.5 (risk in the unexposed is 10%) and the true odds ratio is 1.588.

The first row is the baseline in which everyone is analysed. The second is loss unrelated to both the exposure and the outcome, where every person is retained with the same probability of 35%. The three rows meant to be compared with one another are the third to the fifth: in those, the retention probability is always one of 90% or 20%, and the only thing that differs is what decides which of the two a person gets.

The benchmark to compare against is the first row, not the theoretical truth. This is one draw of the simulation, so the 1.523 and 1.616 in that row are the answer this dataset gives when nobody is lost at all, and the gap between them and the theoretical values is the scale of the simulation’s own noise. Read row by row, the table says four different things:

• Loss unrelated to both exposure and outcome (second row) — even after discarding more than six people in ten, both estimates are still on the truth. What was lost is precision (the confidence interval widens), not correctness.
• Loss that depends on the exposure only (third row) — also unbiased. This row is the one most often misread: “the exposed group was followed up much more completely” sounds alarming, but as long as, within each exposure stratum, the people who stayed have the same risk as the people who left, the risk inside each stratum is untouched — and the risk ratio is a ratio of within-stratum risks.
• Loss that depends on the outcome only (fourth row) — the risk ratio falls to 1.341, but the odds ratio is 1.610, essentially unmoved. The next section is about exactly this.
• Loss that depends on both exposure and outcome (fifth row) — a risk ratio of 4.52 and an odds ratio of 7.32, overstating the truth by factors of 3.0 and 4.6 respectively.

Why case-control studies report odds ratios and not risk ratios

The fourth row above was not a coincidence; it is an algebraic fact. Sampling on the outcome multiplies the odds within every exposure stratum by the same constant, so the ratio between strata is unchanged. Risk has no such property: risk is capped at 1, so multiplying it by a constant compresses it.

A case-control study does precisely this — find the cases first, then find the controls. It is the fourth row by construction. So reporting an odds ratio rather than a risk ratio is not a convention or a preference: the odds ratio is the only one of the two that remains estimable under that sampling design.

If you know the selection probability you can weight it back — and knowing it is the hard part

The wrecked estimate in the last row — the first row of the table below, which is the same draw and the same people as the fifth row above — can be repaired: if you know each person’s probability of being retained, use its reciprocal as a weight — inverse probability of selection weighting — and scale the survivors back up to the group they stand for.

After weighting, both estimates return to the neighbourhood of the truth. But the demonstration cheats in one specific way, and that is the point: in a simulation the selection probability is hard-coded, so it is known to the last decimal place.

In a real study nobody knows that probability. All you can do is build a model that predicts it from the variables that happen to be recorded on the people who stayed and the people who left — and if any part of the reason for leaving went unrecorded (deteriorating disease, moving away, an unreported death), that model is systematically wrong and the weighting only relocates the error.

The named biases in clinical research, and which row each one is

What to look for when reading a paper

1. Every box in the flow diagram needs a number and a reason. If the CONSORT or STROBE diagram just says “n excluded” without the reasons and the count for each reason, a reader cannot tell whether this is the third row or the fifth.
2. Loss to follow-up must be reported by group. One overall rate can hide differential loss; separate rates for the exposed and the comparison group cannot.
3. Baseline characteristics of those lost. A good paper includes a table comparing those who stayed with those who left. Without it you can only take the authors’ word for it.
4. Look in the eligibility criteria for phrases like “currently taking”, “able to attend follow-up”, “consented to participate”. Each of those defines a selection rule.
5. Where the controls came from. Hospital controls, neighbourhood controls and population-registry controls have entirely different bias structures.

Run it yourself

set.seed(20260823)
n <- 200000

# A cohort with a known truth: risk 0.10 unexposed, risk ratio 1.5
E <- rbinom(n, 1, 0.4)
Y <- rbinom(n, 1, ifelse(E == 1, 0.15, 0.10))

rr <- function(e, y) mean(y[e == 1]) / mean(y[e == 0])
rr(E, Y)                                  # the truth, computed on everybody

# Selection that depends on the exposure ONLY — still unbiased
keep <- rbinom(n, 1, ifelse(E == 1, 0.9, 0.2))
rr(E[keep == 1], Y[keep == 1])

# Selection that depends on BOTH — this is the one that breaks
keep2 <- rbinom(n, 1, ifelse(E == 1 & Y == 1, 0.9, 0.2))
rr(E[keep2 == 1], Y[keep2 == 1])

# Weight back by the inverse of the (here: known) selection probability
p  <- ifelse(E == 1 & Y == 1, 0.9, 0.2)[keep2 == 1]
Es <- E[keep2 == 1]; Ys <- Y[keep2 == 1]; w <- 1 / p
wrisk <- function(g) sum(w[Es == g] * Ys[Es == g]) / sum(w[Es == g])
wrisk(1) / wrisk(0)

import numpy as np
rng = np.random.default_rng(20260823)
n = 200_000

E = rng.binomial(1, 0.4, n)
Y = rng.binomial(1, np.where(E == 1, 0.15, 0.10))

def rr(e, y):
    return y[e == 1].mean() / y[e == 0].mean()

print(rr(E, Y))                                   # truth

keep = rng.binomial(1, np.where(E == 1, 0.9, 0.2))
print(rr(E[keep == 1], Y[keep == 1]))             # exposure-only: unbiased

keep2 = rng.binomial(1, np.where((E == 1) & (Y == 1), 0.9, 0.2))
print(rr(E[keep2 == 1], Y[keep2 == 1]))           # both: broken

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B8-03-selection-bias.R

Watch next

▶Principles of Epidemiology 09. Case-Control Study 2: M-H Methods & Selection Bias

▶Confounding, chance, and bias

Report a problem on this page