Fixed-effect and random-effects models

The two models are not asking the same question

The usual formulation is that “random effects is more conservative”. That happens to describe the outcome in most cases, but it reverses cause and effect, and it leads to the wrong operation (“heterogeneity is high, so switch models to widen the interval”). The real difference is that the two make different assumptions about the world, and therefore estimate different things.

The fixed-effect model assumes every included study is estimating the same true value $\theta$:

Any apparent difference between studies is sampling error and nothing else. The question it answers is: “what is the common effect these studies point to?”

The random-effects model assumes each study has its own true value $\theta_i$, and that those true values are themselves drawn from a distribution:

The question it answers is: “where does the average of those true effects lie?” The extra $\tau^2$ is the between-study variance.

How far apart the two are on one dataset

Both models run on metadat::dat.bcg, the same 13 BCG vaccine trials used on the previous page and on B7-04:

Two things are worth stopping on:

1. The confidence interval becomes 4.4 times as wide (measured on the log scale). That is τ² = 0.313 being added to every study’s variance.
2. The point estimate also moves a long way, from RR 0.650 to 0.489. This gets overlooked constantly — many people believe switching models only changes the width of the interval. It changes the point estimate too, and by how much depends on whether the large and the small studies agree with each other.

What happens to the weights

Under a fixed-effect model the weight is $1 / v_i$; under random effects it is $1 / (v_i + \tau^2)$. Adding a constant $\tau^2$ that is the same for every study flattens the weights towards equality — the advantage a very precise study had is diluted away by τ².

Now the previous section’s question — why did the point estimate move — has an answer. The largest trial (TPT Madras, 1980, 176782 participants) has an RR of 1.012 on its own, which is to say it detected no protective effect. Under fixed effects that single trial holds 41.4% of the weight and drags the pooled value towards 1; random effects cut it to 10.2%, and the pooled value falls back towards what the remaining trials indicate.

τ² has many estimators, and they disagree

The τ² that the random-effects model adds has to be estimated; it is not something the data hand over directly. The common estimators:

On this dataset (13 studies, a lot of heterogeneity) the differences are modest, because there are enough studies. When there are few studies the differences start to matter.

Where DL goes wrong when there are few studies

A simulation makes it visible: the true τ² is fixed at the value estimated from the BCG data (0.313), each study’s within-study variance is resampled from the 13 variances actually observed there, only the number of studies $k$ changes, and each scenario is run 1500 times.

Means are pulled around by a handful of extreme estimates, so the paragraph below describes the median, which is closer to what a single meta-analysis typically obtains; the zero-estimate column shows that all three methods can conclude there is no heterogeneity at all.

The true value is 0.313. At $k$ = 5, DL’s typical estimate is only 0.206 — an underestimate of about 34% — against 0.241 for REML and 0.244 for PM. And DL returns τ² = 0 outright 6.8% of the time, announcing that there is no heterogeneity between these studies — in a world where the true τ² is plainly not small.

The direct consequence of underestimating τ² is a confidence interval that is too narrow, since $\tau^2$ is added to every study’s variance. That is what the next section repairs.

What the Knapp-Hartung adjustment repairs

The standard random-effects confidence interval uses a normal quantile (1.96) and treats the estimated $\hat\tau^2$ as if it were the true value. When $k$ is small and $\hat\tau^2$ is itself unstable, both of those make the interval too narrow.

The Knapp-Hartung adjustment (also called Hartung-Knapp-Sidik-Jonkman, HKSJ) does two things: it replaces the normal quantile with a $t$ quantile on $k - 1$ degrees of freedom, and it uses a variance estimator that absorbs some of the uncertainty in $\hat\tau^2$.

On the same BCG data, each estimator with and without KH:

With 13 studies the widening is around a tenth, which does not look like much. What KH rescues becomes obvious with fewer studies — the same simulation, now looking at how often the interval actually covers the true value:

At $k$ = 5, a nominally 95% interval covers the truth only 85.7% of the time; adding Knapp-Hartung brings it back to 93.4%. Put another way, a meta-analysis of five studies reporting a standard DL interval is reporting something closer to a 86% interval than a 95% one.

So which one should you choose

Run it yourself

library(metafor)
library(metadat)

data(dat.bcg, package = "metadat")
d <- escalc(measure = "RR", ai = tpos, bi = tneg, ci = cpos, di = cneg,
            data = dat.bcg, slab = paste(author, year, sep = ", "))

fe <- rma(yi, vi, data = d, method = "FE")     # fixed-effect
re <- rma(yi, vi, data = d, method = "REML")   # random-effects (today's default)
fe; re

# What happens to the weights -- this is where the third section's table comes from
# slab does not become a column of d (d$slab is NULL); it travels as the
# names of weights().
data.frame(study = names(weights(fe)),
           fixed  = round(weights(fe), 1),
           random = round(weights(re), 1))

# Swap the tau^2 estimator and watch the pooled value and interval move
for (m in c("DL", "REML", "PM", "SJ", "ML", "HE")) {
  r <- rma(yi, vi, data = d, method = m)
  cat(sprintf("%-5s tau2=%.4f  RR=%.3f (%.3f-%.3f)\n",
              m, r$tau2, exp(r$beta), exp(r$ci.lb), exp(r$ci.ub)))
}

# Knapp-Hartung: a t distribution, plus the uncertainty in tau^2
rma(yi, vi, data = d, method = "REML", test = "knha")

# The meta package does the same thing (its output is closer to a journal table)
# library(meta)
# metabin(tpos, tpos + tneg, cpos, cpos + cneg, data = dat.bcg,
#         sm = "RR", method.tau = "REML", hakn = TRUE)

import numpy as np
from statsmodels.stats.meta_analysis import combine_effects

# The 2x2 counts of metadat::dat.bcg, 13 trials, carried here so this
# block runs on its own.
tpos = np.array([4, 6, 3, 62, 33, 180, 8, 505, 29, 17, 186, 5, 27])
tneg = np.array([119, 300, 228, 13536, 5036, 1361, 2537, 87886, 7470, 1699,
                 50448, 2493, 16886])
cpos = np.array([11, 29, 11, 248, 47, 372, 10, 499, 45, 65, 141, 3, 29])
cneg = np.array([128, 274, 209, 12619, 5761, 1079, 619, 87892, 7232, 1600,
                 27197, 2338, 17825])
log_rr = np.log((tpos / (tpos + tneg)) / (cpos / (cpos + cneg)))
var    = 1 / tpos - 1 / (tpos + tneg) + 1 / cpos - 1 / (cpos + cneg)

# Input is the log RR and variance computed just above
res = combine_effects(log_rr, var, method_re="dl")
print(res.summary_frame())        # gives both the fixed-effect and random-effects rows

# The weights are easy to compute yourself -- and they are the point of this page
w_fe = 1 / var
w_re = 1 / (var + res.tau2)
print(np.round(100 * w_fe / w_fe.sum(), 1))
print(np.round(100 * w_re / w_re.sum(), 1))

# Knapp-Hartung has to be written by hand: a t quantile plus a weighted residual sum of squares
k  = len(log_rr)
mu = np.sum(w_re * log_rr) / np.sum(w_re)
q  = np.sum(w_re * (log_rr - mu) ** 2) / (k - 1)
se = np.sqrt(q / np.sum(w_re))
from scipy.stats import t
lo, hi = mu - t.ppf(0.975, k - 1) * se, mu + t.ppf(0.975, k - 1) * se
print("KH CI:", np.exp(lo), np.exp(hi))

Reading the report

In a paper, this page usually corresponds to a sentence or two in the Methods plus the line underneath the forest plot. Five things to find:

1. Which model was used, and why. “Random-effects model” with no reason is normal but not ideal; no statement of the model at all is a substantive omission.
2. Which estimator produced τ². If it is not stated, it is usually DL — especially in older reviews.
3. Whether Knapp-Hartung was applied. It appears as “Hartung-Knapp adjustment” or “HKSJ”. Fewer than ten studies and no mention of it means you should widen that confidence interval in your head.
4. Whether the model was chosen after the fact. A sentence like “because I² exceeded some value, a random-effects model was adopted” is data-driven model selection.
5. Where the weight is concentrated. If one box on the forest plot is conspicuously large, remove that study and look at the conclusion again — any reader can do this, as long as the paper reports the per-study data.

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B7-02-fixed-random.R

Watch next

▶醫學統計 EP18 統合分析：加權整合多項研究、看懂森林圖

▶Systematic reviews and meta analysis

▶How to do your first meta-analysis from start to finish

Report a problem on this page