Effect measures and their variances

What a meta-analysis pools is a pair of numbers, not a set of conclusions

A meta-analysis looks like several studies’ conclusions being added together, but only two things are ever handed to the model: an effect measure (each study’s estimate of the effect) and its variance (how imprecise that estimate is). Everything else — sample size, length of follow-up, number of events — reaches the result only through the variance.

That has two consequences, and the rest of this page is about them:

1. Choosing the effect measure is a real decision, not a formatting one. The same set of 2×2 tables expressed as RR, OR or RD gives a different pooled estimate and a different amount of heterogeneity.
2. The variance is the weight. How much influence a study has relative to the others is decided entirely by its variance, and the variance is driven by the number of events, not the number of participants.

Six common effect measures, with different properties

Why ratio measures have to be pooled on the log scale

This is not a convention. It follows from the shape of the sampling distribution.

A ratio is structurally asymmetric: downward it can only reach 0, upward it has no limit. “Halving the risk” is 0.50 and “doubling the risk” is 2.00. Clinically these are mirror images of each other, but on the number line one sits 0.50 away from 1 and the other sits 1.00 away. Take logs and the two become symmetric about 0.

A simulation makes it concrete: 100 participants per arm, an event rate of 0.20 in the control arm and 0.10 on treatment (so the true RR is 0.50), repeated over 40000 trials.

The difference is not merely cosmetic. Using the same simulated data, build a 95% Wald interval each way:

• Directly on the original scale (RR ± 1.96 × SE): the intervals actually cover the truth 92.9% of the time
• On the log scale, then exponentiated back (exp(log RR ± 1.96 × SE)): 95.7%

The first falls about 2.1 percentage points short of its nominal 95%, and its lower limit can drop below 0 — a negative risk ratio. This is why the internals of every meta-analysis package work on the log scale: the yi that escalc() hands back is a log RR, not an RR, and the exponentiation happens only at the plotting and reporting step.

Where the variance comes from, and how it becomes a weight

For a log RR, the variance is

where $a$ and $c$ are the event counts in the two arms and $n_1$, $n_2$ are the arm sizes. Note that the dominant terms are the reciprocals of the event counts, not of the sample sizes. A study that enrolled thirty thousand people but observed twenty events carries roughly the precision of a study that enrolled two hundred people and also observed twenty events. In meta-analysis, “a large study” always means a study with many events.

Under a fixed-effect model the weight is the reciprocal of the variance, normalised to a percentage:

Here it is computed on metadat::dat.bcg, the 13 controlled trials of BCG vaccination against tuberculosis (only some of which allocated participants at random; the allocation method is used as a subgroup variable on the heterogeneity page):

The largest trial (TPT Madras, 1980, 176782 participants) takes 41.4% of the weight on its own, while the trial with the smallest weight (Comstock & Webster, 1969) gets 0.31% — a difference of more than a hundredfold. “Smallest weight” is not the same as “fewest participants”: that trial enrolled 4839 people, which puts it in the middle of these 13 trials, and every trial that enrolled fewer people has a higher weight than it does. Its weight is low because it observed few events, not because it recruited few people. This is exactly why the size of the boxes on a forest plot is worth staring at, and it is also the quantity that the next page rewrites substantially by switching to a random-effects model.

Continuous outcomes: MD and SMD are not interchangeable

When every study measured the same thing with the same instrument (length of stay, systolic blood pressure, body weight), pool the mean differences directly. The unit survives, and the reader understands the result immediately.

When the studies measured one construct on different scales (three depression inventories, four pain scores), the unit has to be divided out first. The SMD does that by dividing each study’s mean difference by its own within-group standard deviation:

Running both on metadat::dat.normand1999 (9 trials of stroke units, with length of hospital stay as the outcome):

Neither pooled estimate reaches statistical significance — both confidence intervals cross 0 — but that is not the point of the table. The point is that I² moves from 99.0% to 95.5%. Those two I² values must not be compared directly. I² is a proportion, and changing the effect measure changes both its numerator and its denominator at once: the units of τ² go from “days squared” to no units at all, and each study’s within-study variance is recomputed as well (the variance of an SMD carries an extra term for the uncertainty in the estimated standard deviation). So the drop cannot be read as “the real differences between studies got smaller” — answering that question means looking at an absolute quantity on a single scale (see the heterogeneity page).

Zero cells: a problem with no settled answer

When events are rare, some studies will have a zero in one cell. The log RR or log OR is then infinite, so is the variance, and the model simply stops.

metadat::dat.nielweise2007 is a clean example: 18 trials comparing antimicrobial-coated central venous catheters with standard ones, with catheter-related bloodstream infection as the outcome. 6 of them contain a zero cell, and 1 of those 6 has zeros in both arms. Four common ways of handling this give:

Two things vary across this table at once, and they need separating when you read it. The first two rows are both random-effects models (REML) and differ only in how the zero cells are treated. The Mantel-Haenszel and Peto rows exist only as fixed-effect procedures, so they differ from the first two rows in the model and in the zero-cell handling. For a comparison that isolates the zero-cell handling, read the first two rows.

On one dataset, the pooled OR ranges from 0.299 to 0.439, a spread of more than 40%. All four appear in the published literature and none of them is wrong, but if a paper does not say which one it used, the number cannot be reproduced.

Change the effect measure and heterogeneity changes its face

One last consequence that is easy to miss: the choice of effect measure changes how inconsistent the studies appear to be. The same BCG trials, three measures:

The reason is this: baseline tuberculosis incidence differs across these trial sites by orders of magnitude (an urban slum and a rural district are not remotely comparable). On a ratio scale the baseline rate cancels out, so as long as the vaccine’s relative protection is similar the trials look consistent. On a difference scale the baseline rate stays inside the effect measure, so sites with a high baseline naturally show a much larger risk difference — and because the variance of an RD is very small, that spread looks statistically certain.

Run it yourself

library(metafor)
library(metadat)

data(dat.bcg, package = "metadat")

# escalc() is the step that turns raw data into (yi, vi).
# The yi it returns is a log RR, not an RR -- pooling happens entirely in logs.
rr <- escalc(measure = "RR", ai = tpos, bi = tneg, ci = cpos, di = cneg,
             data = dat.bcg, slab = paste(author, year, sep = ", "))
head(rr[, c("yi", "vi")])

# The same 2x2 tables as OR and RD: watch the pooled value and the heterogeneity move
or <- escalc(measure = "OR", ai = tpos, bi = tneg, ci = cpos, di = cneg, data = dat.bcg)
rd <- escalc(measure = "RD", ai = tpos, bi = tneg, ci = cpos, di = cneg, data = dat.bcg)
sapply(list(RR = rr, OR = or, RD = rd), function(d) rma(yi, vi, data = d)$I2)

# The weight is the reciprocal of the variance (fixed-effect)
fe <- rma(yi, vi, data = rr, method = "FE")
round(weights(fe), 1)

# -- Continuous outcomes: MD and SMD --
data(dat.normand1999, package = "metadat")
escalc(measure = "MD",  m1i = m1i, sd1i = sd1i, n1i = n1i,
       m2i = m2i, sd2i = sd2i, n2i = n2i, data = dat.normand1999)
escalc(measure = "SMD", m1i = m1i, sd1i = sd1i, n1i = n1i,
       m2i = m2i, sd2i = sd2i, n2i = n2i, data = dat.normand1999)  # small-sample corrected

# -- Zero cells: four ways of handling them --
data(dat.nielweise2007, package = "metadat")
d <- dat.nielweise2007

# (1) Default: add 0.5, but only to studies that contain a zero
rma(escalc("OR", ai = ai, n1i = n1i, ci = ci, n2i = n2i, data = d,
           add = 1/2, to = "only0"))
# (2) No correction -- studies with a zero become NA and are dropped, which is the problem
rma(escalc("OR", ai = ai, n1i = n1i, ci = ci, n2i = n2i, data = d, add = 0))
# (3) Mantel-Haenszel: no correction needed
rma.mh(measure = "OR", ai = ai, n1i = n1i, ci = ci, n2i = n2i, data = d)
# (4) Peto: the choice when events are rare and the arms are similar in size
rma.peto(ai = ai, n1i = n1i, ci = ci, n2i = n2i, data = d)

import numpy as np
from statsmodels.stats.meta_analysis import combine_effects

# The 2x2 counts of metadat::dat.bcg, 13 trials, carried here so this
# block runs on its own.
tpos = np.array([4, 6, 3, 62, 33, 180, 8, 505, 29, 17, 186, 5, 27])
tneg = np.array([119, 300, 228, 13536, 5036, 1361, 2537, 87886, 7470, 1699,
                 50448, 2493, 16886])
cpos = np.array([11, 29, 11, 248, 47, 372, 10, 499, 45, 65, 141, 3, 29])
cneg = np.array([128, 274, 209, 12619, 5761, 1079, 619, 87892, 7232, 1600,
                 27197, 2338, 17825])

# statsmodels has no escalc(); the 2x2 tables must be converted by hand
a, n1, c, n2 = tpos, tpos + tneg, cpos, cpos + cneg      # each study's 2x2 cells
log_rr = np.log((a / n1) / (c / n2))
var    = 1 / a - 1 / n1 + 1 / c - 1 / n2                 # event counts dominate

res = combine_effects(log_rr, var, method_re="dl")
print(res.summary_frame())          # still on the log scale; exponentiate yourself
print("pooled RR =", np.exp(res.mean_effect_re))

# Zero cells have to be handled manually, or log() returns -inf
zero = (a == 0) | (c == 0)
a_c, c_c = a.astype(float).copy(), c.astype(float).copy()
a_c[zero] += 0.5; c_c[zero] += 0.5   # the crudest possible approach -- see the three problems above

Reading the report

In a published meta-analysis, this page corresponds to the first few sentences of the “Statistical analysis” paragraph in the Methods. Four things to look for:

1. Which effect measure was pooled, and whether a reason is given. “Pooled using a random-effects model” without saying whether the quantity pooled was an RR, an OR or an RD is a substantive omission. The reason should be mechanistic or clinical, not “because heterogeneity was lower”.
2. How the raw data became effect sizes. If the paper reports only the pooled value without each study’s $n$ and event counts, no reader can check a single cell. A per-study table of 2×2 counts or effect sizes belongs in the supplement.
3. Whether there were zero cells, and how they were handled. A meta-analysis of rare events that never mentions zero cells has usually accepted a package default without noticing.
4. Whether an OR is being read as an RR. An abstract saying “risk reduced by X%” when the table contains odds ratios is the most common form of overstatement, especially when events are not rare.

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B7-01-effect-size.R

Watch next

▶【Biostatistics Corner】Meta-analysis 基礎篇

▶Understand What a Meta-Analysis is in Less Than 5 Minutes

▶An Introduction to Systematic Review and Meta-analysis

Report a problem on this page