Heterogeneity — I², τ² and prediction intervals
Why I² can run from around half to nearly a hundred per cent while the real differences between studies stay exactly the same, whether a non-significant Q test is evidence of homogeneity, why a prediction interval is closer to what readers actually want to know than a confidence interval is, and the trap in meta-regression where a study-level association gets read as an individual-level one.
Three statistics, measuring three different things
I² appears in nearly every meta-analysis, τ² and Q occasionally, H² rarely. They are related by a set of identities, but they answer different questions:
| Statistic | Definition | Unit | What it answers |
|---|---|---|---|
| Q | Weighted sum of squared deviations of the studies from the pooled value | none | “Is the observed spread larger than sampling error alone would produce?” — this is a test statistic |
| τ² | Variance of the true effects across studies | squared units of the effect measure (here, squared log RR) | “How widely are the true effects spread?” — an absolute quantity |
| H² | Total variation ÷ sampling variation (see the note below on how it is estimated) | none | “How many times sampling error is the total spread?” — its floor is 1 |
| I² | Percentage of the total variation that is between-study variation | % | “What proportion of the spread is not sampling error?” — a proportion |
On these 13 BCG vaccine trials:
| Statistic | Value | How to read it |
|---|---|---|
| Q | 152.2 (df = 12, p < 0.001) | The spread is far beyond what sampling error can account for |
| τ² | 0.3132 | The variance of the log RR across studies |
| τ | 0.560 | The standard deviation of the log RR across studies — a more useful number than τ² |
| H² | 12.86 | The total variation is 12.9 times the sampling variation |
| I² | 92.2% | About nine tenths of the total variation is not sampling error |
I² is a proportion, so precision drags it around
This is where I² is most often misread, and one experiment settles it: hold τ² fixed at the observed value and multiply every study’s within-study variance by a constant (as if imagining these trials had enrolled more or fewer people), then watch what I² does.
| Within-study variances × | τ² (held fixed) | H² | I² |
|---|---|---|---|
| 0.1 | 0.3132 | 119.56 | 99.2% |
| 0.25 | 0.3132 | 48.42 | 97.9% |
| 1 | 0.3132 | 12.86 | 92.2% |
| 4 | 0.3132 | 3.96 | 74.8% |
| 10 | 0.3132 | 2.19 | 54.2% |
The real differences between the effects are identical throughout — τ² is the same number in every row — yet I² travels from 54.2% all the way to 99.2%. The only thing that changed is how precisely each study measured.
A non-significant Q test does not mean homogeneity
The null hypothesis of the Q test is that every study’s true effect is the same. It is a test, so it has a power problem — and in meta-analysis the sample size is the number of studies , which is usually small.
A simulation draws the power curve: each study’s within-study variance is resampled from the 13 variances actually observed here, the true τ² is fixed at four levels, and only varies.
figures/scripts/B7-03-heterogeneity.R| Number of studies k | tau2 = 0 (no heterogeneity) | I2 about 25% | I2 about 50% | BCG data (I2 about 92%) |
|---|---|---|---|---|
| 3 | 5.2% | 9.1% | 14.6% | 53.4% |
| 5 | 4.7% | 12.0% | 24.4% | 75.1% |
| 8 | 5.0% | 16.1% | 35.0% | 92.8% |
| 10 | 4.9% | 18.5% | 42.2% | 96.4% |
| 15 | 4.5% | 22.9% | 56.3% | 99.5% |
| 20 | 4.8% | 27.9% | 66.8% | 100.0% |
| 30 | 4.3% | 36.9% | 80.5% | 100.0% |
In a world where the true I² is about 50%, with 10 studies included, the Q test reaches significance only 42.2% of the time. Which is to say that more than half the time the authors will see a non-significant Q and write “no heterogeneity was found between studies” — and be wrong.
Prediction intervals: the question readers are actually asking
A confidence interval answers “where is the average effect μ”. But the clinical question is usually a different one: “where will my next group of patients, the next centre, the next trial, end up?”
That question is answered by a prediction interval, which carries both the uncertainty in μ and the spread of the effects themselves:
Two versions of that quantile are in circulation, and neither of them is simply “whatever the software defaults to”. Higgins and colleagues’ original version uses a distribution on degrees of freedom; metafor’s predict() defaults to the normal quantile 1.96. When is large the two barely differ; when is small they differ a lot, so the table below lists both — and a paper quoting a prediction interval has to say which one it used.
On the BCG data, the difference between the prediction interval and the confidence interval is decisive:
| Interval | Lower | Upper | Crosses 1? |
|---|---|---|---|
| 95% confidence interval (where the average effect lies) | 0.344 | 0.696 | No |
| 95% prediction interval (where the next study will land) — normal quantile, metafor’s default | 0.155 | 1.549 | Yes |
| 95% prediction interval — t quantile, k − 2 degrees of freedom | 0.134 | 1.785 | Yes |
figures/scripts/B7-03-heterogeneity.RThis sentence is worth writing out in full: on average across these trials, BCG significantly reduces the risk of tuberculosis; but on the present evidence, a new trial run at a new site could perfectly well land on the side where no protective effect is detected. Both halves are true, and reporting only the first is selective presentation.
Explaining heterogeneity: meta-regression
The right response to heterogeneity is not to suppress it but to ask where it comes from. The BCG dataset became a teaching classic precisely because its source is visible: the latitude of the trial site.
ablat is the absolute latitude of each trial’s location. Put it in the model as a covariate:
| Quantity | Value |
|---|---|
| Change in log RR per degree of latitude | -0.0291 (SE 0.0072, p < 0.001) |
| Converted: RR multiplied per ten degrees of latitude | 0.748 |
| R² (between-study variance explained) | 75.6% |
| Residual τ² | 0.0764 (was 0.3132) |
| Residual I² | 68.4% (was 92.2%) |
| Test for residual heterogeneity, QE | 30.7 (p 0.001) |
figures/scripts/B7-03-heterogeneity.RSubgroup analysis: the coarse version of the same thing
Splitting the studies by a categorical variable and pooling each group separately is the commonest way of exploring heterogeneity. Splitting by alloc (how the trial allocated its participants):
| Group | k | Pooled RR | 95% CI | I² | τ² | 95% prediction interval (normal quantile) |
|---|---|---|---|---|---|---|
| Alternate allocation | 2 | 0.582 | 0.335–1.011 | 82.0% | 0.133 | 0.236–1.435 |
| Random allocation | 7 | 0.379 | 0.221–0.650 | 89.9% | 0.393 | 0.099–1.449 |
| Systematic allocation | 4 | 0.654 | 0.323–1.324 | 86.4% | 0.400 | 0.157–2.725 |
τ² has to be read together with k. The Alternate allocation group, for instance, has only k = 2, so its τ² of 0.133 is an extremely unstable model estimate and cannot be taken as a stable statement about absolute heterogeneity.
The point estimates look very different (from 0.379 to 0.654), but the formal test for a difference between groups does not reach statistical significance: QM = 1.77, df = 2, p = 0.413.
The ecological fallacy: a study-level association is not an individual-level one
This is the error easiest to commit in your own analysis, and it is a limitation shared by meta-regression and subgroup analysis alike.
The latitude regression line above says that “trials at higher average latitude observed better vaccine effects”. It does not say that “within a single trial, the participants who lived further north were better protected after vaccination”. The first is an aggregate-level association, the second an individual-level one, and the two can differ completely — even in direction.
The version you meet more often in clinical work: a meta-analysis finds that “trials with a higher mean age showed larger treatment effects”, and this gets written up as “older patients benefit more”. That inference might be true, but this data cannot support it — trials with a higher mean age may also have had stricter entry criteria, more comorbidity, longer follow-up, or different dosing.
Run it yourself
library(metafor)
library(metadat)
data(dat.bcg, package = "metadat")
d <- escalc(measure = "RR", ai = tpos, bi = tneg, ci = cpos, di = cneg,
data = dat.bcg, slab = paste(author, year, sep = ", "))
res <- rma(yi, vi, data = d, method = "REML")
res # I^2, H^2, tau^2 and Q all arrive in one output
# Prediction interval: the pi.lb / pi.ub columns; transf = exp returns to the RR scale
predict(res, transf = exp) # normal quantile 1.96 by default
predict(res, transf = exp, pi.type = "riley") # the t(k-2) version (Higgins et al. 2009)
# I^2 is a proportion: multiply every within-study variance by 4, pin tau^2, watch I^2 fall
rma(d$yi, d$vi * 4, method = "DL", tau2 = res$tau2)$I2
# -- Meta-regression: explaining heterogeneity with latitude --
mr <- rma(yi, vi, mods = ~ ablat, data = d, method = "REML")
mr # R^2, residual I^2 and the residual heterogeneity test QE are all in here
regplot(mr, xlab = "Absolute latitude", las = 1) # the bubble plot
# -- Subgroups: the between-group difference is QM, not a comparison of two p-values --
rma(yi, vi, mods = ~ factor(alloc), data = d, method = "REML")
# For a pooled value within each group, fit them one at a time (with their own prediction intervals)
for (lv in levels(factor(dat.bcg$alloc))) {
r <- rma(yi, vi, data = d, subset = (dat.bcg$alloc == lv), method = "REML")
print(predict(r, transf = exp))
}Verified with R 4.6.0, metafor 5.0.1 and metadat 1.6.0
import numpy as np
from scipy.stats import norm, t
from statsmodels.stats.meta_analysis import combine_effects
# The 2x2 counts of metadat::dat.bcg, 13 trials, carried here so this
# block runs on its own.
tpos = np.array([4, 6, 3, 62, 33, 180, 8, 505, 29, 17, 186, 5, 27])
tneg = np.array([119, 300, 228, 13536, 5036, 1361, 2537, 87886, 7470, 1699,
50448, 2493, 16886])
cpos = np.array([11, 29, 11, 248, 47, 372, 10, 499, 45, 65, 141, 3, 29])
cneg = np.array([128, 274, 209, 12619, 5761, 1079, 619, 87892, 7232, 1600,
27197, 2338, 17825])
log_rr = np.log((tpos / (tpos + tneg)) / (cpos / (cpos + cneg)))
var = 1 / tpos - 1 / (tpos + tneg) + 1 / cpos - 1 / (cpos + cneg)
res = combine_effects(log_rr, var, method_re="dl")
print("I2 =", res.i2, " Q =", res.q, " tau2 =", res.tau2)
# The prediction interval has to be built by hand: mu +- c * sqrt(tau2 + SE(mu)^2)
k = len(log_rr)
w_re = 1 / (var + res.tau2)
mu = np.sum(w_re * log_rr) / np.sum(w_re)
se = np.sqrt(1 / np.sum(w_re))
spread = np.sqrt(res.tau2 + se ** 2)
# (1) Normal quantile -- matches metafor's predict() default, and the second table row above
half_z = norm.ppf(0.975) * spread
print("PI (normal):", np.exp(mu - half_z), np.exp(mu + half_z))
# (2) t quantile on k-2 df -- the original version (Higgins et al. 2009), the third table row
# in this version of metafor, write predict(res, pi.type = "riley"), an option
# named after Riley et al. 2011 rather than after the paper the formula is from
half_t = t.ppf(0.975, k - 2) * spread
print("PI (t, k-2):", np.exp(mu - half_t), np.exp(mu + half_t))
# Meta-regression means weighted least squares by hand, with no R^2 or residual I^2 providedstatsmodels gives you I² and Q and nothing else — no prediction interval, no meta-regression, no between-group test. What is on this page has to be done in R.
Reading the report
- Whether τ² (or τ) is given alongside I². With only I², a reader cannot judge how large that “inconsistency” is clinically.
- Whether there is a prediction interval. A random-effects model reporting only a confidence interval has left out the thing the model is describing. Failing that, estimate it roughly from τ: two standard deviations of the effect distribution is about a factor of exp(±2τ).
- The wording used when Q is not significant. “No heterogeneity was found” and “no statistically significant heterogeneity was detected” are methodologically different sentences, and only the second is defensible.
- Whether subgroup differences come with a between-group test. Listing each group’s own p-value without QM is where the difference-in-significance fallacy breeds.
- Whether subgroups and meta-regressions were pre-specified or dug out afterwards. The protocol (a PROSPERO registration number) is something you can go and check yourself.
- Whether a study-level association has been written up as a patient-level conclusion. When you see “older / female / more severely ill patients benefited more”, first check whether the data were IPD or study-level covariates.
Common misuses
| Misuse | Why it is wrong |
|---|---|
| Interpreting I² against fixed cut-points (25/50/75%) | The same I² corresponds to very different clinical spreads on different data; Cochrane says explicitly not to use them as cut-points |
| Describing “how large the differences in effect are” with I² | I² is a proportion and is driven by how precise the studies are; absolute size means τ² or the prediction interval |
| Writing “no heterogeneity” when Q is not significant | Power is low when studies are few; the most that can be said is “no statistically significant heterogeneity was detected” |
| Choosing fixed or random effects from the Q test’s p-value | Data-driven model selection, resting on a test with very little power |
| Reporting a random-effects confidence interval with no prediction interval | The confidence interval is about the average; readers want to know where the next study will land |
| Reporting a prediction interval from very few studies and drawing conclusions from it | τ² is badly estimated, the interval is uninformatively wide, and the normality assumption cannot be checked |
| Comparing “which subgroup was significant” | The difference-in-significance fallacy; use the between-group QM |
| Carving out subgroups after the fact and treating them as findings | A subgroup that was not pre-specified is a hypothesis, not a discovery |
| Loading a meta-regression with many covariates | The sample size is the number of studies; each covariate needs about ten studies behind it |
| Reading a study-level covariate association as patient-level effect modification | The ecological fallacy; answering that question requires IPD-MA |
| Deleting outlying studies to bring I² down | This lets the result decide the inclusion criteria, and outliers are often the most informative studies of all |
Reproducing every number on this page
/opt/homebrew/bin/Rscript figures/scripts/B7-03-heterogeneity.RRead the figure
The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.
Hold τ² for these 13 BCG trials fixed at 0.3132 and multiply every study's within-study variance by a factor. What happens to I²?
Show the answer and why
Correct answer: At ten times the variances I² falls to 54.25% — the true differences between studies have not moved at all, only how precisely each one was measured
I² is a ratio: τ² on top, τ² plus the typical within-study variance underneath. The numerator is nailed down all the way through this table while the denominator follows how precise the studies are, so one and the same set of true differences can show as 54.25% or as 99.16%. Two reading rules follow. A set of large, precise trials will show a high I² even when the true differences are clinically negligible; a set of small, imprecise studies can show a low I² even when the true effects are wildly different, because sampling error has inflated the denominator. The 92.22% is simply the row computed from the observed variances — a cell in this table, not a fixed constant. To answer how large the differences actually are you need an absolute quantity with units, τ² or a prediction interval; and the low-medium-high thresholds are an illustration from one methods paper, which the Cochrane Handbook now explicitly says not to treat as cut-offs.
A simulation shows that with a true I² of about 50% and 10 studies, the Q test reaches significance only 42.2% of the time. What does that say about choosing a fixed-effect model because the Q test was not significant?
Show the answer and why
Correct answer: It says non-significant means undetected this time: one step less heterogeneity, still 10 studies, and the rate is down to 18.5% — choosing a model on this test lets power choose the model
The sample size of a Q test is the number of studies, and that number is usually small. With a true I² near 50% and 10 studies it reaches significance 42.2% of the time; one step less heterogeneity and only 18.5%. A non-significant Q therefore says not detected here, not absent, and using it to pick a model makes three errors at once: treating undetected as absent, deciding on a low-powered test, and letting the data choose the model. The 4.9% is the significance rate when the true τ² is zero, which sits exactly at the nominal level and shows the test is not broken — it simply cannot see moderate heterogeneity in a small set, and those are different things. The 80.5% belongs to the row with the most studies, where one in five is still missed, and most clinical meta-analyses never assemble that many. The honest wording is that no statistically significant heterogeneity was detected in this analysis, reported alongside τ² and a prediction interval.
In one random-effects model the pooled RR of 0.489 has a 95% confidence interval of 0.344 to 0.696, entirely to the left of 1. Why report a second interval that does cross 1?
Show the answer and why
Correct answer: Because it is a prediction interval, answering where one new trial at one new site would land: its upper bound reaches 1.549, so landing on the no-detected-protection side is entirely possible
A confidence interval answers where the mean effect lies; a prediction interval answers where the next study will land, and it carries both τ² and the standard error of the mean. So the honest report is two sentences: on average across these trials BCG significantly lowers the risk of tuberculosis, and on the present evidence a new trial at a new site could perfectly well land on the side where no protective effect is detected — that is what the upper bound of 1.549 says. Reporting only the first sentence is selective presentation. The 3.270 is the ratio of the two widths on the log scale, a consequence rather than a reason: the prediction interval is not a widened confidence interval, it answers a different question. The 1.785 is the upper bound with t quantiles; the two versions differ in the quantile chosen, so a citation should say which was used rather than pick the wider one, since picking is deciding after seeing the result.
Adding the absolute latitude of the trial site to a meta-regression gives R² of 75.6% with a residual I² of 68.4%. What claim does that support?
Show the answer and why
Correct answer: An associational one only: the residual I² is still 68.39%, and the covariate is observed rather than assigned — era, strain and diagnostic criteria all move with latitude
R² says how much between-study variance latitude explains; it is not evidence of causation. The covariate is observed, not randomised: latitude travels with era, nutrition, strain, diagnostic criteria and study quality, and a meta-regression cannot separate them, so the conclusion is observational and hypothesis-generating even when every included study was a randomised trial. The residual I² of 68.39% is a further reminder that latitude has not explained the heterogeneity away. The third option commits the ecological fallacy: this regression line says that trials at higher average latitude observed larger effects, not that within a trial the people living further north were better protected. Answering the individual-level question takes an individual participant data meta-analysis, estimating the interaction inside each study and pooling those. As for 0.75, the conversion per ten degrees is correct as a number; what is wrong is calling an association a cause. And this many studies with one covariate is already marginal — the Cochrane rule of thumb is at least ten studies per covariate.
In the subgroup analysis the randomised-allocation group pools to an RR of 0.379 with an interval clear of 1, while the systematic-allocation group pools to 0.654 with an interval crossing 1. Can that be written up as only the randomised trials showing protection?
Show the answer and why
Correct answer: No. A subgroup difference is judged by the between-group test, which here is 1.768 on two degrees of freedom and not significant; comparing two separate tests is not a test of the difference
One group significant and another not is a comparison of two separate tests, not a test of the difference — the mistake has a name, the difference-in-significance fallacy. The fix is one more number: the between-group statistic of 1.768 on two degrees of freedom, not significant, so no difference by allocation method is detected in this data. The counts explain why it misreads so easily: the randomised group holds several times as many studies as the others, and the smaller groups were never likely to reach significance. The 0.400 is the systematic-allocation group's τ², resting on four studies; a τ² that unstable cannot support a verdict that something is noise. The 0.582 is the alternate-allocation group's RR, and the three point estimates do fall in an order, but joining three points into a trend needs the between-group test behind it, and that test is precisely the one that is not significant. Subgroup discipline is to specify them in advance, keep them few, give each enough studies, and always report the between-group test.
One model reports I² = 92.22%, τ² = 0.3132 and τ = 0.560. Explaining to a clinical colleague how inconsistent these trials are, which number speaks most directly?
Show the answer and why
Correct answer: The τ of 0.56, because it is the standard deviation of the log RR, and exponentiating it says how many times the pooled RR a typically deviating trial would show
I² is a ratio and is dragged around by how precise the studies are, so 92.22% cannot say how large the clinical gap is — the same I² corresponds to wildly different real differences in different data. The H² of 12.86 is the same identity written another way, total variation as a multiple of sampling variation, still a unitless relative quantity; turning it into a multiple has not made it translatable. The τ of 0.56 carries the units of the effect measure: it is the standard deviation of the log RR, and exponentiating it says directly how many times, or what fraction of, the pooled RR a typically deviating trial would show. A clinician understands that sentence; an I² of ninety per cent is not one. The τ² of 0.3132 is the same thing squared, in units of squared log RR, and is not a number to read aloud. Put τ or a prediction interval beside I² in the report, and the reader can judge whether the inconsistency matters.
Chapters that use this method
Watch next
統合分析的異質性(Heterogeneity)
What is Heterogeneity?
統合分析異質性和小樣本研究偏誤Sources and licences
This page is original writing