ExpertIndependently reviewed, not yet spot-checked by a human

Heterogeneity — I², τ² and prediction intervals

Why I² can run from around half to nearly a hundred per cent while the real differences between studies stay exactly the same, whether a non-significant Q test is evidence of homogeneity, why a prediction interval is closer to what readers actually want to know than a confidence interval is, and the trap in meta-regression where a study-level association gets read as an individual-level one.

Three statistics, measuring three different things

I² appears in nearly every meta-analysis, τ² and Q occasionally, H² rarely. They are related by a set of identities, but they answer different questions:

StatisticDefinitionUnitWhat it answers
QWeighted sum of squared deviations of the studies from the pooled valuenone“Is the observed spread larger than sampling error alone would produce?” — this is a test statistic
τ²Variance of the true effects across studiessquared units of the effect measure (here, squared log RR)“How widely are the true effects spread?” — an absolute quantity
Total variation ÷ sampling variation (see the note below on how it is estimated)none“How many times sampling error is the total spread?” — its floor is 1
Percentage of the total variation that is between-study variation%“What proportion of the spread is not sampling error?” — a proportion

On these 13 BCG vaccine trials:

StatisticValueHow to read it
Q152.2 (df = 12, p < 0.001)The spread is far beyond what sampling error can account for
τ²0.3132The variance of the log RR across studies
τ0.560The standard deviation of the log RR across studies — a more useful number than τ²
12.86The total variation is 12.9 times the sampling variation
92.2%About nine tenths of the total variation is not sampling error

I² is a proportion, so precision drags it around

This is where I² is most often misread, and one experiment settles it: hold τ² fixed at the observed value and multiply every study’s within-study variance by a constant (as if imagining these trials had enrolled more or fewer people), then watch what I² does.

Within-study variances × τ² (held fixed)
0.10.3132119.5699.2%
0.250.313248.4297.9%
10.313212.8692.2%
40.31323.9674.8%
100.31322.1954.2%

The real differences between the effects are identical throughout — τ² is the same number in every row — yet I² travels from 54.2% all the way to 99.2%. The only thing that changed is how precisely each study measured.

A non-significant Q test does not mean homogeneity

The null hypothesis of the Q test is that every study’s true effect is the same. It is a test, so it has a power problem — and in meta-analysis the sample size is the number of studies kk, which is usually small.

A simulation draws the power curve: each study’s within-study variance is resampled from the 13 variances actually observed here, the true τ² is fixed at four levels, and only kk varies.

A line chart. The horizontal axis is the number of studies, from 3 to 30; the vertical axis is the simulated percentage of Q tests reaching significance. The lowest line is a nearly flat grey dashed line at around the 5% significance level, the no-heterogeneity scenario. Two further lines above it correspond to low-moderate and moderate heterogeneity and rise slowly with the number of studies; the moderate one only just reaches 80.5% at 30 studies. The topmost red line corresponds to the high heterogeneity of the BCG data itself, already 75.1% at 5 studies and 92.8% by 8.
Each point is 4000 simulations. The grey dashed line is the scenario with a true τ² of 0; it stays at 5%, which shows the test itself is not broken — it simply cannot see anything when heterogeneity is moderate and studies are few.Plotting script figures/scripts/B7-03-heterogeneity.R
Number of studies ktau2 = 0 (no heterogeneity)I2 about 25%I2 about 50%BCG data (I2 about 92%)
35.2%9.1%14.6%53.4%
54.7%12.0%24.4%75.1%
85.0%16.1%35.0%92.8%
104.9%18.5%42.2%96.4%
154.5%22.9%56.3%99.5%
204.8%27.9%66.8%100.0%
304.3%36.9%80.5%100.0%

In a world where the true I² is about 50%, with 10 studies included, the Q test reaches significance only 42.2% of the time. Which is to say that more than half the time the authors will see a non-significant Q and write “no heterogeneity was found between studies” — and be wrong.

Prediction intervals: the question readers are actually asking

A confidence interval answers “where is the average effect μ”. But the clinical question is usually a different one: “where will my next group of patients, the next centre, the next trial, end up?”

That question is answered by a prediction interval, which carries both the uncertainty in μ and the spread of the effects themselves:

μ^±cτ^2+SE(μ^)2\hat{\mu} \pm c \sqrt{\hat{\tau}^2 + \mathrm{SE}(\hat{\mu})^2}

Two versions of that quantile cc are in circulation, and neither of them is simply “whatever the software defaults to”. Higgins and colleagues’ original version uses a tt distribution on k2k - 2 degrees of freedom; metafor’s predict() defaults to the normal quantile 1.96. When kk is large the two barely differ; when kk is small they differ a lot, so the table below lists both — and a paper quoting a prediction interval has to say which one it used.

On the BCG data, the difference between the prediction interval and the confidence interval is decisive:

IntervalLowerUpperCrosses 1?
95% confidence interval (where the average effect lies)0.3440.696No
95% prediction interval (where the next study will land) — normal quantile, metafor’s default0.1551.549Yes
95% prediction interval — t quantile, k − 2 degrees of freedom0.1341.785Yes
A forest plot of the 13 BCG trials ordered by effect size, with box sizes showing the random-effects weights. Two rows sit side by side at the bottom: a blue diamond for the pooled estimate and its confidence interval, lying entirely to the left of the line of no effect, and a red horizontal bar for the prediction interval, which extends noticeably further in both directions and crosses the line of no effect.
Two intervals from one model. The pooled RR is 0.489 with a confidence interval of 0.344–0.696, entirely to the left of 1; the prediction interval (the normal-quantile version, which is what the figure draws) of 0.155–1.549 crosses 1. On the log scale the prediction interval is 3.3 times as wide as the confidence interval, or 3.7 times using the t quantile.Plotting script figures/scripts/B7-03-heterogeneity.R

This sentence is worth writing out in full: on average across these trials, BCG significantly reduces the risk of tuberculosis; but on the present evidence, a new trial run at a new site could perfectly well land on the side where no protective effect is detected. Both halves are true, and reporting only the first is selective presentation.

Explaining heterogeneity: meta-regression

The right response to heterogeneity is not to suppress it but to ask where it comes from. The BCG dataset became a teaching classic precisely because its source is visible: the latitude of the trial site.

ablat is the absolute latitude of each trial’s location. Put it in the model as a covariate:

QuantityValue
Change in log RR per degree of latitude-0.0291 (SE 0.0072, p < 0.001)
Converted: RR multiplied per ten degrees of latitude0.748
R² (between-study variance explained)75.6%
Residual τ²0.0764 (was 0.3132)
Residual I²68.4% (was 92.2%)
Test for residual heterogeneity, QE30.7 (p 0.001)
A bubble plot. The horizontal axis is the absolute latitude of the trial site and the vertical axis is that trial's log risk ratio; each circle is a trial, sized by its weight, with a clearly downward-sloping regression line and a pale confidence band drawn over them. Trials at low latitude cluster near 0, where no protective effect is detected, while trials at high latitude mostly sit at distinctly negative values. Two trials lie well away from the trend line.
Latitude explains 75.6% of the between-study variance. The usual explanation is that environmental exposure to non-tuberculous mycobacteria is greater at low latitudes, so the population already has some cross-immunity and there is less left for BCG to add. Note that the residual I² is still 68.4% — latitude has not explained the heterogeneity away.Plotting script figures/scripts/B7-03-heterogeneity.R

Subgroup analysis: the coarse version of the same thing

Splitting the studies by a categorical variable and pooling each group separately is the commonest way of exploring heterogeneity. Splitting by alloc (how the trial allocated its participants):

GroupkPooled RR95% CIτ²95% prediction interval (normal quantile)
Alternate allocation20.5820.335–1.01182.0%0.1330.236–1.435
Random allocation70.3790.221–0.65089.9%0.3930.099–1.449
Systematic allocation40.6540.323–1.32486.4%0.4000.157–2.725

τ² has to be read together with k. The Alternate allocation group, for instance, has only k = 2, so its τ² of 0.133 is an extremely unstable model estimate and cannot be taken as a stable statement about absolute heterogeneity.

The point estimates look very different (from 0.379 to 0.654), but the formal test for a difference between groups does not reach statistical significance: QM = 1.77, df = 2, p = 0.413.

The ecological fallacy: a study-level association is not an individual-level one

This is the error easiest to commit in your own analysis, and it is a limitation shared by meta-regression and subgroup analysis alike.

The latitude regression line above says that “trials at higher average latitude observed better vaccine effects”. It does not say that “within a single trial, the participants who lived further north were better protected after vaccination”. The first is an aggregate-level association, the second an individual-level one, and the two can differ completely — even in direction.

The version you meet more often in clinical work: a meta-analysis finds that “trials with a higher mean age showed larger treatment effects”, and this gets written up as “older patients benefit more”. That inference might be true, but this data cannot support it — trials with a higher mean age may also have had stricter entry criteria, more comorbidity, longer follow-up, or different dosing.

Run it yourself

library(metafor)
library(metadat)

data(dat.bcg, package = "metadat")
d <- escalc(measure = "RR", ai = tpos, bi = tneg, ci = cpos, di = cneg,
            data = dat.bcg, slab = paste(author, year, sep = ", "))

res <- rma(yi, vi, data = d, method = "REML")
res                        # I^2, H^2, tau^2 and Q all arrive in one output

# Prediction interval: the pi.lb / pi.ub columns; transf = exp returns to the RR scale
predict(res, transf = exp)                       # normal quantile 1.96 by default
predict(res, transf = exp, pi.type = "riley")    # the t(k-2) version (Higgins et al. 2009)

# I^2 is a proportion: multiply every within-study variance by 4, pin tau^2, watch I^2 fall
rma(d$yi, d$vi * 4, method = "DL", tau2 = res$tau2)$I2

# -- Meta-regression: explaining heterogeneity with latitude --
mr <- rma(yi, vi, mods = ~ ablat, data = d, method = "REML")
mr                         # R^2, residual I^2 and the residual heterogeneity test QE are all in here
regplot(mr, xlab = "Absolute latitude", las = 1)   # the bubble plot

# -- Subgroups: the between-group difference is QM, not a comparison of two p-values --
rma(yi, vi, mods = ~ factor(alloc), data = d, method = "REML")

# For a pooled value within each group, fit them one at a time (with their own prediction intervals)
for (lv in levels(factor(dat.bcg$alloc))) {
  r <- rma(yi, vi, data = d, subset = (dat.bcg$alloc == lv), method = "REML")
  print(predict(r, transf = exp))
}

Verified with R 4.6.0, metafor 5.0.1 and metadat 1.6.0

Reading the report

  1. Whether τ² (or τ) is given alongside I². With only I², a reader cannot judge how large that “inconsistency” is clinically.
  2. Whether there is a prediction interval. A random-effects model reporting only a confidence interval has left out the thing the model is describing. Failing that, estimate it roughly from τ: two standard deviations of the effect distribution is about a factor of exp(±2τ).
  3. The wording used when Q is not significant. “No heterogeneity was found” and “no statistically significant heterogeneity was detected” are methodologically different sentences, and only the second is defensible.
  4. Whether subgroup differences come with a between-group test. Listing each group’s own p-value without QM is where the difference-in-significance fallacy breeds.
  5. Whether subgroups and meta-regressions were pre-specified or dug out afterwards. The protocol (a PROSPERO registration number) is something you can go and check yourself.
  6. Whether a study-level association has been written up as a patient-level conclusion. When you see “older / female / more severely ill patients benefited more”, first check whether the data were IPD or study-level covariates.

Common misuses

MisuseWhy it is wrong
Interpreting I² against fixed cut-points (25/50/75%)The same I² corresponds to very different clinical spreads on different data; Cochrane says explicitly not to use them as cut-points
Describing “how large the differences in effect are” with I²I² is a proportion and is driven by how precise the studies are; absolute size means τ² or the prediction interval
Writing “no heterogeneity” when Q is not significantPower is low when studies are few; the most that can be said is “no statistically significant heterogeneity was detected”
Choosing fixed or random effects from the Q test’s p-valueData-driven model selection, resting on a test with very little power
Reporting a random-effects confidence interval with no prediction intervalThe confidence interval is about the average; readers want to know where the next study will land
Reporting a prediction interval from very few studies and drawing conclusions from itτ² is badly estimated, the interval is uninformatively wide, and the normality assumption cannot be checked
Comparing “which subgroup was significant”The difference-in-significance fallacy; use the between-group QM
Carving out subgroups after the fact and treating them as findingsA subgroup that was not pre-specified is a hypothesis, not a discovery
Loading a meta-regression with many covariatesThe sample size is the number of studies; each covariate needs about ten studies behind it
Reading a study-level covariate association as patient-level effect modificationThe ecological fallacy; answering that question requires IPD-MA
Deleting outlying studies to bring I² downThis lets the result decide the inclusion criteria, and outliers are often the most informative studies of all

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B7-03-heterogeneity.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

Hold τ² for these 13 BCG trials fixed at 0.3132 and multiply every study's within-study variance by a factor. What happens to I²?

Show the answer and why

Correct answer: At ten times the variances I² falls to 54.25% — the true differences between studies have not moved at all, only how precisely each one was measured

I² is a ratio: τ² on top, τ² plus the typical within-study variance underneath. The numerator is nailed down all the way through this table while the denominator follows how precise the studies are, so one and the same set of true differences can show as 54.25% or as 99.16%. Two reading rules follow. A set of large, precise trials will show a high I² even when the true differences are clinically negligible; a set of small, imprecise studies can show a low I² even when the true effects are wildly different, because sampling error has inflated the denominator. The 92.22% is simply the row computed from the observed variances — a cell in this table, not a fixed constant. To answer how large the differences actually are you need an absolute quantity with units, τ² or a prediction interval; and the low-medium-high thresholds are an illustration from one methods paper, which the Cochrane Handbook now explicitly says not to treat as cut-offs.

A simulation shows that with a true I² of about 50% and 10 studies, the Q test reaches significance only 42.2% of the time. What does that say about choosing a fixed-effect model because the Q test was not significant?

Show the answer and why

Correct answer: It says non-significant means undetected this time: one step less heterogeneity, still 10 studies, and the rate is down to 18.5% — choosing a model on this test lets power choose the model

The sample size of a Q test is the number of studies, and that number is usually small. With a true I² near 50% and 10 studies it reaches significance 42.2% of the time; one step less heterogeneity and only 18.5%. A non-significant Q therefore says not detected here, not absent, and using it to pick a model makes three errors at once: treating undetected as absent, deciding on a low-powered test, and letting the data choose the model. The 4.9% is the significance rate when the true τ² is zero, which sits exactly at the nominal level and shows the test is not broken — it simply cannot see moderate heterogeneity in a small set, and those are different things. The 80.5% belongs to the row with the most studies, where one in five is still missed, and most clinical meta-analyses never assemble that many. The honest wording is that no statistically significant heterogeneity was detected in this analysis, reported alongside τ² and a prediction interval.

In one random-effects model the pooled RR of 0.489 has a 95% confidence interval of 0.344 to 0.696, entirely to the left of 1. Why report a second interval that does cross 1?

Show the answer and why

Correct answer: Because it is a prediction interval, answering where one new trial at one new site would land: its upper bound reaches 1.549, so landing on the no-detected-protection side is entirely possible

A confidence interval answers where the mean effect lies; a prediction interval answers where the next study will land, and it carries both τ² and the standard error of the mean. So the honest report is two sentences: on average across these trials BCG significantly lowers the risk of tuberculosis, and on the present evidence a new trial at a new site could perfectly well land on the side where no protective effect is detected — that is what the upper bound of 1.549 says. Reporting only the first sentence is selective presentation. The 3.270 is the ratio of the two widths on the log scale, a consequence rather than a reason: the prediction interval is not a widened confidence interval, it answers a different question. The 1.785 is the upper bound with t quantiles; the two versions differ in the quantile chosen, so a citation should say which was used rather than pick the wider one, since picking is deciding after seeing the result.

Adding the absolute latitude of the trial site to a meta-regression gives R² of 75.6% with a residual I² of 68.4%. What claim does that support?

Show the answer and why

Correct answer: An associational one only: the residual I² is still 68.39%, and the covariate is observed rather than assigned — era, strain and diagnostic criteria all move with latitude

R² says how much between-study variance latitude explains; it is not evidence of causation. The covariate is observed, not randomised: latitude travels with era, nutrition, strain, diagnostic criteria and study quality, and a meta-regression cannot separate them, so the conclusion is observational and hypothesis-generating even when every included study was a randomised trial. The residual I² of 68.39% is a further reminder that latitude has not explained the heterogeneity away. The third option commits the ecological fallacy: this regression line says that trials at higher average latitude observed larger effects, not that within a trial the people living further north were better protected. Answering the individual-level question takes an individual participant data meta-analysis, estimating the interaction inside each study and pooling those. As for 0.75, the conversion per ten degrees is correct as a number; what is wrong is calling an association a cause. And this many studies with one covariate is already marginal — the Cochrane rule of thumb is at least ten studies per covariate.

In the subgroup analysis the randomised-allocation group pools to an RR of 0.379 with an interval clear of 1, while the systematic-allocation group pools to 0.654 with an interval crossing 1. Can that be written up as only the randomised trials showing protection?

Show the answer and why

Correct answer: No. A subgroup difference is judged by the between-group test, which here is 1.768 on two degrees of freedom and not significant; comparing two separate tests is not a test of the difference

One group significant and another not is a comparison of two separate tests, not a test of the difference — the mistake has a name, the difference-in-significance fallacy. The fix is one more number: the between-group statistic of 1.768 on two degrees of freedom, not significant, so no difference by allocation method is detected in this data. The counts explain why it misreads so easily: the randomised group holds several times as many studies as the others, and the smaller groups were never likely to reach significance. The 0.400 is the systematic-allocation group's τ², resting on four studies; a τ² that unstable cannot support a verdict that something is noise. The 0.582 is the alternate-allocation group's RR, and the three point estimates do fall in an order, but joining three points into a trend needs the between-group test behind it, and that test is precisely the one that is not significant. Subgroup discipline is to specify them in advance, keep them few, give each enough studies, and always report the between-group test.

One model reports I² = 92.22%, τ² = 0.3132 and τ = 0.560. Explaining to a clinical colleague how inconsistent these trials are, which number speaks most directly?

Show the answer and why

Correct answer: The τ of 0.56, because it is the standard deviation of the log RR, and exponentiating it says how many times the pooled RR a typically deviating trial would show

I² is a ratio and is dragged around by how precise the studies are, so 92.22% cannot say how large the clinical gap is — the same I² corresponds to wildly different real differences in different data. The H² of 12.86 is the same identity written another way, total variation as a multiple of sampling variation, still a unitless relative quantity; turning it into a multiple has not made it translatable. The τ of 0.56 carries the units of the effect measure: it is the standard deviation of the log RR, and exponentiating it says directly how many times, or what fraction of, the pooled RR a typically deviating trial would show. A clinician understands that sentence; an I² of ninety per cent is not one. The τ² of 0.3132 is the same thing squared, in units of squared log RR, and is not a number to read aloud. Put τ or a prediction interval beside I² in the report, and the reader can judge whether the inconsistency matters.

Watch next

統合分析的異質性(Heterogeneity)
繁中杜裕康老師研究室· 11 minThe clearest Traditional Chinese treatment of heterogeneity there is, taught by a biostatistics professor at National Taiwan University. Worth watching before the first two sections here.
What is Heterogeneity?
ENTerry Shaneyfelt· 9 minThe clinician's angle, separating clinical, methodological and statistical heterogeneity. This page only deals with the third kind, but the decision to pool at all rests on the first two.
統合分析異質性和小樣本研究偏誤
繁中杜裕康老師研究室· 10 minPicks up where the first video leaves off and ties heterogeneity to small-study effects, which leads straight into the last section here and into B7-04.

Sources and licences

This page is original writing

Report a content problem

The statistics on this site are written by AI and reviewed by AI; a human only spot-checks. What you can see may be what we cannot.

The more specific, the more fixable — e.g. which sentence disagrees with which textbook or paper.

Needed only if you want a reply; reports without it are still read.

Sent along with your report

These are attached automatically. You can drop any of them.