AdvancedIndependently reviewed, not yet spot-checked by a human

Cumulative meta-analysis and information size

Ordering the trials by year and re-pooling after each one does two things at once. It shows when the answer was already in the evidence, and it tests the same question dozens of times along a calendar. This page uses the canonical intravenous streptokinase dataset to measure the step at which the pooled estimate stopped moving and how many people were randomised after it, to show significance arriving, leaving and returning, and to explain why a required information size is four assumptions rather than one fact.

Re-pooling after every arrival is testing the same question many times

A systematic review is not a one-off product. Once the question exists, new trials keep being published, and the authors — or Cochrane’s updating process — add them, re-run the pooled model, and look again at whether the confidence interval excludes the null. Cumulative meta-analysis simply lays that process out as a picture: order the trials by year of publication, re-pool after each one, and give each trial a row showing what the evidence looked like up to that point.

The picture says two opposite things, and both of them need saying:

Looking backwards, it shows that the pooled estimate often stopped moving at some early step, and that everything after it only narrowed the interval. This is the use cumulative meta-analysis is famous for.
Looking forwards, it is a sequence of tests of one question. Each arriving trial prompts the question “is it significant now?” — asked dozens of times, each time against the same unadjusted threshold. That is the same mechanism as multiple comparisons, with calendar time in place of endpoints and subgroups.

This page measures both of them in the same dataset.

The data: the intravenous streptokinase trials

metadat::dat.lau1992 is the canonical teaching dataset for this method: 33 randomised trials of intravenous streptokinase against control, with death as the outcome, spanning 1959 to 1988 and 36,974 participants in total. It is the dataset behind Lau et al.’s 1992 paper in the New England Journal of Medicine, and it arrives already ordered by year.

The treated arms accumulate 1,892 deaths and the control arms 2,375; the pooled control-arm risk is 12.88%. Pooling everything, the random-effects (REML) risk ratio is 0.795, 95% confidence interval 0.746 to 0.847, p < 0.001, a relative risk reduction of 20.5%. Heterogeneity is low: I² is 2.0%, τ² is 0.00079, and the Q test of 38.5 on 32 degrees of freedom does not reach statistical significance (p = 0.199).

Why the risk ratio rather than the odds ratio, and what happens to the one zero cell.

Lau et al. used a fixed-effect Peto odds ratio. This page uses the risk ratio, for two reasons. First, death is common in this question — the pooled control-arm risk is around 12.9% — and at that event rate the odds ratio sits visibly further from 1 than the risk ratio readers will take it for. A page whose whole argument is that the answer was already there should not state that answer in the measure that overstates it. Second, the required information size below is parameterised by a control event rate and a relative risk reduction, so keeping the pooled estimate on the risk-ratio scale makes the assumed reduction and the observed reduction the same quantity, directly comparable without conversion.

The cost is that the year the answer arrived moves, which is exactly why four combinations are laid side by side further down.

As for the zero cell: Baroffio 1986 recorded no deaths at all in the treated arm. The script writes add = 1/2, to = “only0” explicitly in the escalc() call, so half an event is added only to the one study with a zero cell and nothing else is touched. That is the metafor default, but stating it and inheriting it are different acts: the amount added by a continuity correction is a convention rather than a derivation, and putting it in the code is what makes the choice visible. Zero cells are treated in full on the effect measures page.

The answer was already there

Start with the whole cumulative forest plot. Each row is not one trial’s result but all the evidence up to and including that trial, pooled.

A cumulative forest plot of 33 rows, ordered by year from top to bottom, each labelled with a trial name and year and showing the risk ratio and 95% confidence interval obtained by pooling that trial with every trial above it. The top rows have very wide intervals that straddle the reference line; the intervals narrow steadily going down, and in the middle a few rows sit entirely to the left of the line before touching it again, after which they settle to the left for good. A horizontal dashed line runs the full width of the plot just below the row for European 3 (1979), splitting the evidence into a before and an after; below it the point estimates form an almost vertical line and only the intervals keep shrinking. — Every row re-pools all the trials up to that point. Below the dashed line the pooled risk ratio no longer moves materially — and 14 more trials, 31,780 more participants and 9 more years came after it.Plotting script `figures/scripts/B7-06-cumulative-meta.R`

Saying the pooled estimate stopped moving needs a criterion, or it is just narrating a picture. The script records the criterion alongside the numbers, in the stability.criterion field of the stats file: the earliest step j such that every later step both (a) has a cumulative pooled risk ratio within 10% of the final pooled value and (b) has a 95% confidence interval excluding 1.

Both halves carry weight. Condition (a) alone is met early and then broken; condition (b) alone is the thing that flickers in this dataset. And the requirement is that the condition hold at every later step, not merely at step j — otherwise a lucky step that the next trial undoes would still be declared the settling point.

On that criterion:

Quantity	Value
Settles at step	19 (European 3, 1979)
Cumulative pooled risk ratio there	0.826 (95% CI 0.701 to 0.973)
Participants accumulated by then	5,194
Trials that came after	14
Participants who came after	31,780 (86.0% of everyone)
Years that came after	9

Put plainly: 86.0% of everyone ever randomised in this question was randomised after the pooled estimate had stopped changing materially. This is the class of number cumulative meta-analysis is usually cited for, and Lau et al.’s 1992 paper is the original version of the argument.

Significance arrived, left, and came back

More telling than a settling point estimate is that significance itself flickers. Laying out whether each step’s interval excludes 1 gives this:

Step	Trial (year)	Cumulative n	Cumulative RR	95% CI	CI excludes 1	Why this step is worth a look
4	European 2 (1971)	962	0.788	0.493 – 1.258	No	Under a fixed-effect model, or a Peto odds ratio, the interval already excludes 1 here
14	Austrian (1977)	4,084	0.824	0.681 – 0.996	Yes	Under the random-effects risk ratio, the interval excludes 1 for the first time
17	N Ger Collab (1977)	4,821	0.844	0.709 – 1.006	No	Back across 1. There is nothing wrong with this trial
19	European 3 (1979)	5,194	0.826	0.701 – 0.973	Yes	The pooled estimate is settled from here on; criterion in the next section
21	GISSI-1 (1986)	18,647	0.828	0.740 – 0.926	Yes	GISSI-1, the first trial of five figures
32	ISIS-2 (1988)	36,908	0.796	0.747 – 0.849	Yes	ISIS-2, a single trial that nearly doubles the accumulated total
33	Wisenberg (1988)	36,974	0.795	0.746 – 0.847	Yes	The last step, and what is usually quoted as the meta-analysis

The interval excludes 1 for the first time at step 14 (Austrian, 1977, 4,084 participants accumulated). It holds for three steps. At step 17 (N Ger Collab, 1977) the cumulative estimate becomes 0.844 with a 95% interval of 0.709 to 1.006 — back across 1. It does not return until step 19, and only from there does it stay. Across the whole accumulation the interval crosses the null line 3 times.

Not one of the trials that removed significance is defective. They are not low-quality studies, not outliers, not something anyone slipped in. They are simply new data. This is what testing one question once per arriving study looks like: over a long sequence of looks, crossing the threshold at some point is close to inevitable, and crossing it once says little about which side of it the next look will fall on.

The mechanism is identical to the one on multiple comparisons and subgroup analyses, with the axis swapped: there it is many endpoints looked at once, here it is one endpoint looked at many times. So is the consequence — a nominal 5% type I error rate is no longer 5% once the looks are repeated.

There are two practical readings. For people reading meta-analyses: an updated review reporting that the effect has now reached statistical significance quotes a p-value that has been adjusted for none of the earlier looks, so it cannot be read at face value. For people running them: if the plan is to keep updating and to declare a conclusion at some point, that stopping rule belongs in the protocol, not in a decision taken after seeing each update.

Which year the answer arrived depends on the scale you use

The previous section says the interval first excludes 1 in 1977. That sentence holds only for the model and effect measure used on this page. Change the combination and the year moves. Same data, same criterion (the cumulative interval excludes the null), four common combinations:

Model and effect measure	Year the null is first excluded	Step	Cumulative n then	Trials after	Participants after
Random-effects (REML), risk ratio — the analysis on this page	1977	14	4,084	19	32,890
Fixed-effect, risk ratio	1971	4	962	29	36,012
Random-effects (REML), odds ratio	1977	15	4,314	18	32,660
Fixed-effect Peto odds ratio — what Lau et al. used in 1992	1971	4	962	29	36,012

Holding the risk ratio fixed, the fixed-effect and random-effects analyses differ by 6 years. The reason is not mysterious: a fixed-effect model gives the early small trials more weight and pays nothing for between-study variance, so its intervals are narrower and exclude the null sooner, whereas the random-effects model estimates a non-trivial between-study variance early on (I² exceeds forty per cent at several of the first twenty steps) and its intervals are correspondingly wider. The gap between risk ratio and odds ratio comes from the two not being equivalent when events are common.

The threshold here is a nominal two-sided 5%, adjusted for none of the earlier looks. The table is also not a reproduction of what Lau et al. reported — they quote a stricter p-value threshold, so the year in their paper is not directly comparable with any row above — it falls later than the two fixed-effect rows and earlier than the two random-effects ones, so no single statement of direction covers the table. Comparing a cell of this table against the textbook line about the answer being available by 1973 is comparing two different things.

What to keep is the shape of the table, not any single year in it. When the answer arrived is not a property of the data; it is a joint result of data, effect measure, model and threshold. When someone cites a specific year, the first question is what those four were.

Required information size: one line, four assumptions

A single trial has a sample size calculation: state the effect worth detecting and the tolerable type I and type II error rates, and the required number of participants follows. Required information size (RIS) moves the same arithmetic to a meta-analysis: how many participants does this question need in total before it is adequately powered?

The formula is the ordinary two-sided two-proportion sample size, summed over both arms:

n_{\text{arm}} = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p\,(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_1-p_2)^2}

Here $p_1$ is the assumed control event rate and $p_2$ is what it becomes under the assumed relative risk reduction. The four inputs used on this page are:

Input	Value	Where it comes from
Control event rate	12%	A round figure for short-term mortality after acute myocardial infarction in the pre-thrombolytic era, chosen a priori. The control-arm risk actually observed in these trials is 12.88%; the two are not the same object
Relative risk reduction	20%	A round target: the smallest mortality benefit a trialist of the period would plausibly have designed to detect. It is not derived from this meta-analysis
α	0.05	The conventional two-sided value, with no adjustment for the repeated looks this page is about
Power	80%	Conventional

Those give an RIS of 5,250 (2,625 per arm). This body of evidence crosses that line at step 20 (ISAM, 1986), with 6,935 participants accumulated, and 13 trials and 30,039 participants still to come.

Two stacked panels sharing one horizontal axis of cumulative participants on a logarithmic scale, running from tens of people to forty thousand. The upper panel plots the cumulative pooled risk ratio with a shaded 95% confidence band: the band is extremely wide at the left and straddles the reference line, narrows quickly as participants accumulate, and settles to the protective side of the line from the middle onwards. The lower panel plots the cumulative Z statistic for the same steps against two dashed horizontal lines at plus and minus 1.96; the curve descends past the lower dashed line, rises back above it at a point marked with a filled dot, and only then falls away for good. Both panels carry a vertical dotted line marking the required information size, positioned left of centre. — The horizontal axis is logarithmic on purpose: on a linear scale the 17,187 participants of ISIS-2 would compress the first 19 trials into a sliver at the far left, and that sliver is what this page is about. The filled dot in the lower panel is the loss of significance described in the previous section.Plotting script `figures/scripts/B7-06-cumulative-meta.R`

Claiming they separate is not worth much as an assertion, so here is the grid. Only the assumed relative risk reduction changes; the other three inputs are held fixed:

Assumed relative risk reduction	RIS (both arms)	Reached at step	Year	Trials after	Participants after
10%	22,020	32	1988	1	66
15%	9,560	21	1986	12	18,327
20%	5,250	20	1986	13	30,039
25%	3,276	13	1976	20	33,618
30%	2,218	7	1973	26	34,748

Moving from 10% to 30% takes the RIS from 22,020 down to 2,218, close to a tenfold difference, and moves the year it is reached from 1988 back to 1973. The grid is the honest presentation; a single line is not. One vertical line on a plot reads as a property of the data. Five of them let the reader see a dial.

A heterogeneity-adjusted RIS was computed but deliberately not drawn. The usual adjustment divides the RIS by $(1 - I^2)$ , on the grounds that between-study variance makes the effective information less than the nominal head count. Here I² is only 2.0%, so the adjusted figure is 5,360 against an unadjusted 5,250 — 110 participants apart, about 2.1%, and the two lines would sit on top of each other.

That near-identity is itself worth stating. The heterogeneity adjustment is not a step that always inflates the threshold substantially; what it does depends entirely on I². In a question with high I² the same adjustment can multiply the RIS several times over; in this one it changes almost nothing. When a paper reports a heterogeneity-adjusted information size, glancing at the I² beside it tells you what the adjustment actually did. (I² and how it misleads has its own page, heterogeneity.)

No O’Brien-Fleming boundary is drawn on this page, and that is deliberate. Transplanting sequential monitoring boundaries onto a cumulative meta-analysis produces what is generally called trial sequential analysis (TSA). It addresses exactly the repeated-testing problem described in the first half of this page, by overlaying boundaries that tighten as information accumulates and treating only a crossing as conclusive.

The script leaves its reason in the boundaries_note field of the stats file: whether and how a cumulative meta-analysis should be adjusted for repeated testing is still contested, and drawing one boundary here would present a single convention as settled practice.

Both sides are worth knowing. The case for it: each update really is a test, so it should pay the same price a sequential trial pays, or the moment of first significance means nothing. The case against it: the order in which evidence arrives in a meta-analysis is not an experimental design the investigator controls, so boundaries built for prospective interim analyses do not map cleanly onto it — and a TSA conclusion still rests on the RIS assumptions, which packages the sensitivity of a set of assumptions as a line that looks hard.

So this page stops at the concept: show the repeated looks, compute the information size, expose the assumptions. Faced with a paper reporting a TSA, the thing to do is check which assumptions its RIS used and which family of boundaries it drew, rather than treating crossed or not crossed as a binary conclusion.

Run it yourself

library(metafor)
library(metadat)

data(dat.lau1992, package = "metadat")
d <- dat.lau1992          # already ordered by year, which is the order that matters

# add = 1/2, to = "only0" adds half an event only to the study with a zero cell
# (Baroffio 1986, no deaths in the treated arm). It is the metafor default;
# writing it out is what makes the choice visible.
e <- escalc(measure = "RR", ai = ai, n1i = n1i, ci = ci, n2i = n2i,
            data = d, slab = paste(trial, year), add = 1/2, to = "only0")
res <- rma(yi, vi, data = e, method = "REML")

cu <- cumul(res)          # one row per step: the pooled value up to that trial
forest(cu, atransf = exp, refline = 0,
       xlab = "Cumulative pooled risk ratio (death)")

cud <- as.data.frame(cu)
cud$rr    <- exp(cud$estimate)
cud$ci_lb <- exp(cud$ci.lb)
cud$ci_ub <- exp(cud$ci.ub)
cud$sig   <- cud$ci_ub < 1        # interval excludes 1, on the protective side

# how many times the interval crossed the null line: sign changes in sig
sum(diff(as.integer(cud$sig)) != 0)

# where it settles: both conditions, and they must hold at every later step
tol <- 0.10
ok  <- abs(cud$rr / cud$rr[nrow(cud)] - 1) <= tol & cud$sig
min(which(rev(cumprod(rev(as.integer(ok)))) == 1L))

# change the scale or the model and the year of first significance moves
cumul(rma(yi, vi, data = e, method = "FE"))            # fixed-effect risk ratio
cumul(rma(measure = "PETO", ai = ai, n1i = n1i, ci = ci, n2i = n2i,
          data = d, method = "FE"))                    # the Peto OR Lau et al. used

# required information size: two proportions, two-sided, both arms summed
ris_total <- function(pc, rrr, alpha, power) {
  p1 <- pc; p2 <- pc * (1 - rrr); pbar <- (p1 + p2) / 2
  n_arm <- (qnorm(1 - alpha / 2) * sqrt(2 * pbar * (1 - pbar)) +
            qnorm(power) * sqrt(p1 * (1 - p1) + p2 * (1 - p2)))^2 / (p1 - p2)^2
  2 * ceiling(n_arm)
}
# never report just one: sweep the assumed reduction to see how sensitive it is
sapply(c(0.10, 0.15, 0.20, 0.25, 0.30),
       function(r) ris_total(0.12, r, 0.05, 0.80))

# the heterogeneity-adjusted version, reported here but not plotted: I^2 is so
# low that the adjusted line would sit on top of the unadjusted one
ceiling(ris_total(0.12, 0.20, 0.05, 0.80) / (1 - res$I2 / 100))

Verified with R 4.6.0, metafor 5.0.1 and metadat 1.6.0

import numpy as np
from scipy.stats import norm
from statsmodels.stats.meta_analysis import combine_effects

# metadat::dat.lau1992, 33 trials, already in year order -- that order is
# what a cumulative meta-analysis consumes. The 0.5 goes only to the study
# with a zero cell, matching add = 1/2, to = "only0" on the R side.
ai  = np.array([1, 4, 20, 69, 22, 19, 26, 13, 7, 6, 11, 4, 38, 37, 25, 1, 63,
                5, 18, 54, 628, 1, 0, 1, 1, 3, 3, 2, 4, 1, 12, 791, 2])
n1i = np.array([12, 21, 83, 373, 219, 164, 264, 102, 53, 55, 49, 14, 302, 352,
                123, 13, 249, 32, 156, 859, 5860, 28, 29, 19, 21, 49, 35, 107,
                52, 13, 191, 8592, 41])
ci  = np.array([4, 7, 15, 94, 17, 18, 32, 29, 3, 6, 9, 1, 40, 65, 31, 3, 51, 5,
                30, 63, 758, 2, 6, 3, 1, 6, 4, 12, 7, 2, 17, 1029, 5])
n2i = np.array([11, 21, 84, 357, 207, 157, 253, 104, 54, 53, 42, 9, 293, 376,
                107, 11, 234, 26, 159, 882, 5852, 24, 30, 19, 23, 49, 29, 112,
                55, 12, 177, 8595, 25])
a, b = ai.astype(float), (n1i - ai).astype(float)
c, d = ci.astype(float), (n2i - ci).astype(float)
zero = (a == 0) | (b == 0) | (c == 0) | (d == 0)
a[zero] += 0.5; b[zero] += 0.5; c[zero] += 0.5; d[zero] += 0.5
log_rr = np.log((a / (a + b)) / (c / (c + d)))
var    = 1 / a - 1 / (a + b) + 1 / c - 1 / (c + d)

# The accumulation has to be an explicit loop: statsmodels has no cumul().
# log_rr and var were computed from the 2x2 tables just above,
# continuity correction included.
rows = []
for k in range(1, len(log_rr) + 1):
    res = combine_effects(log_rr[:k], var[:k], method_re="dl")
    mu, se = res.mean_effect_re, res.sd_eff_w_re
    rows.append((np.exp(mu), np.exp(mu - 1.96 * se), np.exp(mu + 1.96 * se)))

rr, lb, ub = map(np.array, zip(*rows))
sig = ub < 1
print("null-line crossings =", int(np.sum(np.diff(sig.astype(int)) != 0)))

# required information size: the arithmetic is easy, the four assumptions are not
def ris_total(pc, rrr, alpha=0.05, power=0.80):
    p1, p2 = pc, pc * (1 - rrr)
    pbar = (p1 + p2) / 2
    n_arm = (norm.ppf(1 - alpha / 2) * np.sqrt(2 * pbar * (1 - pbar)) +
             norm.ppf(power) * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p1 - p2) ** 2
    return 2 * int(np.ceil(n_arm))

print([ris_total(0.12, r) for r in (0.10, 0.15, 0.20, 0.25, 0.30)])

# Note: the only random-effects estimator here is DerSimonian-Laird, not the
# REML used above, so these numbers will not match the page exactly. Use R.

Python has no equivalent of metafor's cumul(), and no ready-made required information size either; both have to be written as an explicit loop and an explicit formula. What follows is an honest hand-rolled version, not library functionality.

Reading the report

What a row of a cumulative forest plot is. Not one trial’s result but the pooled value up to that trial. Read it like an ordinary forest plot and every trial looks implausibly precise.
What the ordering is. By year is the usual choice, but people also order by sample size, by quality or by risk of bias. Anything other than year is a sensitivity analysis, not a statement about when the evidence became sufficient, and should be labelled as such.
Whether the threshold behind first significance is stated. A sentence saying the effect became statistically significant by some year, with no model, no effect measure and no p-value threshold attached, is close to unverifiable.
Whether a required information size is reported, and where its four inputs are. An RIS without a stated control event rate and target effect is not an RIS; those two numbers all but determine where the line sits.
Whether the RIS comes with a sensitivity analysis. One line is an assumption; a set of lines is a conclusion.
Whether a TSA boundary is drawn, and from which family. O’Brien-Fleming, Pocock and alpha-spending boundaries tighten at different rates, so the same data can cross one and not another.
Whether a heterogeneity-adjusted RIS is reported next to its I². Without the I² there is no way to tell whether the adjustment was substantial or cosmetic.
The wording of the conclusion. The evidence is sufficient and, under this set of assumptions the accumulated information exceeds the required information size, are two different sentences.

Common misuses

Misuse	Why it is wrong
Reading a cumulative forest plot as an ordinary one	Each row is a pooled value, not a trial; consecutive rows are heavily correlated rather than independent pieces of evidence
Quoting the year the answer was available without the model and effect measure	That year is a joint result of data, effect measure, model and threshold, and moves when any of them changes
Using a cumulative meta-analysis to blame past investigators for continuing to randomise	The criterion uses the final pooled value, information that did not exist at the time; judging in advance needs a stopping rule defined in advance
Reading the p-value at which an update first reaches significance at face value	It is adjusted for none of the dozens of earlier looks, so the nominal 5% is not the operating error rate
Declaring a conclusion when significance first appears and not mentioning that a later update lost it	That is precisely what repeated testing looks like; reporting only the favourable look is selective presentation
Reporting a required information size without its four inputs	The position of the line is almost entirely determined by the assumptions, so the reader cannot judge whether it means anything
Drawing a single RIS line with no sensitivity analysis	One line makes an assumption look like a property of the data
Treating a coincidence between the RIS and some substantive step as evidence	When the assumed and observed effects are close, the two were always going to land together
Reading accumulated participants below the RIS as proof that more trials are needed	RIS is a power calculation for one assumed effect, not a summary of evidence quality; a larger true effect needs fewer participants
Reading a TSA that has not crossed its boundary as proof of no effect	Not crossing means no conclusion has been reached under those assumptions and boundaries, not that the effect is absent
Reporting a heterogeneity-adjusted RIS without the I²	The size of the adjustment is entirely a function of I², so without it the adjustment is uninterpretable
Treating a settled pooled estimate as meaning no further research is needed	Outcomes other than death, subgroups, long-term safety and alternative regimens may all remain open

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B7-06-cumulative-meta.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

In the cumulative meta-analysis of these 33 streptokinase trials the pooled estimate settles in 1979 — from that step on it stays within a tenth of the final value and the interval no longer crosses the null — with 5194 participants accumulated. What does that establish?

Show the answer and why

Correct answer: A further 31780 people were randomised after that point, over eighty per cent of everyone in the dataset, while the pooled estimate no longer moved

What a cumulative meta-analysis is for is not the final number but the step at which it stopped changing. Here that is 1979 and 5194 participants; a further 31780 people were randomised afterwards, over eighty per cent of the dataset, while neither the pooled estimate nor its interval moved materially. The 5250 is this page's hypothetical required information size, and its landing within a few dozen people of 5194 is a coincidence: the formula was set to a twenty per cent relative risk reduction and this data happens to measure about twenty per cent, so the two were always going to coincide, and they separate the moment the target changes. The 4084 is the step at which the interval first excluded the null — but the interval crossed back afterwards, so reaching significance first is not the same as settling, which is what the next question looks at.

On this cumulative plot the interval first excludes the null in 1977 at step 14. By step 17, still 1977, the upper limit is back at 1.006 and the interval crosses the null again. What does that mean?

Show the answer and why

Correct answer: That significance flickers along the way: the final value is 0.795, the interval crossed the null three times en route, and looking at the result after every added trial is thirty-odd unadjusted repeated tests

A cumulative meta-analysis recomputes and looks at the result after every added study, thirty-odd times in this data, each time at an unadjusted nominal level. The consequence is on the plot: the interval excludes the null, then crosses back, three crossings in all. The 0.844 is the pooled estimate at the step where it crossed back, and it is not an outlier — it is every study available at that moment combined. Relabelling an inconvenient interim state as an outlier is exactly the behaviour repeated looking invites. The 0.795 is the final value, and the direction never reverses; what reverses is the binary verdict of significance, which is precisely where repeated testing does damage. The 0.847 is the final upper limit, and using the final result to vindicate each interim reading is hindsight. It is also why this page draws no sequential monitoring boundary: how to adjust for repeated looks is still contested, and drawing the line would present one convention as settled.

The question of which year the evidence became sufficient gets different answers on the same data depending on the model: a fixed-effect risk ratio reaches significance in 1971, a random-effects one not until 1977. What does that show?

Show the answer and why

Correct answer: That sufficient evidence is a verdict contingent on the model and the effect measure: an odds ratio under the same random-effects model crosses at 4314 accumulated participants, and four combinations give four answers

The same data and the same threshold of an interval excluding the null move the answer from 1971 to 1977 when the model or the effect measure changes. An odds ratio under random effects crosses at 4314 accumulated participants, a risk ratio under random effects at a different step, and both fixed-effect versions earlier still — four combinations, four years. So the claim that the evidence was sufficient by a given year carries a model assumption inside it rather than being a property of the data, and a report has to state the model, the measure and the threshold together. The 36012 and the 32890 are the people still recruited after each model calls it sufficient, and treating those as waste inverts cause and effect: a model is chosen for whether its assumptions hold, not for how quickly it turns significant, and choosing on which turns significant first is choosing the analysis after seeing the result. None of these verdicts is adjusted for the repeated looks either.

The required information size computed on this page is 5250, and the data crosses it in 1986 at step 20. Can that number be written up as how many people this question needs?

Show the answer and why

Correct answer: No. It rests entirely on four assumptions, and the relative risk reduction it targets is the most sensitive: set that target to ten per cent and the requirement becomes 22020, cleared only by the very last trial

A required information size transplants the sample size formula for a single trial onto a meta-analysis: control event rate, the relative risk reduction worth detecting, the significance level and the power are all assumptions chosen in advance, not quantities measured in this data. Targeting twenty per cent gives 5250; targeting ten per cent gives 22020, cleared only by the final trial; targeting thirty per cent gives 2218, cleared more than a decade earlier. One dataset, three entirely different moments at which the evidence became sufficient. The 5360 is the version adjusted for I², which is tiny here, so the two differ by about two per cent — that is not where the sensitivity lives; the four assumptions are. The number therefore travels only with its assumptions, ideally alongside a grid like this page's; quoted alone, a reader has no way to tell what it says.

The final pooled risk ratio across these 33 trials is 0.795 (95% CI 0.746 to 0.847) with an I² of only 2.0%. Given how clear the final conclusion is, what does the cumulative version add?

Show the answer and why

Correct answer: It adds a time axis, naming the step after which the estimate stops changing — the 0.826 in 1979 is already within a tenth of the final value, and fourteen trials were still to come

An ordinary meta-analysis answers what the conclusion is once every study is combined. The cumulative version answers one more question: from which moment that conclusion already held. Here it is the 0.826 in 1979, after which every step stays within a tenth of the final value and the interval never crosses the null again, with fourteen trials and thirty thousand-odd people still to come. The 0.001 is the final τ², and heterogeneity really is negligible — but that says the studies agree with each other, not that the plot carries nothing: the same conclusion reached at different moments has entirely different clinical and ethical weight. The 0.229 is the first trial's estimate, with an interval so wide it carries almost nothing; early instability is what small samples do, not grounds for excluding those studies, and excluding studies by their results after the fact is the way this method is most easily misused.

After 1979 this dataset added 14 more trials and 31780 more people while the pooled estimate did not move materially. What verdict does that support about those later trials?

Show the answer and why

Correct answer: No blanket verdict: this page's required information size is not cleared until 6935 accumulated participants, and a settled estimate and a sufficient information size are two different criteria

This is where a cumulative meta-analysis is most easily over-extended. What the plot shows is retrospective: the estimate stopped moving materially after 1979, and 14 trials and 31780 people followed. Getting from there to those trials were waste needs several further premises — nobody was running this analysis at the time, the trials differ in timing of administration and in population, and a settled estimate is a different criterion from a sufficient information size, which on this page is not cleared until 6935 accumulated participants. As for the 36974, that is everyone in the dataset, and saying every trial was equally necessary goes to the opposite extreme: it claims the cumulative plot carries no information, when what it shows is that the later half contributed a narrower interval rather than a different conclusion. The defensible reading is that this plot turns whether another trial is needed into a question that can be examined in advance, rather than a scorecard for decisions already made.

Chapters that use this method

Systematic review and meta-analysis

Sources and licences

This page is original writing