Network meta-analysis — indirect comparison and treatment ranking

A table with two drugs nobody ever compared

Clinical questions often look like this: there are ten drugs for the condition, and no trial has ever put the first one against the seventh. A network meta-analysis (NMA) exists for exactly that situation. It joins every trial into one graph — nodes are treatments, edges are comparisons somebody actually randomised — and then uses the shared comparators to work out the pairs that have no edge between them.

The arithmetic behind that is a subtraction. Given trials of A against B and of C against B,

The identity always holds arithmetically. For it to be a valid comparison of A with C, two assumptions have to hold as well:

• Transitivity: the factors that modify treatment effect (severity, age, background therapy, follow-up, outcome definition) are distributed similarly enough across the A-versus-B trials and the C-versus-B trials that the shared comparator B is interchangeable between them. This is a clinical and methodological judgement, not a test.
• Consistency: the direct and indirect evidence for the same pair are estimating the same quantity. It is the only statistically visible face of transitivity — and it is visible only on pairs that have both a direct and an indirect route.

This page uses metadat::dat.senn2013: 26 randomised trials of glucose-lowering drugs, 10 treatments including placebo, 6646 participants in total. The outcome is change in HbA1c in percentage points, and lower is better.

The raw data are arm-based long format, one row per trial arm, 53 rows in all. netmeta needs contrast-based input, so the first step is always pairwise(). That produces 28 contrasts: 25 two-arm trials contribute one each, and the single three-arm trial contributes choose(3, 2).

The evidence network: which pairs were actually tested

Three numbers to establish first:

1. The network is connected. netconnection() reports 1 subnetwork. More than one, and there is no such thing as putting all these treatments on a single table; they have to be analysed separately.
2. There are 15 designs, meaning 15 distinct combinations of arms. One of them, placebo:acarbose:metformin, is the only three-arm design.
3. Only 15 pairs carry direct evidence. 10 treatments give 45 pairwise comparisons, so 30 pairs have no head-to-head trial whatsoever.

The third point is the most important thing in the network plot, and it is the one thing the plot does not draw: an absent edge is invisible. Reading a network plot means actively counting which pairs are empty, not admiring the ones that are joined.

After pooling: the league table and its heterogeneity

The model is a random-effects netmeta (generalised least squares) with mean difference as the summary measure, placebo as the reference group, and the model told that smaller is better (small.values = "desirable").

Heterogeneity: τ² = 0.109, τ = 0.330, I² = 81.4% (72.0% to 87.7%). The total Q is 97.0 (df = 18, p < 0.001), of which the within-design part is Q = 74.4 (df = 11, p < 0.001).

An I² tells you what proportion of the dispersion is not sampling error. It does not tell you where that dispersion lives. The design-level decomposition does. The table below lists only the 5 designs with more than one trial, which are the only ones with a within-design Q to compute:

57% of the within-design heterogeneity comes from a single design, placebo:metformin (Q = 42.2, df = 2), and adding placebo:rosiglitazone accounts for 85% of it. That is far more useful than “I² is 81%”: it points at which trials’ entry criteria to go and read, rather than telling you to distrust the whole network. What each of these statistics actually measures is in Heterogeneity — I², τ² and prediction intervals.

The league table below is an excerpt, and it does not mark the empty cells

The full league table has 45 rows, and printing all of them just makes readers scroll past. What follows is an excerpt: the 9 comparisons against placebo plus the 3 comparisons among the top three, 12 rows in all. A negative value means the first treatment lowers HbA1c more than the second.

Do direct and indirect agree? Net-splitting

Consistency is the one thing this method can check about itself. netsplit() — also called SIDE, separating indirect from direct evidence — does three things for every pair that has direct evidence: estimate it from the direct trials alone, estimate it from the indirect routes alone, and test the difference between the two.

The number of pairs whose difference reaches statistical significance is 0, and the smallest p value is 0.156. That is: on this data, no inconsistency between direct and indirect evidence was detected.

Read that sentence literally. SIDE tests have very little power — most pairs here rest on one or two direct trials, both intervals are wide, and declaring them different would take an enormous gap. Not detecting inconsistency is not the same as consistency holding, in exactly the way a non-significant Q test is not evidence of homogeneity.

A pair with no inconsistency test did not pass one

The indirect column is empty for the last 4 rows above: benfluorex:placebo, miglitol:placebo, sitagliptin:placebo, vildagliptin:placebo.

The reason is that these four drugs hang off the network through placebo alone — they are leaves. There is one route from benfluorex to placebo, namely those direct trials themselves; with no second route, there is nothing to compare the direct estimate against. netsplit() returns NA for them, which is not a failed test but no test at all.

This is the mirror image of the sulfonylurea case in the previous section: one is an estimate with no direct trial, the other a direct trial with no indirect route. Both print as an ordinary cell in the league table, and the evidence behind them is of entirely different kinds.

What it means in practice is that the network point estimate for these four is essentially their own pairwise meta-analysis — the table in the next section shows the largest shift is only 0.046 percentage points. Their point estimates borrow nothing from the network and undergo no consistency check.

What does change is the interval: the network’s τ² is estimated across all designs rather than from those one or two trials, so the interval can get wider (sitagliptin, vildagliptin) or narrower (benfluorex, miglitol). Listing them in the same ranking table as the estimate for rosiglitazone, which several routes corroborate, puts two different grades of evidence side by side.

What borrowing indirect evidence costs

Network estimates are usually described as borrowing strength and therefore being more precise. On this data the comparison can be made directly: for each drug, put the pairwise meta-analysis of its placebo-controlled trials alone next to the estimate from the whole network.

Most rows shift very little, but pioglitazone runs against the intuition. Its direct evidence is 1 trial, giving MD -1.30 (-1.55 to -1.05); the network estimate is -1.13 (-1.56 to -0.70). The indirect evidence pulls the point estimate 0.17 percentage points towards the null and widens the interval (0.50 to 0.86).

An interval can widen for two reasons. An indirect route chains several contrasts together, and the uncertainty of each link accumulates; and the network’s τ² is estimated across all designs, so that whole parcel of heterogeneity is added to every cell — including a cell built on one trial that looked clean on its own. Borrowing strength does not guarantee precision. It guarantees borrowing the whole network’s uncertainty as well.

The other direction: the direct column for sulfonylurea is empty, so its network estimate exists out of nothing — a pairwise meta-analysis simply cannot produce that number. This is where the real value of an NMA lies, at the cost of that entire cell resting on transitivity.

The ranking comes last, and never without the effect size

Only now does ranking arrive. netmeta gives a P-score, the frequentist counterpart of the SUCRA (surface under the cumulative ranking curve) that Bayesian NMAs usually report; the two are equivalent by definition and normally close in value. It means the extent to which a treatment outperforms the others on average, and it runs between 0 and 1.

No difference was detected between first and second place

The P-score for rosiglitazone is 0.893 and for metformin it is 0.782. Presented as probabilities, that gap looks like something. In effect-size terms it is MD -1.23 against -1.13, a difference of 0.11 percentage points of HbA1c.

And the head-to-head network estimate for those two is already in the league table: metformin:rosiglitazone is 0.11 (-0.22 to 0.43). The interval includes zero, so the network detected no difference between first and second place. The fact that one is printed on the first row and the other on the second carries no statistical evidence in itself.

Second against third is starker. metformin has an MD of -1.127 and pioglitazone -1.129, a difference of 0.002 — and the point estimate of third place is the more negative of the two, that is, better on effect size. It ranks lower because the P-score also accounts for the uncertainty in its comparisons with every other treatment, and pioglitazone has the wider interval (0.86 against 0.60).

Three limitations of P-score and SUCRA

1. It contains no effect size. A P-score is only the extent of outperforming the others. First and second place here differ by 0.112 in P-score, which is 0.11 percentage points of HbA1c; a P-score gap of the same size on other data could correspond to an entirely different clinical difference. Whether a gap is worth switching drugs for is not a question a ranking can answer.
2. First place may be winning by very little. Already seen above: the head-to-head interval for first and second place includes zero.
3. The uncertainty in the ranking itself is far larger than it looks. A P-score is a point estimate, and papers rarely give an interval for it. What actually addresses this is the distribution of each treatment across the ranks (a rankogram), or a sensitivity analysis showing whether dropping one or two trials flips the order.

Reading a network meta-analysis

1. Where the network plot has no edges. Find out first whether the pair you care about has any direct trial. If not, that cell rests entirely on transitivity.
2. Whether the network is connected. Splitting into two subnetworks and presenting one combined table anyway is among the worst errors available.
3. Where the transitivity argument is. There should be a passage of text, or an appendix table, comparing the distribution of effect modifiers across designs. “We assume transitivity holds” is the same as writing nothing.
4. Which model the inconsistency test was computed under. When the design-by-treatment Q is highly significant, check whether it shares the heterogeneity assumption of the primary analysis. SIDE tests belong in the paper pair by pair, not as one summary p value.
5. How many pairs have no inconsistency test. Treatments attached to the network only through a shared comparator have estimates that no consistency check has ever touched.
6. Whether the primary conclusion is written in effect sizes or in ranks. An abstract that says “X ranked first” without saying what the MD of X against Y was is a warning sign.
7. Whether the ranking is presented with its uncertainty. A bare P-score or SUCRA point estimate leaves a reader no way to judge whether the order is stable.
8. Which design the heterogeneity lives in. One overall I² is not enough; only the design-level decomposition points at which trials to go and read.

Run it yourself

library(metadat)
library(netmeta)

data(dat.senn2013, package = "metadat")
d <- dat.senn2013            # arm-based: one row per trial arm

# -- Step one: arm-based -> contrast-based --
# A three-arm trial expands into choose(3, 2) rows, and those rows are
# correlated with one another. netmeta handles that correlation; it is the
# thing most easily lost when the data are reshaped by hand.
pw <- pairwise(treat = treatment, n = ni, mean = mi, sd = sdi,
               studlab = study, data = d, sm = "MD")
nrow(pw)

# -- Is the network connected? --
netconnection(treat1, treat2, studlab, data = pw)

# -- Random-effects NMA --
net <- netmeta(TE, seTE, treat1, treat2, studlab, data = pw,
               sm = "MD", common = FALSE, random = TRUE,
               reference.group = "placebo",
               small.values = "desirable")   # lower HbA1c is better
net

netgraph(net, thickness = "number.of.studies", number.of.studies = TRUE)

# -- League table: every cell gets a number, pairs nobody randomised included --
netleague(net, digits = 2)

# -- Direct versus indirect (SIDE) --
# NA in the direct column means no head-to-head trial; NA in the indirect
# column means there is no second route to compare against.
netsplit(net)

# WARNING: the "Test of inconsistency (between designs)" line printed by
#          print(net) is the COMMON-EFFECT decomposition. This page fits a
#          random-effects model; its matching test is here:
decomp.design(net)$Q.inc.random

# Which design the heterogeneity lives in (more useful than one overall I^2)
decomp.design(net)$Q.het.design

# -- Ranking last, and never on its own --
netrank(net, small.values = "desirable")
forest(net, reference.group = "placebo")

import itertools
import numpy as np
import pandas as pd

# arm-based -> contrast-based. Python handles this step comfortably; the model
# itself still has to go back to R (or be written as a hierarchical model in
# PyMC / Stan and validated yourself).
# Three of the studies in metadat::dat.senn2013, including the one
# three-arm trial, carried here so this block runs on its own. The full
# 26 are on the R side above.
arms = pd.DataFrame(
    [("Willms (1999)", "metformin",     29, -2.50, 0.862),
     ("Willms (1999)", "acarbose",      31, -2.30, 1.782),
     ("Willms (1999)", "placebo",       29, -1.30, 1.831),
     ("Baksi (2004)",  "rosiglitazone", 218, -1.20, 1.112),
     ("Baksi (2004)",  "placebo",       233, 0.10, 1.036),
     ("Alex (1998)",   "metformin",     291, 0.13, 1.428),
     ("Alex (1998)",   "sulfonylurea",  300, 0.50, 1.450)],
    columns=["study", "treatment", "ni", "mi", "sdi"],
)

rows = []
for study, g in arms.groupby("study"):
    for a, b in itertools.combinations(g.itertuples(), 2):
        rows.append({
            "studlab": study,
            "treat1": a.treatment,
            "treat2": b.treatment,
            "TE": a.mi - b.mi,
            "seTE": np.sqrt(a.sdi ** 2 / a.ni + b.sdi ** 2 / b.ni),
        })

contrasts = pd.DataFrame(rows)
print(contrasts.shape)

# Note: the rows a three-arm trial expands into are not independent, since they
# share an arm. Ignoring that correlation understates the standard errors.
# netmeta deals with it internally; a hand-written model has to do it itself.

Common misuses

The rest of the groundwork under a network meta-analysis — how an effect size and its variance are computed, what fixed and random effects are each asking, what every element of a forest plot encodes — lives in Effect sizes and their variances, Fixed-effect and random-effects models and Forest and funnel plots. The systematic review process as a whole is in Systematic review and meta-analysis.

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B7-05-network-meta.R

Watch next

▶為什麼傳遞性假設對網絡統合分析很重要？

▶利用 Stata 進行網絡統合分析

▶使用 SUCRA 進行網絡統合分析治療排名

▶元件網絡統合分析簡介

Report a problem on this page