Non-inferiority and equivalence

“Not statistically significant” cannot be read backwards

The null hypothesis of a superiority trial is that the two arms do not differ. A test that fails to reject it says this study did not detect a difference — not that the difference is zero. An underpowered trial, a trial with too short a follow-up, a trial with a blunt outcome definition: each of them hands you an interval that crosses the null, and on paper those are indistinguishable from a trial of two genuinely comparable treatments.

Claiming comparability takes a different design. You have to write down a number before any data are seen: if the new treatment is worse than the old one, how much worse may it be before you would stop using it? That number is the margin. With it, the decision rule collapses to one sentence:

Take the difference, experimental minus reference, and ask which side of the margin the upper limit of its 95% confidence interval falls on.

Throughout this page a positive difference means worse. The margin is therefore a positive number sitting on the right of the plot, and an upper limit below the margin means that even the unfavourable end of the interval is still inside what was agreed to be acceptable.

The direction on this page is reversed, on purpose

The data are the ones the design chapter Randomised controlled trial already uses: medicaldata::indo_rct, 602 patients undergoing ERCP, with post-ERCP pancreatitis as the outcome. It is a superiority trial (Elmunzer BJ, et al. N Engl J Med 2012;366:1414-22, registration NCT00820612) and it never declared a margin of any kind.

Read in the direction the trial was actually run, the risk difference is indomethacin minus placebo: -7.8 percentage points, 95% confidence interval -13.1 to -2.5. The whole interval sits to the left of zero — on the better side. A non-inferiority margin sits on the worse side, so relative to this interval every positive margin lies to the right of it: all of them are cleared, and moving the margin cannot change the verdict. Demonstrating that a different margin gives a different answer is arithmetically impossible in this direction.

So this page turns the same data around and asks a different question: could the drug be left out? That is a question trials really do ask — de-implementation. Prophylaxis costs money, staff time and side effects, and if the price of omitting it is small enough it is worth saving. Asked that way, the experimental strategy becomes no prophylaxis, the reference becomes indomethacin, and the risk difference has the same magnitude with the opposite sign: 7.8 percentage points, 95% confidence interval 2.5 to 13.1. Now the margin lands near the interval, and moving it moves the verdict.

One dataset, four margins, three verdicts

The four margins below were all invented for this page; nobody pre-specified them. Each is taken from one of the trial’s own design numbers so that they are at least defensible candidates rather than arbitrary picks. The verdicts, by contrast, are computed: the upper limit of the turned-around interval is compared with the margin, once.

Same patients, same outcome, same interval, and the verdict travels from inferior all the way to non-inferior. The only thing exchanged is one number, and that number does not come out of the data.

The verdict changes between 10 and 15 percentage points. More precisely, it changes at 13.118 percentage points — which is the upper confidence limit itself. Any margin above it yields non-inferiority; any margin below it does not.

The widest margin: non-inferior, and statistically worse

The 15 pp row carries a second lesson of its own. Its verdict is non-inferiority, yet the lower limit of the turned-around interval is 2.5 percentage points, still above zero. Under that lax margin, omitting the drug is simultaneously:

• non-inferior — the unfavourable end of the interval (13.1 percentage points) is inside what was agreed to be acceptable
• statistically worse than the reference — the interval excludes zero altogether, so this study does detect a difference, and its direction is unfavourable

The two are not in conflict, because they answer different questions. Non-inferiority asks whether the loss is small enough, not whether there is one. Every row of the stats file carries a worse_than_reference flag, and all four margins have it set.

Equivalence is two-sided, and strictly harder

Non-inferiority polices one side: the experimental arm may not be much worse, and being much better is fine. Equivalence polices both — the entire interval must lie inside plus and minus the margin. Bioequivalence studies and method-comparison studies usually ask the second question.

Of the same four limits, only the widest establishes equivalence — and the way it does is worth noting: the lower limit 2.5 sits above minus 15 percentage points and the upper limit 13.1 sits below plus 15, so the whole interval is boxed in — even though the interval itself excludes zero. Equivalence bounds the magnitude, not the direction.

Five results a non-inferiority trial can report

The figure below is a diagram, not an analysis: its five intervals were chosen to illustrate the verdicts and are not computed from medicaldata::indo_rct or from any other data, so they must not be read as results. Their use is to let you place a published interval into one of five rows.

The fourth row needs the most care. An upper limit beyond the margin means nothing has been established: neither comparability nor inferiority. The honest word is inconclusive — not comparable, and certainly not “no difference shown, so the two are interchangeable”.

The margin is a clinical judgement, not a statistical choice

Because the margin decides the conclusion, its provenance has to be defensible. Convention requires that it is:

1. Written into the protocol and fixed before recruitment. Choosing a margin after seeing the interval is painting the target around the arrow.
2. Derived from the reference treatment’s historical effect against placebo. The usual construction allows the new treatment to give up only part of the old one’s benefit — a half, say — so that it still beats no treatment at all.
3. Checked against the constancy assumption. That historical effect was measured in another era, another population, another standard of care. If today’s patients are at much lower risk, the old treatment’s effect shrinks too, and the margin has effectively widened on its own.
4. Watched for biocreep. Each generation is non-inferior to the last, and far enough down the chain a treatment may be non-inferior to something that no longer works.
5. Powered accordingly — usually a larger trial than a superiority version, because the margin is often smaller than the anticipated difference and squeezing an interval inside it takes a narrower interval. See Sample size and power.

Two intervals: Wald and Newcombe

This page uses the Wald interval for a difference of two independent proportions, because that is the one a non-inferiority report actually prints. The script also computes Newcombe’s hybrid-score interval, which combines the two Wilson intervals rather than assuming the difference itself is normal. Both in percentage points:

They differ by less than a tenth of a percentage point, so at this number of events the choice changes no verdict at any of the four margins. That statement stops there; it is not a general one. When a risk sits close to 0 or 1, or when there are far fewer events, the normal approximation degrades and the two intervals separate — and the gap may be precisely the difference between clearing the margin and not. Non-inferiority designs run into this often, because both arms are frequently at low risk (the reference treatment works), so few events and risks near zero arrive together. The interval method has to be specified in advance, for the same reason the margin does: computing both afterwards and reporting the narrower one is choosing the result.

Intention-to-treat and per-protocol: this design reverses that too

An analysis that cannot be run can still have its concept quantified. The ladder below is a thought experiment on this trial’s own numbers: if some fraction of the experimental arm did not in fact follow the assigned strategy and behaved like the reference arm, the estimate shrinks toward zero by that fraction. The standard error is held fixed so that dilution is the only thing moving.

Look at the second column from the right. Under the 10 pp margin the verdict holds at inconclusive all the way to 40% non-adherence and only flips to non-inferiority at 50%. Dilution pushes the interval toward the safe side of the margin.

The intention-to-treat half is complete on this page: 307 patients assigned to no prophylaxis and 295 to indomethacin, with an observed outcome for all 602 records and nobody dropped for missing follow-up. What is missing is the other half, and this file cannot supply it.

What to look for when reading a paper

1. What the margin is, and where it is stated. Can you find it in the protocol or the registry entry? If it appears only in the results, treat it as post hoc.
2. What justifies that margin. Is the reference treatment’s historical effect against placebo cited, and what fraction of it is being preserved?
3. The upper confidence limit. The verdict is that limit compared with the margin once, so reading the two together beats reading the concluding sentence.
4. Whether intention-to-treat and per-protocol are both reported, and agree. A non-inferiority trial reporting only one deserves a discount.
5. Whether superiority is claimed off the back of non-inferiority. That path has to be pre-specified in the testing sequence; the reverse direction is not available at all.
6. The wording of the abstract. Not statistically significant, no difference detected, non-inferior and comparable are four different claims, and the commonest failure is an abstract that turns the first into the fourth.

Run it yourself

library(medicaldata)
data(indo_rct, package = "medicaldata")
d <- indo_rct

# Experimental strategy = omitting the drug; reference = indomethacin.
# A positive difference therefore means "omitting it is worse".
n1 <- sum(d$rx == "0_placebo")
e1 <- sum(d$rx == "0_placebo" & d$outcome == "1_yes")
n0 <- sum(d$rx == "1_indomethacin")
e0 <- sum(d$rx == "1_indomethacin" & d$outcome == "1_yes")

p1 <- e1 / n1
p0 <- e0 / n0
rd <- p1 - p0
se <- sqrt(p1 * (1 - p1) / n1 + p0 * (1 - p0) / n0)
ci <- rd + c(-1, 1) * qnorm(0.975) * se
round(100 * c(rd, ci), 2)          # 7.79   2.45   13.12

# The entire decision rule, for a margin sitting on the worse side.
verdict <- function(lcl, ucl, margin) {
  if (ucl < 0) "non-inferior and superior"
  else if (ucl < margin) "non-inferior, not superior"
  else if (lcl > margin) "inferior"
  else "inconclusive"
}
for (m in c(0.02, 0.05, 0.10, 0.15)) {
  cat(sprintf("margin %4.0f pp -> %s\n", 100 * m, verdict(ci[1], ci[2], m)))
}

# The margin at which the verdict changes IS the upper confidence limit.
100 * ci[2]

# Flip the sign and you have the trial as it was run: the interval then lies
# entirely below zero, so every positive margin is cleared and none can bite.
round(100 * (-rd + c(-1, 1) * qnorm(0.975) * se), 2)

import numpy as np
from scipy.stats import norm

n1, e1 = 307, 52          # experimental strategy: no prophylaxis
n0, e0 = 295, 27          # reference: indomethacin

p1, p0 = e1 / n1, e0 / n0
rd = p1 - p0
se = np.sqrt(p1 * (1 - p1) / n1 + p0 * (1 - p0) / n0)
z = norm.ppf(0.975)
lcl, ucl = rd - z * se, rd + z * se
print(round(100 * rd, 2), round(100 * lcl, 2), round(100 * ucl, 2))

def verdict(lcl, ucl, margin):
    if ucl < 0:            return "non-inferior and superior"
    if ucl < margin:       return "non-inferior, not superior"
    if lcl > margin:       return "inferior"
    return "inconclusive"

for m in (0.02, 0.05, 0.10, 0.15):
    print(f"margin {100 * m:4.0f} pp -> {verdict(lcl, ucl, m)}")

print(100 * ucl)          # the margin at which the verdict changes

Common misuses

The page that one-line guard points at

Many places on this site stop the reader from turning “not statistically significant” into “the two are the same”, and each adds the same reminder: claiming comparability takes a margin fixed in advance, in a non-inferiority or an equivalence design (the pages differ on which, depending on whether the misreading they block is one-sided or two-sided — the t-test page says equivalence design). They run across all five families, from basic tests to evidence synthesis: t-tests and analysis of variance, the Cox proportional hazards model, the Brier score, NRI and IDI, causal mediation analysis and network meta-analysis.

A sentence can only deny the wrong reading. What this page adds is what the right one looks like: an interval, a line drawn in advance, and a claim that holds only once the two have been compared. Its home among the designs is Randomised controlled trial, whose intention-to-treat section already notes that non-inferiority reverses the direction; this page puts numbers on that remark.

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B8-06-non-inferiority.R

Report a problem on this page