Randomised controlled trial

What this chapter answers

You open a randomised controlled trial (RCT). There is a CONSORT flow diagram, a Table 1, a p-value attached to the primary outcome, and a row of subgroup estimates in a forest plot. This chapter is about what each of those is defending against, and which cell should make you stop reading and look harder.

One real trial runs through the whole chapter: Elmunzer and colleagues, NEJM 2012, testing whether a rectal indomethacin suppository given right after ERCP prevents pancreatitis. It was chosen for three reasons — its participant-level data are public (one line from the medicaldata package), its design includes an interim analysis and an early stop that textbooks rarely demonstrate, and it happens to contain an unusually clean illustration of intention-to-treat.

What randomisation actually buys

The hardest problem in observational research is that people who receive a treatment differ from people who do not, before any treatment is given. Clinicians prescribe to patients who appear to need it, so “received the drug” carries information about “was sicker” and the two cannot be separated afterwards. This is confounding by indication.

Randomisation solves it crudely but completely: assignment is decided by a random number and by nothing about the patient. In expectation the arms then match on every characteristic — the ones you thought of, the ones you did not, and the ones nobody can measure. That last category is the real prize. Statistical adjustment can only handle confounders you measured; randomisation handles the ones you never knew existed.

In this trial, randomisation happened after the ERCP was finished. That looks odd until you see the eligibility criteria: half of them are procedural facts nobody knows in advance — more than eight cannulation attempts, a precut sphincterotomy — so a patient’s eligibility is not determined until the procedure is over.

Concealment and blinding are two different defences

Three separate things hide inside the word “randomised”, and they fail in different ways. Randomisation decides who goes into which arm by chance. Allocation concealment keeps the upcoming assignment unknowable to whoever is entering patients, right up to the moment a patient is entered. Blinding keeps the assignment unknown to people after it has been made. A trial can do the first and skip the other two, and its abstract will still say “randomised”.

Allocation concealment protects against selection bias. If the person recruiting can foresee what the next assignment will be, they can act on it — wait for a healthier patient before entering the next one, discover that this one is not eligible after all, or reorder the queue. None of that requires bad faith; a clinician who believes the drug works will find it hard not to steer the sicker patient towards it. And the damage is done at the moment of allocation: the arms stop being comparable, which is exactly the confounding by indication that the section above said randomisation had solved. In this trial the randomisation schedule was generated centrally at the coordinating centre and stratified by site. The paper does not describe how each assignment reached the procedure room, so how tightly it was concealed is not something this page can assert; what it does say is that randomisation happened only after the ERCP was complete and eligibility settled, which removes the recruiter’s opportunity to act on a foreseen assignment in the first place. Central randomisation, pharmacy-controlled allocation, and sequentially numbered opaque sealed envelopes are the usual mechanisms; “sealed envelopes” without the numbering and the opacity is not one of them.

Blinding protects against something else — what happens after the assignment. Ask who was kept unaware, one group at a time: the patients, the clinicians treating them, whoever ascertains and adjudicates the outcome, and the analysts. Patients who know their arm report symptoms differently; clinicians who know it hand out co-interventions differently (performance bias); outcome assessors who know it classify borderline cases differently (detection bias). This trial used identical-appearing placebo suppositories, and the staff who cared for patients after the procedure and recorded the follow-up data were unaware of the assignment. That matters here because the primary outcome is a composite of new upper abdominal pain, an enzyme threshold, and a decision to keep the patient in hospital — and two of those three run through clinical judgement.

Step one: the CONSORT flow diagram

The flow diagram answers one question: between “somebody was looked at” and “somebody was counted in the analysis”, how many people were lost, and why.

Start with what a standard one looks like. The diagram below follows the CONSORT boxes, but every count in it was invented to show the structure — none of it is this trial’s, or any published trial’s:

Four stages, top to bottom: Enrolment (assessed, excluded with reasons, then randomised), Allocation (how many went to each arm, and how many in each arm did and did not receive what they were allocated), Follow-Up (lost to follow-up and discontinued, per arm), and Analysis (analysed and excluded from analysis, with reasons, per arm).

The relationship between the last two stages is worth noting now. In the illustration, 5 patients in the intervention arm discontinued the allocated treatment and were still counted in the analysis; the ones who actually left the denominator are the 9 with no primary outcome data. That is intention to treat drawn as boxes, and step four below says it in words.

It has two halves. The lower half — everything from Allocation down — can be rebuilt from participant-level data:

library(medicaldata)
data(indo_rct, package = "medicaldata")

# Arms after randomisation
table(indo_rct$rx)

# Primary outcome (post-ERCP pancreatitis) by arm
table(indo_rct$rx, indo_rct$outcome)

import statsmodels.api as sm

# medicaldata is an R package; Rdatasets serves the same file to Python
indo = sm.datasets.get_rdataset("indo_rct", "medicaldata").data

print(indo["rx"].value_counts())
print(indo.groupby(["rx", "outcome"]).size())

That gives 602 patients randomised — 295 to indomethacin and 307 to placebo — with follow-up complete in both arms.

The upper half is not obtainable. Those are the top two boxes of the diagram above: assessed for eligibility, and excluded with its list of reasons. The illustration puts 1,240 and 690 in them; for the real trial in this chapter, this page cannot fill them in. How many were screened, how many excluded, and for what reasons, are not in this file, because a participant-level dataset contains only those who were randomised; everyone screened out was never a row. We also read the published Methods and Results in full: those counts appear in no sentence anywhere. They exist only inside the Figure 1 image.

The only way through is to open the figure and read the counts off it. How to read a clinical research paper does exactly that, and redraws the counts as a flow diagram. Doing so surfaces something worse: even the published Figure 1 does not start at assessed for eligibility. Its top box is the 799 patients who gave consent, and the number of people who were looked at is recorded nowhere in the paper at all.

Step two: Table 1, and the p-value that should not be there

Table 1 lists baseline characteristics by arm. The question you are asking is: do these arms look similar enough? But the way to ask it is not to look for p-values.

Significance testing across arms in an RCT is common and wrong, for a simple reason: the null hypothesis — that the arms were drawn from the same population — is true by construction under randomisation. Testing something you already know to be true yields no information, and any significant result is a type I error by definition.

What to look at instead is the standardised mean difference (SMD). It does not inflate with sample size, and the convention is that |SMD| < 0.1 counts as balanced. The same logic reappears in propensity score matching; the method page covers it.

Step three: the primary outcome and effect measures

n_indo <- sum(indo_rct$rx == "1_indomethacin")
n_plac <- sum(indo_rct$rx == "0_placebo")
e_indo <- sum(indo_rct$rx == "1_indomethacin" & indo_rct$outcome == "1_yes")
e_plac <- sum(indo_rct$rx == "0_placebo"      & indo_rct$outcome == "1_yes")

p_indo <- e_indo / n_indo          # risk on treatment
p_plac <- e_plac / n_plac          # baseline risk

arr <- p_plac - p_indo             # absolute risk reduction
nnt <- 1 / arr                     # number needed to treat
rr  <- p_indo / p_plac             # risk ratio
rrr <- 1 - rr                      # relative risk reduction

fisher.test(matrix(c(e_indo, n_indo - e_indo,
                     e_plac, n_plac - e_plac), nrow = 2))

from scipy.stats import fisher_exact

n_indo = (indo["rx"] == "1_indomethacin").sum()
n_plac = (indo["rx"] == "0_placebo").sum()
e_indo = ((indo["rx"] == "1_indomethacin") & (indo["outcome"] == "1_yes")).sum()
e_plac = ((indo["rx"] == "0_placebo")      & (indo["outcome"] == "1_yes")).sum()

p_indo, p_plac = e_indo / n_indo, e_plac / n_plac
arr = p_plac - p_indo
nnt = 1 / arr
rr  = p_indo / p_plac

odds_ratio, p_value = fisher_exact([[e_indo, n_indo - e_indo],
                                    [e_plac, n_plac - e_plac]])

The gap between RRR and ARR is the row to stop on. The same data described as “a 46% reduction” and as “a 7.8 percentage point reduction” leave completely different impressions. Relative measures are untethered from baseline risk, so they always look larger; how much a patient actually gains depends on how much risk they had to begin with. In a low-risk population the same RRR converts into an ARR small enough to be clinically irrelevant.

A paper or a slide that reports RRR without ARR or NNT is a warning sign in itself.

Step four: intention to treat

The rule of intention-to-treat (ITT) analysis is: analyse every patient in the arm they were randomised to, regardless of what they actually received.

This is counterintuitive — why count someone in the drug arm when they never got the drug? Because the moment you start removing people, you have to use information that only exists after randomisation to decide whom to remove. And whether a patient took the treatment is itself usually related to prognosis: sicker patients are likelier to vomit, to withdraw, to be too unwell to continue. Removing them hands back the comparability randomisation just paid for.

This trial contains a clean instance. The paper states it plainly: one patient assigned to indomethacin could not retain the suppositories, and was analysed in the indomethacin arm regardless. That is ITT executed literally.

Per-protocol (PP) analysis, by contrast, counts only those who complied. It answers a different question — what happens under full adherence — at the cost of the protection randomisation provided. The usual relationship is:

• Superiority trials: ITT is conservative. Non-adherence pulls the arms toward each other, so the effect is understated. ITT is therefore the primary analysis.
• Non-inferiority trials: the direction reverses. The same dilution makes the arms look more alike, which makes it easier to conclude non-inferiority. Non-inferiority trials must therefore report both ITT and PP, and the conclusion only holds if they agree.

Step five: sample size, interim analyses and early stopping

This is the section textbooks skip and real trials cannot.

The protocol planned to enrol 948 patients (474 per arm), on the assumption that placebo risk was 10% and treatment would halve it to 5%, giving at least 80% power at a two-sided α of 0.05 (using Fisher's exact).

Enrolment actually stopped at 602. The data and safety monitoring board (DSMB) saw a clear advantage at an interim analysis and recommended stopping under the pre-specified rule.

One detail matters more than it looks: because 2 interim analyses were performed, the threshold for declaring significance at the end was not 0.05 but 0.041.

Why? Every additional look at accumulating data is another opportunity to make a type I error. Testing at 0.05 each time pushes the overall error rate well above 0.05. An O'Brien-Fleming boundary with a Lan-DeMets alpha spending function allocates a total budget of 0.05 across the looks — the interim analyses spend a little (their thresholds are severe, so stopping early is hard), and what remains funds the final analysis, which is why the final threshold sits just under 0.05.

Step six: subgroup analyses

The subgroup forest plot usually sits near the end. Three things to hold onto while reading it:

1. Subgroup analyses are always underpowered. The sample size was chosen for the primary outcome; cut it into subgroups and every cell is far smaller. Failing to detect a difference is the expected outcome, not evidence of no effect.
2. Read the test for interaction, not each subgroup’s own p-value. “Significant in A, not in B” is entirely compatible with identical effects in both, if A simply has more patients. Whether the effects genuinely differ is a different question, and only an interaction test answers it.
3. Pre-specified and post hoc are not the same evidence. Post hoc subgroup findings are hypothesis-generating and nothing more.

That point applies to this trial too. Its exploratory subgroup analyses mix pre-specified subgroups with ones added post hoc, and the paper does not separate the two. So the sentence below — that the direction of benefit was consistent across subgroups — is supportive observation compatible with the primary result, not confirmatory evidence; the post hoc cells have no prior statistical plan protecting them, so their type I error rate is uncontrolled.

This trial’s subgroup analyses detected no interaction between subgroups — the direction of benefit was the same whether or not a pancreatic stent was placed, and across the sphincter of Oddi dysfunction categories. That is far more credible than a striking finding confined to one subgroup, but hold on to point 1 above: subgroups are underpowered by design, so failing to detect an interaction is not evidence that the effect is identical in every subgroup.

The paper’s own forest plot cannot be reproduced here, but the same public data can be recomputed into one. Below is that recomputation, over 6 subgroup variables that the indo_rct file already carries:

Recomputing is not the same as copying the published table. The paper mixes pre-specified with post hoc subgroups and never says which model produced which row, so all this figure can offer is what one clearly specified model makes of one public dataset: Katz intervals for the stratum-specific risk ratios, and a Wald test on the product term of a log-link Poisson model with a robust variance for the interaction.

Read it against the three rules above. Whether a pancreatic duct stent was placed makes point 2 best: the estimate is 0.30 without a stent and 0.60 with one, roughly a twofold gap — and yet the stratum with the larger apparent effect is the one whose interval crosses the null line (it holds only 106 patients), while the other stratum’s does not. Read off the per-stratum p-values alone, that becomes “it only works in patients who got a stent”; the p-value for interaction is 0.29, and these data did not detect a difference between the two strata. Across all six variables the smallest interaction p-value is 0.23, still nowhere near small enough to carve up the population on.

All 12 point estimates sit left of the null line, which is compatible with the primary result; but every stratum’s interval is wider than the one on the overall row, and 6 of them cross the null line. That is point 1 — underpowered by design, not evidence that the subgroup fails to benefit.

This and the meta-analysis forest plot on B7-04 are two different figures that happen to look alike. There, a row is a study and the diamond pools across studies, so heterogeneity is the question. Here, a row is a stratum within one trial and the diamond is the trial itself, so the last column is the question. Reading trial reports, it is this one you meet.

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/D3-rct-indo.R
/opt/homebrew/bin/Rscript figures/scripts/D3-consort-subgroup.R

The scripts write figures/out/D3-rct-indo.stats.json and figures/out/D3-consort-subgroup.stats.json, and every figure quoted above is read from one of them. None is typed by hand. In the second file the CONSORT block is marked hypothetical: true — those counts are invented for the illustration — while the subgroup block is computed from the public data.

Watch next

▶Randomized control trial (RCT) explained

▶Intention-to-treat analysis: What is it and why is it important?

▶Intention-to-treat (ITT) and other forms of data analysis

Report a problem on this page