AdvancedIndependently reviewed, not yet spot-checked by a human

External validation

Internal validation handles optimism; it cannot handle a different group of people. This page puts one breast-cancer model through three validation designs — a random half-split, a split by year of surgery, and a genuinely external cohort. The first two leave the calibration slope sitting near 1; only the external cohort pulls it down to 0.69. It also covers sample size for the validation study itself — below a hundred events, the interval around the calibration slope is too wide to support any conclusion.

The half internal validation cannot reach

The page on internal validation deals with optimism: the coefficients were chosen to make one dataset look good, so performance measured on that dataset comes out too high. Bootstrapping and cross-validation estimate that surplus and take it away.

But both rest on one assumption: the new people come from the same distribution as the old people. Resampling can only resample the people you already have. It cannot conjure up another hospital’s referral pattern, another decade’s treatment guidelines, or another country’s laboratory methods.

External validation asks a different question: with a different group of people, does this model still get the numbers right?

Three kinds of external, three strengths

Type	How the split is made	What it tests	Strength
Temporal	One institution, divided by date: earlier patients develop the model, later patients validate it	Drift in the population over time, and changes in treatment guidelines	Weakest, but also the easiest to arrange
Geographic	A different hospital, region or country	Referral patterns, case-mix severity, differences in how things are measured	Moderate to strong
Domain	A different level of care or a different indication: a tertiary-centre model taken to primary care, an adult model taken to the elderly	The model in a population it was never designed for	Strongest, and the one that fails most often

A half-split is not external validation

This is the most common mislabelling in the prediction-model literature: split the data at random into two halves, develop on one and validate on the other, then write “external validation” in the title or the abstract.

Why it is not: two halves cut at random come from the same distribution by construction. All you have measured is optimism, and you have measured it worse than cross-validation would (see the previous page: training and testing shrink together).

So let us do it. The same 4-variable Cox model, three validation designs:

Design	External?	Validation n	Events	C-index	Calibration slope (95% CI; P10–P90 for the random split)	Mean predicted 5-year risk	Observed
Apparent (the development cohort itself)	No	2982	1713	0.666	1 (by definition)	43.8%	43.6%
Random 50/50 split (summary of 200 splits)	No	varies by split	varies by split	0.665 ± 0.007	0.996 (P10–P90: 0.856–1.128)	varies by split	varies by split
Split by year of surgery (temporal)	Barely	1815	914	0.672	1.050 (0.942–1.159)	45.0%	41.4%
A genuinely external cohort	Yes	686	299	0.642	0.691 (0.539–0.842)	43.7%	50.8%

On the left, three calibration curves are drawn on one set of axes, with predicted five-year risk on the horizontal axis, observed risk on the vertical axis, and a diagonal dashed line marking perfect calibration; the random-half-split curve and the temporal-split curve both cross back and forth over the diagonal with no consistent direction, while the curve for the genuinely external cohort sits above the diagonal in all five quantiles. On the right, the horizontal axis is the target number of events in the validation cohort and the vertical axis is the average width of the confidence interval; the calibration-slope curve starts around 0.834 at 50 events and is still around 0.303 at 299 events, while the C-index curve stays far lower throughout. — Left: the same model's calibration under three validation designs. The first two wander either side of the diagonal with no consistent direction; the genuinely external cohort departs from it in the same direction in all five quantiles. The random-split curve shown is only one illustrative split — the conclusion about random splits in the table above rests on the summary of 200 of them. Right: the external cohort was randomly thinned, proportionally by headcount, down to each target number of events, repeated 200 times, to see how wide the confidence interval for each measure is on average. The vertical dashed line is the frequently quoted "at least a hundred events".Plotting script `figures/scripts/B5-05-external-validation.R`

Read down the calibration slope column.

Random 50/50 split (summary of 200 splits): mean 0.996, P10–P90 0.856–1.128. Split by year: 1.050. The genuinely external cohort: 0.691, confidence interval 0.539–0.842, clearly excluding 1.

Meanwhile all three C-indices sit between 0.642 and 0.672 — judged on discrimination alone, the three designs look much the same.

The C-index is itself a summary that compresses every time point into one number. What a patient needs is the risk at five years, not an average over all comparable pairs; for what discrimination looks like when it is spread out across time — and why you cannot know in advance whether that curve is flat — see time-dependent AUC.

The random-split row was never a single split: it summarises 200 different random splits, with a mean calibration slope of 0.996, a standard deviation of 0.107, and only 65% of splits landing between 0.9 and 1.1. A mean pinned to 1 is not evidence that the model is good; the design guarantees it — and reporting any single one of those splits on its own could hand you a slope from anywhere in a wide range.

This page has an honest surprise of its own: the temporal validation caught nothing.

Cutting at 1988 (1167 people to develop, 1815 to validate) gives a calibration slope of 1.050, with a confidence interval that includes 1.

That does not mean temporal validation is useless. It means no drift large enough to affect the model was detected in this dataset — and the interval is 0.218 wide, so it cannot rule out moderate drift either. “Not detected” and “did not happen” are two different statements. How much a temporal validation is worth depends on whether anything actually changed in between, and that is an argument you have to make separately; performing a split by date is not on its own a licence to call the model transportable.

Turned round, this makes the size of the geographic and domain differences stand out: moving the same data to a different era detected no clear departure, while moving it to a different country drove the calibration slope to 0.691.

A validation study reports three things, not one

	Measure	What it answers	What reporting only this one hides
Discrimination	C-index / AUC (that page)	Does the model rank higher-risk people above lower-risk people	Completely immune to shifting or rescaling the whole risk scale
Calibration	Calibration curve, slope, O/E (that page)	Are the probabilities it produces numerically right	You cannot tell whether the model will over-treat or under-treat in the clinic
Clinical utility	Decision curve, net benefit (that page)	Is deciding by this model better than treating everyone or treating no one	You cannot tell whether the model is worth using at all

A validation study needs a sample size too

Validation studies are usually treated as “run it on some data we already have”, so almost nobody calculates a sample size for them. They can be underpowered just like any other study.

Thin the external cohort at random, proportionally by headcount, down to a series of target event counts, repeat 200 times, and look at how wide the confidence intervals come out:

Target events	People actually drawn	Mean width of the 95% CI for the C-index	Mean width of the 95% CI for the calibration slope
50	115	0.165	0.834
100	230	0.116	0.559
150	345	0.094	0.451
200	459	0.082	0.377
299	686	0.067	0.303

What is drawn is people, not events, so the first column is a target: the number of events actually obtained fluctuates around it from repeat to repeat. The last row (299 events, 686 people) is the complete external cohort — all 200 repeats draw the same people, so that row has no sampling variation.

At 50 events, the confidence interval for the calibration slope is 0.834 wide on average.

At that width, a true slope of 0.7 and a true slope of 1.0 are indistinguishable. You can still compute a number and draw a calibration plot, but that plot cannot support any conclusion.

The methodological literature commonly recommends at least a hundred events in the validation cohort, and preferably more than two hundred; Van Calster and colleagues recommend, for the calibration curve specifically rather than the slope alone, two hundred with the event and two hundred without. The table shows those recommendations were not invented out of thin air: going from 50 events to 100 cuts the interval width by roughly 33%.

Notice that the interval for the C-index is narrower throughout — another reason reporting only the C-index reads too optimistically: on the same small sample, discrimination looks stable while calibration was never actually measured with any precision.

What to do when calibration is off

Not throw the model away. In most cases the right response is recalibration, which comes in three levels, from adjusting the baseline risk alone to re-estimating every coefficient — the procedures, their costs and their traps are on the calibration page.

The thing to watch is that a recalibrated model has to be validated again, and not on the same people.

Run it yourself

library(survival)
data(cancer, package = "survival")

# The two cohorts define their columns differently; every harmonisation
# decision changes the result (see chapter D5)
rot <- rotterdam
rot$rfs_time  <- pmin(rot$rtime, rot$dtime)
rot$rfs_event <- as.integer(rot$recur == 1 | rot$death == 1)
rot$size_mm   <- c("<=20" = 15, "20-50" = 35, ">50" = 60)[as.character(rot$size)]
rot$grade3    <- as.integer(rot$grade >= 3)

ext <- gbsg
ext$rfs_time  <- ext$rfstime
ext$rfs_event <- ext$status
ext$size_mm   <- ext$size
ext$grade3    <- as.integer(ext$grade >= 3)

v   <- c("age", "size_mm", "nodes", "grade3")
fit <- coxph(Surv(rfs_time, rfs_event) ~ age + size_mm + nodes + grade3, data = rot)

# Transport the model: compute the linear predictor yourself, not the centred
# version predict() returns
lp <- as.numeric(as.matrix(ext[, v]) %*% coef(fit))

# One of three: discrimination
concordance(Surv(ext$rfs_time, ext$rfs_event) ~ lp, reverse = TRUE)

# Two of three: the calibration slope (1 is the right answer)
cal <- coxph(Surv(rfs_time, rfs_event) ~ lp, data = ext)
coef(cal); confint(cal)

# Mean predicted vs observed (the first level of calibration)
bh <- basehaz(fit, centered = FALSE)
h0 <- approx(bh$time, bh$hazard, xout = 5 * 365.25, rule = 2)$y
mean(1 - exp(-h0 * exp(lp)))
km <- survfit(Surv(rfs_time, rfs_event) ~ 1, data = ext)
1 - summary(km, times = 5 * 365.25, extend = TRUE)$surv

# For comparison: what a random half-split "external validation" looks like
i  <- sample(nrow(rot), nrow(rot) %/% 2)
f2 <- coxph(Surv(rfs_time, rfs_event) ~ age + size_mm + nodes + grade3, data = rot[i, ])
te <- rot[-i, ]
lp2 <- as.numeric(as.matrix(te[, v]) %*% coef(f2))
coef(coxph(Surv(rfs_time, rfs_event) ~ lp2, data = te))   # almost always near 1

Verified with R 4.6.0 + survival 3.8.6. The reverse = TRUE argument to concordance() is required: a larger linear predictor from a Cox model means higher risk and shorter survival, which is the opposite of the default direction. The calibration slope is obtained by refitting a Cox model on the validation cohort with the linear predictor as the only covariate; that coefficient is the slope.

import numpy as np, pandas as pd
from lifelines import CoxPHFitter, KaplanMeierFitter
from lifelines.utils import concordance_index

RD = "https://vincentarelbundock.github.io/Rdatasets/csv/"
rot = pd.read_csv(RD + "survival/rotterdam.csv")
ext = pd.read_csv(RD + "survival/gbsg.csv")

# The same derived columns as the R code above. rotterdam ships none of
# them, and without this every line below is a KeyError.
rot["rfs_time"]  = rot[["rtime", "dtime"]].min(axis=1)
rot["rfs_event"] = ((rot["recur"] == 1) | (rot["death"] == 1)).astype(int)
rot["size_mm"]   = rot["size"].map({"<=20": 15, "20-50": 35, ">50": 60})
rot["grade3"]    = (rot["grade"] >= 3).astype(int)
ext["rfs_time"]  = ext["rfstime"]
ext["rfs_event"] = ext["status"]
ext["size_mm"]   = ext["size"]
ext["grade3"]    = (ext["grade"] >= 3).astype(int)
v = ["age", "size_mm", "nodes", "grade3"]

fit = CoxPHFitter().fit(rot[v + ["rfs_time", "rfs_event"]], "rfs_time", "rfs_event")
lp  = ext[v].to_numpy() @ fit.params_[v].to_numpy()

# Discrimination (mind the minus sign)
print(concordance_index(ext["rfs_time"], -lp, ext["rfs_event"]))

# Calibration slope: refit on the validation data with lp as the only covariate
tmp = ext[["rfs_time", "rfs_event"]].copy(); tmp["lp"] = lp
cal = CoxPHFitter().fit(tmp, "rfs_time", "rfs_event")
print(cal.params_["lp"], cal.confidence_intervals_.loc["lp"].to_list())

# Overall risk: mean prediction vs Kaplan-Meier
km = KaplanMeierFitter().fit(ext["rfs_time"], ext["rfs_event"])
print(1 - float(km.predict(5 * 365.25)))

lifelines does the same job; note that concordance_index expects the ordering of survival time, so the linear predictor has to be negated. The calibration slope is again a Cox model refitted on the validation data with lp as the only covariate.

Five questions to ask of any paper

Which kind of external? Temporal, geographic or domain. Abstracts usually just say “external validation”.
Is it a half-split? Check the data sources in the Methods; one dataset cut in two is internal validation.
Who ran the validation? A validation by an independent team is far more persuasive.
How many of the three were reported? A paper with only a C-index has reported a third of the job.
How many events in the validation cohort? Below a hundred, discount the calibration conclusion whichever way it went.

Common misuses

Misuse	Why it is wrong
Calling a random half-split external validation	Both halves come from the same distribution; only optimism is measurable
Reporting only the C-index in an external validation	Calibration can fall apart while the C-index barely moves
Claiming transportability because the temporal validation passed	It can only detect drift over that period, never differences between institutions
Drawing conclusions from a validation cohort with a few dozen events	The interval around the calibration slope is too wide to separate 0.7 from 1.0
Declaring the model a failure because calibration is off	Recalibration is usually enough, especially when only the overall risk level has shifted
Reporting the improvement after recalibration on the same people	That is measuring yourself on data you tuned yourself on; the optimism is back
Re-estimating the coefficients and then saying “the model performed well”	Re-estimating coefficients develops a new model; it does not validate the old one
Applying the model without aligning variable definitions across cohorts	Harmonisation decisions change the result, and they show up in no statistical test
Assuming poor external performance must mean a poor model	It may equally be poor harmonisation, or different measurement in the validation cohort

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B5-05-external-validation.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

Splitting one cohort at random into a development half and a validation half, repeated two hundred times, gives a mean calibration slope of 0.996. What does a mean that close to 1 mean?

Show the answer and why

Correct answer: 0.996 sitting on 1 is guaranteed by the design rather than earned by the model: two halves split at random come from the same distribution by definition, and all this procedure can measure is optimism

0.996 is the mean over two hundred different splits. A mean sitting on 1 is what this design guarantees - the two halves come from the same distribution, so what the model learns in one half naturally still applies in the other. The standard deviation of 0.107 is not small: only about two thirds of splits land in a narrow band around 1, so a single split reported on its own can produce a slope from a wide range. And 0.691 is not an outlier; it is the thing this page is demonstrating - move to a cohort from another country and the same model's calibration slope drops to there. Writing a random half-split up as external validation in the title or abstract is the commonest mislabel in prediction-model papers.

Split by year of surgery, temporal validation gives a calibration slope of 1.050 with a confidence interval containing 1. What does that establish?

Show the answer and why

Correct answer: The interval runs from 0.942 up to that limit, more than two tenths wide - containing 1 only means no drift large enough to matter was detected here

The interval around 1.050 runs from 0.942 to 1.159, more than two tenths wide. The correct reading of containing 1 is that nothing was detected, not that nothing happened - a width like that accommodates drift of moderate size. And temporal validation measures how the population and treatment guidelines changed over time within one institution; by construction it cannot see differences in referral patterns, case-mix severity or laboratory methods between institutions, so stable over time does not imply transportable. Nor is 0.691 evidence of a problem with gbsg: the same data across a different era shows no clear departure, and across a different country the slope drops to there, which is exactly how large geographical differences can be.

The three validation designs all give a C-index inside the narrow band from 0.642 to 0.672, while the calibration slope travels from 1.050 to 0.691. What does that say about external validation papers that report only a C-index?

Show the answer and why

Correct answer: The external cohort's 0.642 is barely different from the other two designs while its calibration is already clearly off - discrimination only asks about ranking

0.642, 0.665 and 0.672 are crowded together while the same model's calibration slope travels from 1.050 to 0.691. A C-index only asks whether the model ranks higher-risk people ahead of lower-risk ones; multiply everyone's predicted risk by a constant, or add a constant to the whole scale, and the ranking is untouched and the C-index does not move. So choosing a design because its discrimination is best is choosing with an instrument that cannot see the problem, and 0.665 landing in the middle proves nothing about sensitivity either - the three values are too close together to support any conclusion. Validation reports three things: discrimination, calibration and clinical utility, in that order, and not one of them alone. Here a C-index-only report would conclude that the model transports well, while in fact it systematically underestimates risk.

Shrinking the external cohort at random until only fifty events remain, the 95% confidence interval for the calibration slope averages 0.834 wide. What does that width mean in practice?

Show the answer and why

Correct answer: An interval 0.834 wide means a clearly depressed slope and a perfectly normal one cannot be told apart at this sample size

A width of 0.834 lets the two ends of the interval hold the opposite conclusions - predictions far too extreme, and calibration entirely normal - at the same time, so the plot can be drawn but cannot support anything. The C-index interval of 0.165 is indeed far narrower, and that is another reason a C-index-only report runs optimistic: in the same small sample discrimination looks steady while calibration has not really been measured at all. 0.303 is the width for the full external cohort, and calling the difference one of precision only ignores whether a conclusion can be drawn at all - the methodological literature recommends at least a hundred events in a validation cohort and preferably two hundred, and this table is where that advice comes from.

In the true external cohort the model's mean predicted five-year risk is 0.437 and Kaplan-Meier observes 0.508. Which layer of problem is that gap?

Show the answer and why

Correct answer: The observed 0.508 exceeds the model's mean, which is a shift in the overall risk level - the first layer of calibration, a different layer from the slope, and usually the first thing recalibration repairs

0.508 against 0.437 says the model systematically underestimates risk in the external cohort, and that is a shift of the whole scale - the first layer of calibration, not the slope layer. The development cohort's mean prediction of 0.438 being close to the external one confirms that the model did not change when it was moved, and that is exactly the problem: the model stayed the same, the people did not, and the model has no way of knowing. The C-index of 0.642 cannot explain it either - discrimination governs only ranking, and pressing everybody's predicted risk down by a constant leaves the ranking untouched and the C-index unmoved while the mean falls away. Longer follow-up does not explain the gap either: both sides are compared at the same fixed five-year horizon, Kaplan-Meier has already dealt with the censoring, and following people for longer only makes that estimate steadier rather than systematically higher than the predictions. Broken calibration is not a failed model: recalibration is usually the answer, only it has to be validated again afterwards, and not in the same people.

On the calibration plot the three lowest quintiles of the temporal validation sit below their predicted risk and the top two sit above it, while all five quintiles of the true external cohort sit above their predicted risk. Why does that difference in shape matter?

Show the answer and why

Correct answer: The lowest external quintile is predicted 0.287 and observed higher, and the other four are likewise predicted below what was observed - five quintiles departing the same way is what makes a departure systematic

What matters is whether the direction is consistent, not whether there is a departure at all. The temporal line has points on both sides of the diagonal: the quintile observed at 0.238 falls below its prediction while the top two, one of them 0.714, come out above theirs, so the departures never settle on one side, and that design has a calibration slope whose confidence interval contains 1. The external cohort runs one way throughout: its lowest quintile is predicted 0.287 and observed higher, and every quintile above it is likewise predicted below what was observed, which is the whole risk scale pressed down. Treating the two lines as the same thing mistakes having a departure for having the same kind of departure, and reading the top quintile alone as though it stood for the whole line is exactly what a single point gets wrong - one agreeing point out of five can be found on almost any pair of curves. This is also why calibration is read as a whole curve rather than one summary number: the shape tells you whether the departure is a global shift, a rescaling of the slope, or confined to one stretch of the risk range, and which layer recalibration has to repair depends on that.

Chapters that use this method

Prediction model study

Watch next

ENNienke de Glas, MD PhD· 7 minCovers both faces of validation in about six and a half minutes. Watch it before the section on the three things a validation study must report.

ENData & Science with Glen Wright Colopy· 68 minThe hour-long version, given by Karel Moons. The clearest treatment anywhere of the difference between temporal, geographic and domain validation.

Sources and licences

Calibration: the Achilles heel of predictive analyticsCC BYOnly that paper's recommendation on the sample size a calibration curve needs (two hundred with and two hundred without the event) is cited here. Everything else is original, and every number was computed on rotterdam and gbsg for this site.