AdvancedIndependently reviewed, not yet spot-checked by a human

Calibration

Calibration asks whether the people a model calls 30% really do have events three times in ten. It is harder to fix and more often skipped than discrimination, and the moment a clinical decision hangs on an absolute probability threshold it matters more than discrimination does. This page measures all four levels of calibration in one real external validation, draws the flexible calibration curve, then applies three levels of recalibration — and checks on a held-out half of the patients whether any of them actually held up.

Why calibration is the part worth worrying about

Discrimination is about ordering: does the model put the people who had the event ahead of the people who did not? Calibration is about the numbers themselves: of the people the model calls 30%, do three in ten really have the event?

Whenever the clinical decision is “rank them and take the top few” — who gets the CT scan first, how organs are allocated — discrimination is enough. But the moment the decision hangs on an absolute probability threshold — start a statin above a 20% ten-year cardiovascular risk, skip adjuvant therapy below some recurrence risk — a badly calibrated model puts patients systematically on the wrong side of the line, and the C-index shows none of it.

There is also a practical reason: calibration breaks more easily than discrimination, and it is also easier to repair. A population’s baseline risk shifts with referral patterns, screening policy and improvements in treatment, and almost none of those shifts change how the predictors rank patients relative to one another. So the most common picture in an external validation is exactly the picture on this page — in this validation cohort the C-index is 0.642, but the mean predicted five-year risk sits 7.21 percentage points below the observed risk (O/E 1.165), with a calibration slope of 0.691 (95% CI 0.539–0.842).

Four levels, each stricter than the last

Level	What it requires	How it is measured	When stopping here is reasonable
1. Mean calibration (calibration-in-the-large)	Mean predicted risk equals the overall event rate	Compare the two numbers, or take the O/E ratio	With a very small sample this is the only thing estimated with any precision
2. Weak calibration	No systematic over- or underestimation, and the spread of predicted risks is right too	Calibration intercept (target 0) and calibration slope (target 1)	A sensible ceiling when the validation cohort has only a few hundred people
3. Moderate calibration	Among the people predicted 10%, one in ten has the event — every band of risk has to be right	A flexible calibration curve (loess or splines)	The standard expectation once the validation cohort is large enough
4. Strong calibration	Every possible combination of predictor values has to be right	Not achievable in practice	Do not aim for it; it is a theoretical limit, not an acceptance criterion

Measuring every level on real data

The model was developed on survival::rotterdam (2982 patients) and validated on survival::gbsg (686 patients, 299 events), at a 5-year horizon. Discrimination is a C-index of 0.642.

Level 1: mean calibration

Mean predicted risk 43.6%, observed (Kaplan-Meier) risk 50.8%. The O/E ratio is 1.165, a gap of 7.21 percentage points.

An O/E above 1 means more events happened than the model expected — the model underestimates risk.

Level 2: weak calibration

The calibration slope is 0.691 (95% CI 0.539–0.842).

A survival model has no literal intercept; its counterpart is the baseline cumulative hazard. Fix the slope at 1 and let the baseline risk move freely, and the new baseline hazard is 1.211 times the original — on the log scale, 0.192. That number plays the role of the calibration intercept, and a positive value means underestimation.

Level 3: moderate calibration

Two calibration plots. The horizontal axis is predicted five-year risk and the vertical axis is observed risk; a dashed diagonal marks perfect calibration and a grey histogram along the bottom shows how the predicted risks are distributed. In the left panel, before any update, all five red quintile points sit above the diagonal and the blue flexible curve runs above it almost throughout, dipping below only at the extreme right-hand end. In the right panel, after updating the baseline risk alone, the curve and the points have shifted right: the lowest-risk group now sits almost on the diagonal (2.9 percentage points above it), the remaining underestimation is concentrated in the middle (the second group is off by 5.8 percentage points), and the highest-risk group has crossed to below the diagonal (4.3 percentage points low) — because the calibration slope is still below 1. — Left: before any update. The blue line is the flexible calibration curve, obtained by refitting a Cox model with the linear predictor entered through a natural spline; the red points come from splitting patients into five groups by predicted risk. Both lie above the diagonal, meaning the model underestimates risk. Right: after replacing the baseline risk with the validation cohort's own. The histogram along the bottom is the distribution of predicted risks — a calibration curve is only meaningful where there are patients.Plotting script `figures/scripts/B5-06-calibration.R`

Predicted-risk quintile	Patients	Mean predicted	Observed	Gap
Group 1	138	28.6%	36.4%	7.8 percentage points
Group 2	137	33.2%	44.4%	11.2 percentage points
Group 3	137	38.7%	46.0%	7.3 percentage points
Group 4	137	47.2%	55.1%	7.9 percentage points
Group 5	137	70.4%	72.1%	1.7 percentage points

What a calibration slope below 1 means

The slope is the coefficient you get when the validation cohort’s linear predictor is refitted as the only predictor in the model.

Slope = 1: the spread of predicted risks is right.
Slope < 1: the predictions are too extreme — high-risk patients are pushed too high and low-risk patients too low.
Slope > 1: the opposite; the model’s predictions are too conservative, too bunched around the average.

Below 1 is the most common pattern in external validation. It has two possible sources, and the calibration slope on its own cannot tell them apart:

Overfitting during development — the coefficients were too large to begin with (see the shrinkage page)
The two populations differ in risk structure — some predictor genuinely has a smaller effect in the new population

In this example the first source is less likely — but only less likely, not ruled out. The development cohort had 2982 patients and 1713 events, roughly 428 events per parameter; the bootstrap internal validation of the same model in chapter D5 measured an optimism in the C-index of 0.0004.

Together those make severe overfitting unlikely. They are not enough on their own to conclude that a slope of 0.691 comes mainly from population differences: what was measured is the optimism of the C-index, not the optimism of the calibration slope itself, and the two are different quantities. Genuinely apportioning the blame would take a separate bootstrap targeting the calibration slope / shrinkage factor.

Whichever way the answer falls, the only way to have a chance of separating the two is to do internal and external validation separately and report both. A paper that reports an external validation without any internal optimism estimate leaves the reader with nothing to judge by.

Do not use the Hosmer-Lemeshow test

It is the “calibration test” that appears most often in papers, and the methodological literature recommends retiring it. Three reasons:

It depends on arbitrary grouping — the number of groups and where the cuts fall change the result.
Its p-value carries neither direction nor magnitude — a non-significant result does not mean calibration is good, only that no departure was detected; a significant one does not tell you whether the model over- or underestimates, or by how much.
It has very low power — with a small sample it is almost bound to be non-significant, and “p was not significant” then gets written up as “calibration was good”, which is precisely the conclusion the data cannot support.

What about the calibration belt?

It is common in the Italian and intensive-care literature (the GiViTI calibration belt, R package givitiR). Mechanically it fits a polynomial of the observed outcome on predicted risk, draws a simultaneous confidence region around that curve across the whole range of predicted risk, and reports a p-value derived from that region — significant exactly when the region excludes the diagonal somewhere.

It asks the same question as the flexible calibration curve at level 3 above, but what it adds is more than a p-value: it adds the uncertainty around the curve, and the plot on this page does not show that at all. The spline curve drawn above is a point estimate with no band around it, so a reader cannot tell whether a visible dip is real miscalibration or the wobble of a spline fitted to a few hundred events. That gap is where the belt earns its keep. The p-value is a by-product of the region, and it is the by-product that the three points above argue against.

This page does not draw one, and the reason is not that the belt adds nothing. It is partly scope: this page is about the four levels and about recalibration, and uncertainty around the flexible curve is a layer it never draws — the calibration slope comes with a confidence interval above, but the curve itself is a bare point estimate. It is also a mismatch — the belt as published, and as implemented in givitiR, is defined for a binary outcome and its predicted probability, whereas the model validated here is a Cox model with censored follow-up, and a large part of this validation cohort has not been observed all the way to five years. Collapsing it to a binary five-year status would mean either discarding those patients or counting them as event-free. So there is no reason to object when a paper reports a belt — but what to read is where the region leaves the diagonal and by how much, not the p-value beside it.

Recalibration: three levels

The default response to broken calibration is not to discard the model but to update it. From lightest to heaviest:

Level	What changes	Validation data needed	What it can fix
1. Update the baseline risk only	The whole risk scale shifts; coefficients and slope untouched	Least	Mean calibration only (level 1)
2. Update baseline risk and slope	One more parameter, rescaling the spread of predicted risks	Moderate	Up to weak calibration (level 2)
3. Refit every coefficient	Equivalent to developing the model again in the new population	Most — with too little data this just relocates the overfitting	May change discrimination

Applying each of them to the whole validation cohort:

Approach	C-index	Calibration slope	Mean predicted	Observed	O/E
No update	0.642	0.691	43.6%	50.8%	1.165
Update the baseline risk only	0.642	0.691	49.4%	50.8%	1.029
Update baseline risk and slope	0.642	1.000	51.3%	50.8%	0.990
Refit every coefficient	0.651	1.000	51.5%	50.8%	0.988

That table has a structural problem: it measures each update on the very data used to make it.

The O/E in the row labelled Update the baseline risk only lands on 1.029 and the slope in the row labelled Update baseline risk and slope comes out at exactly 1.000 by construction, not by validation: whichever quantity you adjust is the quantity that gets pulled into place. Note that updating the baseline risk alone leaves the slope untouched — that row still has a slope of 0.691.

The honest approach is to split the validation cohort in half: recalibrate on one half and measure performance on the other. Repeated 200 times:

Approach	C-index	Calibration slope	Mean predicted	Observed	O/E
No update	0.642	0.724	43.5%	51.0%	1.170
Update the baseline risk only	0.642	0.724	49.0%	51.0%	1.044
Update baseline risk and slope	0.642	1.085	51.0%	51.0%	1.004
Refit every coefficient	0.639	0.946	51.1%	51.0%	1.003

The picture is now completely different:

Updating the baseline risk alone moves the O/E from 1.170 to 1.044, and it holds up in the half of the patients it was not fitted on — the cheapest update is the most durable.
Updating the slope as well brings the O/E to 1.004, but the slope itself lands at 1.085, now slightly above 1: a slope estimated on half a cohort carries its own noise, and applying it to the other half overshoots.
Refitting every coefficient gives a C-index of 0.639, below the 0.642 obtained with no update at all. Across the 200 splits, the mean difference in C-index between a full refit and no update was -0.003 (P10–P90: -0.019–0.011), with the refit lower in 56% of splits. At this sample size a full refit did not show a stable improvement in discrimination, which is consistent with the risk of overfitting.

This is the empirical version of Van Calster’s warning: a full refit only pays off when the validation sample is relatively large. Here it bought no discrimination, and the calibration it did buy was already available from the two cheaper updates.

Whichever level you use, the updated model needs validating again — and not on the same patients the update was fitted to.

Run it yourself

library(survival); library(splines)
data(cancer, package = "survival")

rot <- rotterdam
rot$rfs_time  <- pmin(rot$rtime, rot$dtime)
rot$rfs_event <- as.integer(rot$recur == 1 | rot$death == 1)
rot$size_mm   <- c("<=20" = 15, "20-50" = 35, ">50" = 60)[as.character(rot$size)]
rot$grade3    <- as.integer(rot$grade >= 3)

ext <- gbsg
ext$rfs_time  <- ext$rfstime; ext$rfs_event <- ext$status
ext$size_mm   <- ext$size;    ext$grade3    <- as.integer(ext$grade >= 3)

v   <- c("age", "size_mm", "nodes", "grade3")
H   <- 5 * 365.25
fit <- coxph(Surv(rfs_time, rfs_event) ~ age + size_mm + nodes + grade3, data = rot)

# Breslow baseline cumulative hazard, written out by hand. See the warning above
breslow_h0 <- function(time, event, lp, h) {
  o <- order(time); time <- time[o]; event <- event[o]; r <- exp(lp[o])
  sum(1 / rev(cumsum(rev(r)))[event == 1 & time <= h])
}
lp_dev <- as.matrix(rot[, v]) %*% coef(fit)
h0     <- breslow_h0(rot$rfs_time, rot$rfs_event, lp_dev, H)

lp   <- as.numeric(as.matrix(ext[, v]) %*% coef(fit))
pred <- 1 - exp(-h0 * exp(lp))

# Level 1: mean calibration
km  <- survfit(Surv(rfs_time, rfs_event) ~ 1, data = ext)
obs <- 1 - summary(km, times = H, extend = TRUE)$surv
c(mean_predicted = mean(pred), observed = obs, OE = obs / mean(pred))

# Level 2: the calibration slope
cal <- coxph(Surv(rfs_time, rfs_event) ~ lp, data = ext)
coef(cal); confint(cal)

# Level 3: the flexible calibration curve
flex <- coxph(Surv(rfs_time, rfs_event) ~ ns(lp, df = 3), data = ext)
h0f  <- breslow_h0(ext$rfs_time, ext$rfs_event,
                   as.numeric(predict(ns(lp, df = 3), newx = lp) %*% coef(flex)), H)
g    <- seq(quantile(lp, .02), quantile(lp, .98), length.out = 100)
obs_curve  <- 1 - exp(-h0f * exp(as.numeric(predict(ns(lp, df = 3), newx = g) %*% coef(flex))))
pred_curve <- 1 - exp(-h0 * exp(g))
plot(pred_curve, obs_curve, type = "l", xlim = c(0, 1), ylim = c(0, 1)); abline(0, 1, lty = 2)

# Recalibration level 1: swap in a new baseline risk
h0_new <- breslow_h0(ext$rfs_time, ext$rfs_event, lp, H)
mean(1 - exp(-h0_new * exp(lp)))     # should now sit close to obs

Verified with R 4.6.0 and survival 3.8.6. ⚠️ The figure script does not use basehaz() to obtain the recalibrated baseline risk; it implements the Breslow estimator directly. When a coxph model carries an offset and no covariates, basehaz() returns the cumulative hazard at the mean offset, and centered = FALSE does not undo that — on this dataset it would inflate the baseline risk by nearly a factor of four.

import numpy as np, pandas as pd
from lifelines import CoxPHFitter, KaplanMeierFitter

RD = "https://vincentarelbundock.github.io/Rdatasets/csv/"
rot = pd.read_csv(RD + "survival/rotterdam.csv")
ext = pd.read_csv(RD + "survival/gbsg.csv")

# The same derived columns as the R code above. rotterdam ships none of
# them, and without this every line below is a KeyError.
rot["rfs_time"]  = rot[["rtime", "dtime"]].min(axis=1)
rot["rfs_event"] = ((rot["recur"] == 1) | (rot["death"] == 1)).astype(int)
rot["size_mm"]   = rot["size"].map({"<=20": 15, "20-50": 35, ">50": 60})
rot["grade3"]    = (rot["grade"] >= 3).astype(int)
ext["rfs_time"]  = ext["rfstime"]
ext["rfs_event"] = ext["status"]
ext["size_mm"]   = ext["size"]
ext["grade3"]    = (ext["grade"] >= 3).astype(int)
v = ["age", "size_mm", "nodes", "grade3"]; H = 5 * 365.25

fit = CoxPHFitter().fit(rot[v + ["rfs_time", "rfs_event"]], "rfs_time", "rfs_event")
lp  = ext[v].to_numpy() @ fit.params_[v].to_numpy()

def breslow_h0(time, event, lp, h):
    o = np.argsort(time); t, e, r = time[o], event[o], np.exp(lp[o])
    at_risk = np.cumsum(r[::-1])[::-1]
    return np.sum(1.0 / at_risk[(e == 1) & (t <= h)])

lp_dev = rot[v].to_numpy() @ fit.params_[v].to_numpy()
h0 = breslow_h0(rot["rfs_time"].to_numpy(), rot["rfs_event"].to_numpy(), lp_dev, H)
pred = 1 - np.exp(-h0 * np.exp(lp))

km  = KaplanMeierFitter().fit(ext["rfs_time"], ext["rfs_event"])
obs = 1 - float(km.predict(H))
print(pred.mean(), obs, obs / pred.mean())

tmp = ext[["rfs_time", "rfs_event"]].copy(); tmp["lp"] = lp
print(CoxPHFitter().fit(tmp, "rfs_time", "rfs_event").params_["lp"])   # calibration slope

lifelines has no ready-made flexible calibration curve; feed the linear predictor through a spline basis built with patsy or numpy and refit a CoxPHFitter. The procedure is the same as in R.

Four things to ask when reading a paper

Is there a calibration plot at all? A prediction-model paper reporting only a C-index or an AUC has handed in half the work.
Which level is being reported? O/E alone is level 1; a slope alone is level 2; only a flexible curve reaches level 3.
How many events are in the validation cohort? Below about a hundred, discount any calibration conclusion, favourable or not.
Is a non-significant Hosmer-Lemeshow p-value used to claim good calibration? That inference does not hold.

Common misuses

Misuse	Why it is wrong
Reporting discrimination without calibration	Getting the ordering right does not make the numbers right, and clinical decisions use absolute probabilities
Claiming good calibration from a non-significant Hosmer-Lemeshow test	Non-significance means no departure was detected, and the test has low power to begin with
Claiming good calibration because the intercept is near 0 and the slope near 1	Overestimation in one band and underestimation in another can cancel out; look at the flexible curve
Reporting the improvement on the same patients the recalibration was fitted to	A slope of 1 and an O/E of 1 are then true by construction
Refitting every coefficient in a validation sample of two or three hundred	That only moves the overfitting from one population to another
Declaring a model invalid because calibration is off	Most of the time the baseline risk just needs updating
Expecting recalibration to improve the C-index	The first two levels leave the ordering untouched, so the C-index does not move
Interpreting the calibration curve where the data are sparse	Those stretches are extrapolation; check the distribution plot along the bottom
Treating strong calibration as an acceptance criterion	It is a theoretical limit and unachievable in practice

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B5-06-calibration.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

The external validation reports an O/E ratio of 1.165. Does this model over-predict or under-predict risk in this cohort?

Show the answer and why

Correct answer: The five-year risk these patients actually ran is 0.508, above the model's average prediction, so the model under-predicts

The numerator of O/E is observed and the denominator expected. 0.508 is what was observed and 0.436 is what the model predicted on average; their ratio is the number in the stem, above 1, so more happened than the model expected and the model under-predicts. Swapping the two turns under-prediction into over-prediction, and the two have opposite clinical consequences: under-prediction leaves patients under-treated, over-prediction over-treats them. 0.691 is the calibration slope, which measures whether the spread of predicted risks is right rather than their overall level; a model can be near-perfect on average calibration with a slope far from 1, and the reverse happens too.

The same validation reports a calibration slope of 0.691, 95% CI 0.539 to 0.842. Which reading is right?

Show the answer and why

Correct answer: A slope of 0.691, below 1, means the model spreads risk too far apart - high pushed too high, low pressed too low

The slope is the coefficient you get by refitting with the validation cohort's linear predictor as the only covariate, so it measures the spread of predicted risks, not their level. 0.842 is the upper bound, and the whole interval does sit below 1 - but what that establishes is that the spread is too extreme, not that every patient is predicted too low; the latter is a statement about mean calibration and needs the O/E. 0.539 is the lower bound. The interval is not narrow, yet none of it contains 1, so calling the estimate too imprecise to judge gets it backwards: imprecise and undisturbed are different things. A slope below 1 has two possible sources - overfitting during development, and a genuinely different risk structure in the new population - and the slope alone cannot separate them.

After the baseline risk is replaced with the validation cohort's own, O/E moves from 1.165 to 1.029. What happens to the C-index in the same table?

Show the answer and why

Correct answer: It is still 0.642, unchanged - a shift in one direction reorders no pair of patients

Updating the baseline risk shifts the whole risk scale; updating the slope multiplies the linear predictor by a constant. Neither reorders any pair of patients, so the C-index does not move at all - the first three rows of that table are all 0.642. 0.651 is the fourth row, refitting every coefficient, measured on the very patients it was fitted to, and that row is the only one that can move discrimination. 0.639 is what refitting every coefficient gives when the validation cohort is split in half and the score is taken on the other half - lower than not updating at all. Recalibration fixes calibration, not discrimination; when a paper says the model got better after recalibration, ask which of the two it means.

On the calibration plot after the baseline risk is updated, the point for the highest-risk fifth has moved from above the diagonal to below it. Which reading is right?

Show the answer and why

Correct answer: That group's mean predicted risk after the update is 0.764, now above what it actually observed, which is why its point fell below the diagonal

Updating the baseline risk moves the predicted side only. The observed risk is what happened to these patients and does not change because somebody refitted a model - 0.721 is the same observed value before and after the update, and reading it as having been shrunk assumes recalibration can rewrite follow-up. After the update the top group predicts 0.764, above its observed value, which is why its point falls below the diagonal: the shift adds roughly the same amount to everyone while the slope is still below 1, so the over-spread was never touched. 0.335 is the lowest group's updated prediction, and it does sit almost on the diagonal, but that says the low end is fixed, not that every other stretch is - and least of all that the crossing at the top is noise, since it has a mechanism whose direction could be stated in advance.

Updating both the baseline risk and the slope on the whole validation cohort leaves a calibration slope of exactly 1.000 when it is measured afterwards. Does that show the update worked?

Show the answer and why

Correct answer: It does not. Measured on the half that took no part in the update, the same update gives 1.085, already past 1

Measuring a fit on the data it was fitted to pins the slope at 1.000 by definition, not by validation: whatever you adjust is what gets pulled into place. The honest version splits the cohort, updates on one half and measures on the other, and there the same update gives 1.085 - a slope estimated on half a cohort carries its own noise, and applying it to the other half overshoots. As for the 0.691 row, that is the baseline-only update, and a shift never touches the slope, so the two rows do not contradict each other at all; reading untouched as contradictory suggests the table is wrong when it is not.

Split the validation cohort, refit every coefficient on one half and score on the other, and the C-index is 0.639, below the 0.642 you get by not updating at all. How should that sentence be written?

Show the answer and why

Correct answer: Across 200 splits the mean difference is -0.003, so at this sample size no stable gain in discrimination was detected

Those two percentiles are the ends of a distribution, not a conclusion: -0.019 describes the least favourable tenth of splits and 0.011 the most favourable tenth, and taking either end as the answer is taking a tail for the centre. What matters is where the distribution sits and how wide it is - the mean difference is -0.003 and the ends run from -0.019 to 0.011, with zero in between - so the sentence to write is that no stable gain in discrimination was detected at this sample size, not that refitting everything is worse. That is the usual position of a full refit: it spends the most data, and the calibration it buys was already available from two cheaper updates.

Chapters that use this method

Prediction model study

Watch next

ENNienke de Glas, MD PhD· 7 minThe shortest clinically framed explanation available. Watch it to build the intuition before reading the four levels below.

簡中Bessie Hiram· 17 minIn Simplified Chinese; walks through a calibration plot from drawing it to reading it, and gives the Chinese terminology alongside the English.

Sources and licences

Calibration: the Achilles heel of predictive analyticsCC BYThe four levels of calibration, the levels of recalibration, and the case against the Hosmer-Lemeshow test are adapted from Van Calster et al. (BMC Medicine 2019). The QRISK2 comparison quotes published values cited in that paper; every other number on this page was computed here on rotterdam/gbsg.