ExpertIndependently reviewed, not yet spot-checked by a human

Decision curve analysis

A decision curve does not ask whether a model is accurate. It asks whether deciding by the model beats treating everyone or treating nobody. The relative cost of a false positive and a false negative is encoded in a threshold probability, and the answer is expressed as net benefit. This page builds, on a real external validation, the comparison that gets mentioned far more often than it gets demonstrated — the model with the higher discrimination having the lower net benefit at the thresholds a clinician would actually use.

The question the previous two pages leave unanswered

The AUC tells you how well the model ranks patients. Calibration tells you whether the probabilities it produces are numerically right. Pass both and one question is still open:

Would deciding by this model be better than what we do now?

And “what we do now” usually comes in only two versions: treat everyone (or test everyone), or treat nobody. A model that cannot beat whichever of those two lines is higher is of no use, however accurate it is.

Decision curve analysis (DCA) puts all three on the same axes.

The threshold probability: a clinical judgement written as a number

To say whether a model is useful, you first have to say which is worse, a false positive or a false negative, and by how much. DCA does this by asking you to pick a threshold probability $p_t$ :

At or above a risk of $p_t$ I treat; below $p_t$ I do not.

That single sentence has already stated the relative costs. Being willing to start treatment at exactly $p_t$ says that the harm of treating and the harm of leaving an event untreated break even at that level of risk. Rearranged:

\frac{\text{cost of a false positive}}{\text{cost of a false negative}} = \frac{p_t}{1 - p_t}

Threshold probability	Implied cost ratio	In plain words
30%	0.43	Missing one person who really will have the event is as bad as needlessly treating 2.3 people who will not
40%	0.67	Missing one person who really will have the event is as bad as needlessly treating 1.5 people who will not
50%	1.00	Missing one person who really will have the event is as bad as needlessly treating 1.0 people who will not
60%	1.50	Missing one person who really will have the event is as bad as needlessly treating 0.7 people who will not

The threshold probability is a clinical decision, not a statistical one. It follows from the side effects, cost and invasiveness of the treatment and from the consequences of missing an event — the threshold for chemotherapy cannot possibly be the threshold for a follow-up blood test.

And precisely because it is a clinical decision, on which different clinicians and different patients will disagree, DCA does not answer for one threshold; it draws the whole range of thresholds as a curve. You find your own threshold on the horizontal axis and read off the answer.

This also explains why choosing a cut-off by the Youden index so often makes no clinical sense. Youden gives a percentage point of sensitivity and a percentage point of specificity equal weight — but those percentage points are counted over different groups of people, so equal weight per rate is not equal cost per person. Work it through and the cut-off Youden picks is the one that would be optimal at a threshold probability equal to the prevalence of the cohort it was chosen in: a clinical judgement made for you by the case mix of the sample. Here that is nearly invisible, because this cohort’s observed five-year risk is 50.8%, so Youden’s implied threshold sits close to an even split. Move the same model into a population where the event is rarer and the implied threshold falls with the prevalence, without anyone deciding that it should. The cut-off page derives the same statement from the cost side.

Net benefit: converting false positives into true-positive equivalents

Net benefit is defined as:

\mathrm{NB}(p_t) = \frac{\mathrm{TP}}{n} - \frac{\mathrm{FP}}{n} \times \frac{p_t}{1 - p_t}

The first term is “true positives caught per patient”. The second subtracts the false positives after converting them into true-positive equivalents at the cost ratio.

The unit is therefore net true positives per patient. Multiply by 1000 and it becomes “per 1000 patients, how many extra true positives this model finds compared with doing nothing, with the cost of the extra treatment already deducted”.

Survival data need one more step: you cannot simply count events, because some patients are censored before the horizon. So the proportions behind TP and FP are estimated from the Kaplan-Meier risk within the group flagged as high risk.

Two reference lines:

Treat nobody: nobody is treated, so there are neither true nor false positives and net benefit is always 0.
Treat everyone: everyone is called positive, so $\mathrm{NB} = \text{event rate} - (1 - \text{event rate}) \times \frac{p_t}{1-p_t}$ . It falls faster as the threshold rises, and crosses 0 where the threshold equals the event rate.

This page’s example: at some thresholds the model with better discrimination has the lower net benefit

Three models, all developed on survival::rotterdam, compared on survival::gbsg (686 patients, 299 events, observed 5-year risk 50.8%):

Model	Predictors	C-index	Mean predicted 5-year risk
A: eight predictors, carried over as published	8	0.6617	43.1%
A′: the same model, recalibrated in this cohort	8	0.6617	51.5%
B: four predictors, recalibrated in this cohort	4	0.6416	51.3%

Model A’s C-index exceeds model B’s by 0.020. But A was never recalibrated: it predicts 43.1% on average while this cohort actually experiences 50.8% — systematic underestimation.

Left panel: a decision curve with threshold probability on the horizontal axis and net benefit on the vertical axis. A horizontal grey line marks treat-nobody and a dashed grey line marks treat-everyone. At low thresholds the three model curves sit on or just above the treat-everyone line; the treat-everyone line then falls steeply and crosses zero at 51%, while all three model curves stay above zero across the whole plotted range. Right panel: predicted risks for every patient from two models, sorted from lowest to highest, against a horizontal line at the cohort's observed five-year risk. Model A's curve runs below that line for most patients, while model B's curve crosses it. — Left: decision curves for the three models plus the two reference lines. The jaggedness comes from the Kaplan-Meier estimate within the group flagged as high risk, and is largest where that group is small. Right: each model's predicted risk for every patient, sorted by size. The dashed red line is the cohort's actual five-year risk.Plotting script `figures/scripts/B5-07-decision-curve.R`

Threshold	Treat everyone	Model A	Model A′	Model B	Share flagged by A	Share flagged by B
30%	0.2977	0.3038	0.2977	0.2977	81%	100%
40%	0.1806	0.1925	0.2189	0.1966	44%	90%
50%	0.0167	0.0982	0.1284	0.1218	25%	42%
60%	-0.2291	0.0647	0.0719	0.0617	15%	19%

In the 50% row, model B — the one with the lower C-index — has a net benefit of 0.1218, while model A, with the higher C-index, reaches only 0.0982.

Converted into people, that gap is 24 extra net true positives per 1000 patients.

The mechanism is in the right-hand panel: model A underestimates risk, so at a threshold of 50% it flags only 25% of patients as high risk, where model B flags 42%. Many of the patients A leaves out really do have events — poor calibration translates straight into undertreatment.

This is what “a model with a higher AUC can be less useful” looks like in practice. The C-index sees only the ordering; it cannot see that the whole scale has been shifted downwards. Clinical decisions use absolute probabilities, so as soon as a threshold is drawn, that shift becomes patients placed on the wrong side of it.

The honest part: the recalibration used up the same patients

Both A′ and B were recalibrated on the same cohort they are then evaluated on. That is optimistic, for the reasons set out on the internal validation page.

Split the validation cohort in half: recalibrate model B on one half, measure net benefit on the other, and repeat 200 times (model A needs no recalibration, so it is the same in both halves):

Threshold	Model A	Model B (on the held-out half)	Treat everyone	Share of splits where B beats A
30%	0.3044	0.2996	0.2995	36%
40%	0.1937	0.1874	0.1827	47%
50%	0.0974	0.1114	0.0192	78%
60%	0.0638	0.0561	-0.2259	27%

How to read a decision curve

What to look at	How to judge it
Whether the model curve rises above both reference lines near your threshold	If it does not, the model should not be used at that threshold — treating everyone or nobody does better
Whether the range of thresholds is the right one	The horizontal axis should cover the range clinicians would really use. An axis running to 0.9 when nobody would ever use a threshold that high is decoration
Whether two model curves cross	They often do. A crossing means “which is better” depends on the threshold, and there is no single answer
Where the treat-everyone line crosses zero	At the threshold equal to the event rate. For the data on this page, at about 51%
How jagged the curves are	The jaggedness is estimation error in small subgroups; with a small validation sample, do not read the fine wiggles

Run it yourself

library(survival)
data(cancer, package = "survival")
# The same cohort harmonisation and model fit as B5-05, spelled out so
# this block runs on its own.


H <- 5 * 365.25

rot <- rotterdam
rot$rfs_time  <- pmin(rot$rtime, rot$dtime)
rot$rfs_event <- as.integer(rot$recur == 1 | rot$death == 1)
rot$size_mm   <- c("<=20" = 15, "20-50" = 35, ">50" = 60)[as.character(rot$size)]
rot$grade3    <- as.integer(rot$grade >= 3)

ext <- gbsg
ext$rfs_time  <- ext$rfstime; ext$rfs_event <- ext$status
ext$size_mm   <- ext$size;    ext$grade3    <- as.integer(ext$grade >= 3)

# The two candidate models, defined exactly as this page's figure script does.
BIG   <- c("age", "meno", "size_mm", "grade3", "nodes", "log_pgr", "log_er", "hormon")
SMALL <- c("age", "size_mm", "nodes", "grade3")
rot$log_pgr <- log1p(rot$pgr); rot$log_er <- log1p(rot$er)
ext$log_pgr <- log1p(ext$pgr); ext$log_er <- log1p(ext$er)
keep <- c("rfs_time", "rfs_event", BIG)
dev  <- rot[complete.cases(rot[, keep]), keep]
ext  <- ext[complete.cases(ext[, keep]), keep]

# Breslow cumulative baseline hazard, written out: basehaz(centered = FALSE)
# does not match predict(type = "lp"), which is centred, and pairing them
# leaves every predicted risk short by a constant factor.
breslow_h0 <- function(time, event, lp, h = H) {
  o <- order(time); time <- time[o]; event <- event[o]; r <- exp(lp[o])
  at_risk <- rev(cumsum(rev(r)))
  sum(1 / at_risk[event == 1 & time <= h])
}
form_of <- function(v) as.formula(paste("Surv(rfs_time, rfs_event) ~",
                                        paste(v, collapse = " + ")))
fit_big   <- coxph(form_of(BIG),   data = dev)
fit_small <- coxph(form_of(SMALL), data = dev)
h0_big   <- breslow_h0(dev$rfs_time, dev$rfs_event,
                       as.numeric(as.matrix(dev[, BIG]) %*% coef(fit_big)))
lp_big   <- as.numeric(as.matrix(ext[, BIG])   %*% coef(fit_big))
lp_small <- as.numeric(as.matrix(ext[, SMALL]) %*% coef(fit_small))

# A: eight predictors, exactly as it left development. B: four predictors,
# then recalibrated to this cohort.
pred_A <- 1 - exp(-h0_big * exp(lp_big))
sl_B   <- unname(coef(coxph(Surv(rfs_time, rfs_event) ~ lp_small, data = ext))[1])
h0_B   <- breslow_h0(ext$rfs_time, ext$rfs_event, sl_B * lp_small)
pred_B <- 1 - exp(-h0_B * exp(sl_B * lp_small))


km_risk <- function(d, h = H) {
  if (nrow(d) < 5 || sum(d$rfs_event) == 0) return(NA_real_)
  k <- survfit(Surv(rfs_time, rfs_event) ~ 1, data = d)
  1 - summary(k, times = h, extend = TRUE)$surv
}

# Net benefit: two lines of arithmetic; the rest is selecting patients
net_benefit <- function(pred, d, pt) sapply(pt, function(p) {
  treat  <- pred >= p
  if (sum(treat) < 10) return(0)          # too few people for a reliable KM
  ptreat <- mean(treat)
  r      <- km_risk(d[treat, , drop = FALSE])
  r * ptreat - (1 - r) * ptreat * p / (1 - p)
})

nb_all <- function(d, pt) { r <- km_risk(d); r - (1 - r) * pt / (1 - pt) }

pt <- seq(0.05, 0.8, by = 0.01)
plot(pt, nb_all(ext, pt), type = "l", lty = 2, ylim = c(-0.05, 0.5),
     xlab = "Threshold probability", ylab = "Net benefit")
abline(h = 0)                              # treat nobody
lines(pt, net_benefit(pred_A, ext, pt), col = "darkorange", lwd = 2)
lines(pt, net_benefit(pred_B, ext, pt), col = "steelblue",  lwd = 2)

# Converted to "extra net true positives per 1000 patients"
1000 * (net_benefit(pred_B, ext, 0.5) - net_benefit(pred_A, ext, 0.5))

Verified with R 4.6.0 and survival 3.8.6. The dcurves package is deliberately not used here; the formulas are written out instead — a decision curve is only two lines of arithmetic, and writing it once is more useful than memorising argument names. In practice dcurves::dca() takes a Surv object directly and produces the same curves.

import numpy as np, pandas as pd
from lifelines import KaplanMeierFitter

H = 5 * 365.25

def km_risk(d, h=H):
    if len(d) < 5 or d["rfs_event"].sum() == 0:
        return np.nan
    km = KaplanMeierFitter().fit(d["rfs_time"], d["rfs_event"])
    return 1 - float(km.predict(h))

def net_benefit(pred, d, pts):
    out = []
    for p in pts:
        treat = pred >= p
        if treat.sum() < 10:
            out.append(0.0); continue
        ptreat = treat.mean()
        r = km_risk(d[treat])
        out.append(r * ptreat - (1 - r) * ptreat * p / (1 - p))
    return np.array(out)

def nb_all(d, pts):
    r = km_risk(d)
    return r - (1 - r) * pts / (1 - pts)

pts = np.arange(0.05, 0.81, 0.01)
# Once the curves are drawn, always check first whether the model clears nb_all and 0

Python has no widely adopted survival version of DCA; the code below implements the formula directly, with Kaplan-Meier from lifelines. The binary-outcome version is simpler still — replace the KM step with a plain proportion.

Four things to ask when reading a paper

Is the threshold range sensible? The horizontal axis should cover the range clinicians would really use, and the authors should say why that range.
Are both reference lines drawn? Without the treat-everyone line, the reader cannot tell whether the model adds anything.
Which data was the decision curve drawn on? A decision curve drawn on the development data carries the same optimism as an apparent C-index and cannot be taken at face value.
Was the model calibrated first? Decision curves are extremely sensitive to calibration — this whole page is one demonstration of that.

Common misuses

Misuse	Why it is wrong
Choosing a model by its AUC	The AUC contains no clinical costs; a poorly calibrated model with a high AUC can have the lower net benefit
Drawing a decision curve without the treat-everyone line	Without a reference line there is no way to tell whether the model adds anything
Reporting net benefit at a single threshold	The threshold is a clinical judgement and differs between clinicians and patients; report a range
Comparing net benefit values across studies	Net benefit contains the event rate, so different populations are not comparable
Presenting a decision curve drawn on the development data as evidence	It is as optimistic as an apparent C-index
Reporting the net benefit gain on the same patients used for recalibration	Optimistic; split the data or find another cohort
Tuning the threshold probability to make the model look best	It encodes clinical costs; it is not a quantity to optimise
Insisting on a winner when the curves cross	A crossing means the answer depends on the threshold
Reading fine wiggles in a curve from a small sample	Those come from estimation error in small subgroups
Describing net benefit as a treatment effect	It measures the value of deciding with this model, not the effect of the treatment itself

What about the reclassification measures?

The decision curve asks how many more people this model finds per thousand without treating more people unnecessarily. The other family of “is the new model better” measures — the Brier score, NRI and IDI — is not asking that, and on the very same models A and B used here it points the opposite way from net benefit. That comparison is on Brier score, NRI and IDI.

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B5-07-decision-curve.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

The horizontal axis of a decision curve is the threshold probability. What does putting the threshold at forty per cent declare?

Show the answer and why

Correct answer: A false positive costs 0.667 of a false negative - missing one person who will have an event is as bad as treating one and a half who never would

A threshold probability converts to a cost ratio as the threshold divided by one minus the threshold, and forty per cent gives 0.667: being willing to start treatment at exactly that risk says that missing one person who will have an event is as bad as treating one and a half who will not. 0.429 is the cost ratio at a thirty per cent threshold. The threshold is a clinical decision - set by the harms, cost and invasiveness of treatment and by the consequences of missing an event - and the model's performance never enters it. 1.500 is the cost ratio at sixty per cent: a higher threshold means you demand more certainty before acting, so fewer people are flagged, not more.

In the fifty per cent row, model A has net benefit 0.0982 against model B's 0.1218, while A has the higher C-index. What is the mechanism?

Show the answer and why

Correct answer: At this threshold model A flags only 25% of patients, and many of those it keeps below the line really will have an event

The C-index sees ranking only; it cannot see that the whole scale has been pressed down. Model A predicts 43.1% on average while these patients actually ran 50.8%, so the moment a threshold is drawn at fifty per cent the depressed scale puts people on the wrong side of it: A flags 25% where B flags 42%. Treating more people does not raise net benefit by itself - net benefit has already subtracted the false positives, converted into true-positive equivalents at the cost ratio, and a model that treats too many falls below the treat-all line at high thresholds. As for 15%, that is model A at the sixty per cent threshold; every model flags fewer people as the threshold rises, and it says nothing about ranking.

The third model, A-prime, is model A recalibrated in this cohort. Its C-index is identical to A's, and at the fifty per cent threshold its net benefit is 0.1284, above B's 0.1218. What does that establish?

Show the answer and why

Correct answer: A-prime predicts 0.5148 on average, close to the five-year risk these patients actually ran - what beats A is not having fewer predictors, it is being calibrated

Recalibration reorders no pair of patients, which is why A-prime and A have exactly the same C-index - what was fixed is not the ranking but the scale: A predicted 0.4312 on average, far below the five-year risk these patients actually ran, and A-prime predicts 0.5148 after recalibration. B's mean prediction of 0.5135 is almost the same as A-prime's, and that is precisely what shows the difference is not about the number of predictors. The extra predictors do carry value; you just cannot collect it until the calibration is fixed.

Split the validation cohort, recalibrate model B on one half, measure net benefit on the other, and repeat 200 times. How should the conclusion be written?

Show the answer and why

Correct answer: At the fifty per cent threshold B beats A in 78.0% of splits, so write that near the higher thresholds the simpler model wins on net benefit

A decision-curve conclusion is always tied to a range of thresholds. Near fifty per cent B wins in 78.0% of splits, and that stretch holds up; at forty per cent it is 46.5%, about a coin flip, and the advantage the earlier table showed at that threshold did not survive - but a coin flip means no difference was detected, which does not license the reverse claim that B is worse. At sixty per cent it is 26.5%, lower than fifty, yet forty is higher than sixty, so the three numbers are not monotone and the same table refutes the claim that a higher threshold always costs B.

The treat-all line on a decision curve crosses zero at one particular threshold. Why there?

Show the answer and why

Correct answer: It crosses zero near threshold 0.510 because that is this cohort's event rate - past the event rate, treating everyone is a losing trade

Treat-all has net benefit equal to the event rate minus one minus the event rate times the cost ratio, and the second term grows with the threshold until the two cancel exactly where the threshold equals the event rate - near 0.510 in this cohort. 0.508 is indeed the five-year risk these patients ran, and the near-coincidence of the two numbers is exactly the point being made, but it is not model A's average prediction: A under-predicts systematically and its mean sits well below that. As for -0.229, that is treat-all at the sixty per cent threshold and it is already negative - the crossing is where the sign changes, not wherever some negative value happens to sit.

In the fifty per cent row, model B gives 24 more net true positives per thousand patients than model A. What can that number be used for?

Show the answer and why

Correct answer: At the thirty per cent threshold the same difference is -6, the other way round - a difference like this means something only at the threshold it was computed for

A net-benefit difference is tied to one threshold: at thirty per cent model B is behind, at forty it is a little ahead, at fifty it is 24 ahead per thousand - same pair of models, same patients, only the threshold changed. So the sentence B is better than A has no content without a threshold attached. Nor does it travel between studies: the net-benefit formula contains the event rate, the event rate is a property of a population, and the same model moved to a cohort with a different event rate returns a different net benefit without having become better or worse. And all three thresholds are positive is contradicted by the table itself - the thirty per cent row is negative.

Chapters that use this method

Prediction model study

Sources and licences

This page is original writing