Choosing a cut-off, and comparing two ROC curves

What is hiding inside the word “optimal”

Every point on the ROC curve is one trade-off between sensitivity and specificity. To turn a continuous test value into a rule a clinician can actually follow, you have to pick one.

The trouble is that “optimal” has no built-in definition. Picking a point requires two pieces of information that are not in the data:

1. What a false positive and a false negative each cost (missing a case versus subjecting someone to an unnecessary invasive test)
2. The pre-test probability of the population the rule will be applied to

The most common approach in practice — the Youden index — quietly fills both of these in with defaults. This page digs those defaults back out.

The Youden index

The Youden index is defined as:

Geometrically it is the vertical distance between a point on the ROC curve and the diagonal, running from 0 (no information) to 1 (perfect). The Youden optimal cut-off is the point that maximises $J$.

For S100β in pROC::aSAH:

(The cut-off pROC reports is the midpoint between two adjacent observed values, which is why you see a decimal that appears nowhere in the data: 0.205 µg/L. No patient in this dataset has an S100β between the round 0.20 µg/L used on the earlier pages of this family and 0.205 µg/L, so that coarser 0.20 µg/L cut-off happens to yield exactly the same sensitivity and specificity here — two different numbers that put the same people on the same side in this particular dataset.)

That flat summit has a practical consequence: 6 cut-offs sit within 0.05 of the maximum, spread from 0.175 to 0.245 µg/L; across that range sensitivity runs from 58.5% to 63.4% and specificity from 76.4% to 81.9%. Writing an “optimal” cut-off into a clinical guideline to three decimal places reports a precision the data do not contain.

Choosing a cut-off by cost, and how it collapses

The standard way to bring costs in is to maximise a weighted version:

where $r$ is “the cost of missing a case ÷ the cost of one false positive”. pROC’s coords(..., best.weights = c(r, prev)) does exactly this.

Take $r$ from 1 up to 10, holding prevalence at this cohort’s 36.3%:

Several cost ratios appear on more than one row: that is because at that weight the optimum is not unique. coords(..., "best") returns every cut-off that attains the maximum, and taking only the first row drops the tied solutions silently. Five things are worth noticing:

• The 1 : 1 row is not the Youden index. The prevalence term never leaves the formula: this cohort’s prevalence of 36.3% makes the weight 1.756, so even when the two errors are equally costly, specificity is still weighted more heavily than sensitivity (there are more non-diseased people, so each percentage point of specificity involves more of them). At a cost ratio of 1 the weight is exactly $(1-\text{prev})/\text{prev}$, and the whole objective reduces to “number classified correctly ÷ number of cases” — what this row actually maximises is overall accuracy, not $J$. In the language of the decision curve page, this row is a threshold probability of one half; the Youden cut-off is the row you would get at a threshold probability of 36.3%, which is why the two are the same objective only in a cohort split evenly between cases and non-cases.
• And on this row the optimum is not unique either. The cut-off 0.205 (sensitivity 63.4%, specificity 80.6%) and the cut-off 0.510 (sensitivity 29.3%, specificity 100.0%) classify the same 84 people correctly, with identical objective values (2.049). The first happens to coincide with the Youden choice — a coincidence, not an identity. The second is a clinically almost opposite rule: it hardly ever mislabels a healthy person, at the price of missing seven out of ten cases.
• For cost ratios between 1 and 3, the chosen cut-offs all include 0.205. The cut-off does not respond continuously to the cost ratio — it sticks on one observed value for a while, then jumps in one step. So uncertainty of the “is the cost ratio about two or about three” kind often makes no difference to the decision at all.
• At 4 : 1 the cut-off drops to 0.065, sensitivity leaps to 97.6% and specificity is left at 13.9%.
• At 10 : 1 the optimum is again not unique: the degenerate solution “set no cut-off, call everyone positive” has exactly the same objective value as the finite cut-off 0.065 µg/L (both 1.000).

A cut-off chosen on the same data is optimistic

This is the most important section on the page. The act of choosing a cut-off spends information from the data: you picked, out of many candidates, the one that performed best in this sample, and “best in this sample” is part real signal and part the random fluctuation of this particular sample. Reporting how that cut-off performs on the same data counts the fluctuation as achievement.

The way to quantify it is bootstrap optimism correction: in every resample, rerun the entire selection procedure, then compare its performance in the resample with its performance in the original data. The difference is that replicate’s optimism.

Over 2000 resamples:

Comparing two ROC curves: the DeLong test

This section and the two that follow all answer the same question. The S100β AUC is 0.731, NDKA’s is 0.612, a difference of 0.119. Does that difference hold up?

The key point is that both markers were measured in the same patients, so the two AUC estimates are correlated and cannot be compared as if they came from two independent samples. That is precisely what the DeLong test handles: it uses each participant’s contribution to the AUC (structural placement values) to compute the variance of each AUC and the covariance between them, then tests the difference.

A claim this page nearly got wrong

Writing the DeLong section, the natural way to close it is the line you see everywhere: “because these are the same patients, the paired test exploits the correlation and is therefore more efficient than an unpaired one.”

On this dataset, that line is false. Test the same two curves as if they were unpaired:

Ignoring the pairing gives the smaller p value. The reason is in the algebra — the variance of a difference is

The covariance is subtracted, but it is not guaranteed to be positive. In these data the covariance between the two AUC estimates is -0.00076, which is negative (the Spearman correlation between the markers themselves is also close to zero and slightly negative, -0.060). So $-2\mathrm{Cov}$ becomes an addition, the paired variance 0.0074 exceeds the unpaired 0.0059, and the test is more conservative.

Partial AUC: only the region you will actually use

Both tests in the previous sections were run on the whole curve. But a large stretch of that curve consists of operating points no one would adopt: in screening and imaging, a false positive goes on to an invasive confirmatory test, so a cut-off whose specificity falls below some value never makes it into the pathway at all. The area under the whole ROC curve includes that stretch, which means part of the score comes from how the test behaves at operating points it will never be used at.

The partial AUC restricts the integral to a stated region of specificity. This section uses specificity 0.90 to 1.00 — the high-specificity end, where most screening and image-reading rules actually operate.

Uncorrected partial areas cannot be compared across regions, because a wider region has room for more area. The ceiling on the region from specificity 0.90 to 1 is simply its width, 0.10; from specificity 0.80 to 1 it is 0.20. So the S100β value of 0.0328 in the narrow region and 0.0806 in the wide one are not on a common scale — part of why the second is larger is simply that the region is wider. For the same reason, a paper reporting a pAUC without naming the region has reported an uninterpretable number.

The floor moves too. Even the diagonal — a test with no discriminative ability at all — encloses some area inside the region (0.005 and 0.020 here), so the no-information benchmark for an uncorrected partial area is neither 0.5 nor 0, but a quantity that changes with the region.

McClish’s correction maps the region linearly onto 0.5 to 1. It subtracts the area under the diagonal, then divides by the range the region can offer:

where the upper limit is the width of the region and the lower limit is the area under the diagonal inside it. In pROC this is partial.auc.correct = TRUE; in sklearn it is what roc_auc_score(..., max_fpr = ...) returns by default. Corrected values are comparable across regions and against the 0.5 no-information line — but a corrected partial area is not a whole-curve AUC, and the two cannot stand in for each other.

How the paired comparison is done. pROC refuses to run DeLong on a partial AUC outright — the error reads DeLong's test is not supported for partial AUC — because the DeLong variance is derived for the placement values of the whole curve. The alternative is a paired bootstrap: each resample draws patients, so both markers travel together and the pairing survives, and the distribution of the difference in regional area is taken from those resamples. This page uses 2000 of them.

One argument must not be left out: reuse.auc = FALSE. It defaults to TRUE, meaning the AUC specification already stored in the roc object is reused and the partial.auc you just passed in is silently ignored — so you believe you are comparing a region while you are still comparing the whole curve, the output looks entirely normal, and the only clue is that the p value is identical to the whole-curve one.

Standardising does not change the strength of the evidence. The McClish correction is an affine transform of the raw area, so the difference and the bootstrap standard deviation are rescaled by the same constant and the test is untouched: on the raw scale the statistic is 2.283 with p = 0.022, on the corrected scale 2.283 with p = 0.022. What the correction buys is comparability, not significance.

On these data the two comparisons reach different conclusions. Across the whole curve no difference was detected (p = 0.164); restricted to specificity 0.90 and above, with the same patients and the same paired design, the difference is statistically significant (p = 0.022). And that is not an artefact of switching tests — the same 2000 paired bootstrap resamples applied to the whole curve give p = 0.169, essentially the DeLong value of 0.164. What changed is the range being compared.

The mechanism is visible in the numbers. In the high-specificity region S100β reaches 32.8% of the available area and NDKA only 10.7%, a factor of about 3.1; across the whole curve the ratio of the two AUCs is only about 1.20. A whole-curve AUC averages the stretch where the markers differ most with the stretches where they differ less, so a difference concentrated in one region gets diluted in the summary.

Running it yourself

library(pROC)
data(aSAH, package = "pROC")

r1 <- roc(aSAH$outcome, aSAH$s100b, levels = c("Good", "Poor"),
          direction = "<", quiet = TRUE)
r2 <- roc(aSAH$outcome, aSAH$ndka, levels = c("Good", "Poor"),
          direction = "<", quiet = TRUE)

# The Youden optimal cutoff, and the whole Youden curve
coords(r1, "best", best.method = "youden",
       ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)

# Cutoff by cost ratio: c(relative cost of missing a case, prevalence in the setting)
coords(r1, "best", best.weights = c(5, 0.36),
       ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)

# Compare the two curves, paired
roc.test(r1, r2, method = "delong")          # paired defaults to TRUE
cov(r1, r2, method = "delong")               # the covariance, which may be negative

# Partial AUC: restrict the integral to the high-specificity region
auc(r1, partial.auc = c(1, 0.9), partial.auc.focus = "sp")
auc(r1, partial.auc = c(1, 0.9), partial.auc.focus = "sp",
    partial.auc.correct = TRUE)              # McClish, rescaled onto 0.5 to 1

# Compare two partial AUCs. DeLong errors out on a partial AUC, so this is the
# bootstrap; reuse.auc = FALSE is required or partial.auc is silently ignored
# and you are still comparing whole curves
set.seed(20260824)
roc.test(r1, r2, method = "bootstrap", reuse.auc = FALSE,
         partial.auc = c(1, 0.9), partial.auc.focus = "sp")

# When the question is really sensitivity at a fixed specificity, compare that
roc.test(r1, r2, method = "specificity", specificity = 0.9)

# Optimism correction: every resample must rerun the whole selection procedure
youden_at <- function(y, x, thr) {
  d <- y == "Poor"; p <- x >= thr
  sum(p & d) / sum(d) + sum(!p & !d) / sum(!d) - 1
}
best_thr <- function(y, x)
  as.numeric(coords(roc(y, x, levels = c("Good", "Poor"), direction = "<",
                        quiet = TRUE), "best", ret = "threshold",
                    transpose = FALSE)[1, 1])

app <- youden_at(aSAH$outcome, aSAH$s100b, best_thr(aSAH$outcome, aSAH$s100b))
opt <- replicate(2000, {
  b  <- aSAH[sample(nrow(aSAH), replace = TRUE), ]
  tb <- best_thr(b$outcome, b$s100b)
  youden_at(b$outcome, b$s100b, tb) - youden_at(aSAH$outcome, aSAH$s100b, tb)
})
c(apparent = app, optimism = mean(opt), corrected = app - mean(opt))

# Exported for the Python block below
write.csv(aSAH, "aSAH.csv", row.names = FALSE)

import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score

d = pd.read_csv("aSAH.csv")            # exported by the R block above
y = (d["outcome"] == "Poor").to_numpy().astype(int)
a1, a2 = d["s100b"].to_numpy(), d["ndka"].to_numpy()

# Paired bootstrap: resample PATIENTS, so both markers travel together
# and the pairing structure survives
rng = np.random.default_rng(20260822)
diff = []
for _ in range(2000):
    i = rng.integers(0, len(y), len(y))
    if y[i].sum() == 0 or y[i].sum() == len(i):
        continue
    diff.append(roc_auc_score(y[i], a1[i]) - roc_auc_score(y[i], a2[i]))
diff = np.array(diff)
print(roc_auc_score(y, a1) - roc_auc_score(y, a2),
      np.percentile(diff, [2.5, 97.5]))

# Partial AUC: max_fpr returns the McClish-standardised value (0.5 to 1),
# the same quantity as partial.auc.correct = TRUE in pROC
print(roc_auc_score(y, a1, max_fpr=0.1), roc_auc_score(y, a2, max_fpr=0.1))

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B4-05-cutoff.R

Watch next

▶ROC 系列 5/6：最佳切點與代價權衡

▶ROC 系列 6/6：罕見疾病與過度樂觀陷阱

▶ROC Curves

▶Biostatistics – All You Need To Know About The ROC Curve

Report a problem on this page