AdvancedIndependently reviewed, not yet spot-checked by a human

ROC curves and the area under them

An ROC curve is the trace of every cut-off at once, and the area under it has one clean probabilistic meaning — the chance that a randomly drawn diseased patient scores higher than a randomly drawn non-diseased one. This page verifies that meaning by counting pairs, then shows the two things AUC cannot see: it says nothing about calibration, and by construction it does not react to prevalence, so it does not degrade on imbalanced data — precision-recall does.

One curve is an entire stack of 2×2 tables

Every sensitivity/specificity pair on the 2×2 table page is attached to one particular cut-off. Move the cut-off and all four cells rearrange.

Sweep the cut-off from the lowest value to the highest, compute sensitivity and 1 − specificity at each position, and join the points. That line is the ROC curve (receiver operating characteristic curve — the name comes from radar signal research during the Second World War).

Here are five cut-offs applied to S100β in the same pROC::aSAH data:

Cut-off (µg/L)	TP / FP / FN / TN	Sensitivity (y axis)	1 − specificity (x axis)
0.05	40 / 67 / 1 / 5	0.976	0.931
0.10	34 / 44 / 7 / 28	0.829	0.611
0.20	26 / 14 / 15 / 58	0.634	0.194
0.40	17 / 8 / 24 / 64	0.415	0.111
0.60	9 / 0 / 32 / 72	0.220	0.000

The lower the cut-off, the more easily a patient is called positive, so sensitivity rises while specificity falls. Every point on the curve is a trade-off, and none of them is inherently the correct one. Which one is correct depends on what a false positive and a false negative each cost, and that is the subject of the next page.

An ROC plot with 1 minus specificity on the horizontal axis and sensitivity on the vertical axis. Two step-shaped curves show S100B and NDKA, with a dashed diagonal for random guessing. The S100B curve sits closer to the top-left corner throughout. — Two biomarkers measured in the same patients. The diagonal is a test carrying no information at all — every gain in sensitivity is matched by an equal rise in the false positive rate. The closer a curve runs to the top-left corner, the more cases it catches at any given false positive rate.Plotting script `figures/scripts/B4-04-roc-auc.R`

AUC: the area under the curve, and what it really means

AUC (area under the curve, also written as the C-statistic or the concordance index) compresses the whole curve into a single number:

Marker	AUC	95% CI (DeLong)
S100β	0.731	0.630–0.833
NDKA	0.612	0.501–0.723

An area is hard to interpret on its own, but it is equal to a very concrete probability:

Draw one patient with a poor outcome and one with a good outcome at random. AUC is the probability that the poor-outcome patient has the higher marker value (ties count as half).

This is not an analogy, it is an identity, and you can count it out for yourself: 41 patients with a poor outcome and 72 with a good outcome make 2952 pairs. In 2124 of them the poor-outcome patient has the higher S100β, and 70 are ties. Counting ties as half: (2124 + 70 / 2) / 2952 = 0.7314, which is exactly the AUC pROC reports, 0.7314 (an assertion in the figure script guards this — if the two disagree, no file is written).

The same identity explains several facts that are usually memorised separately:

AUC sees only the ranking, never the values. Take logs, square everything, multiply by ten — AUC does not change at all, as long as the ordering is unchanged.
AUC is equivalent to the Mann-Whitney U statistic (after dividing by the product of the two group sizes). So “AUC significantly above 0.5” and “the two distributions differ in location” are the same test.
AUC 0.5 means no ranking ability at all, and 1 means perfect ranking. AUC does not get worse just because you enrolled more non-diseased people — which becomes a trap two sections further down, where AUC meets an imbalanced cohort.

The first thing AUC cannot see: calibration

AUC measures discrimination: can the model rank diseased patients above non-diseased ones? It says nothing about calibration: when the model says “this person has a 20% chance of disease”, do 20% of such people actually have it?

The proof takes one line. Shift a set of predicted probabilities by a constant on the log-odds scale. The ranking is untouched, so AUC does not move at all — but every individual predicted probability is now wrong.

Fit a logistic regression on aSAH (poor outcome ~ S100β + NDKA + age), then shift the predictions down by 1.50 log-odds:

	Original predictions	Shifted predictions
AUC	0.7751	0.7751
Brier score (lower is better)	0.174	0.222
Mean predicted probability	36.3%	15.9%
Observed event rate	36.3%	36.3%
Calibration intercept (0 is correct)	0.00	1.50

Left: the ROC curves of the two sets of predictions coincide, and their AUCs agree to four decimal places. Right: the calibration plot for those same two sets of predictions — the shifted set underestimates risk across the board. AUC does not react to any of this. (With only a hundred-odd patients, splitting the calibration plot into four groups leaves about twenty per group, so the points jitter; that is normal behaviour in a small sample.)Plotting script figures/scripts/B4-04-roc-auc.R

The shifted model predicts 15.9% on average while the observed event rate is 36.3% — it systematically underestimates risk. The Brier score deteriorates from 0.174 to 0.222, and AUC does not move.

The Brier score is only corroboration here. On its own it has no scale — it mixes discrimination and calibration together, so a single value means something only against a reference. And computing it directly on censored survival data is wrong. The full account, along with how it relates to the reclassification measures (NRI and IDI), is on Brier score, NRI and IDI.

Why this matters clinically.

Whenever the decision is “rank the patients and take the top few” — who gets scanned first, who gets the intensive care bed, how organs are allocated — discrimination is enough. But as soon as the decision hangs on an absolute probability threshold (“anticoagulate above 10% risk”, “an invasive test is only worth it above 5%”), a miscalibrated model puts patients on the wrong side of the line systematically, and AUC gives you no warning.

The practical requirement: a prediction model report cannot stop at AUC. The TRIPOD reporting guideline asks for discrimination and calibration together — at minimum a calibration plot, and usually the calibration intercept and slope as well. Calibration itself belongs to the B5 prediction-model family, after the next page.

The second thing AUC cannot see: how many of these people are cases

You often hear that “AUC is misleading under severe class imbalance, so use a PR curve instead”. The conclusion is usable, but the reason given is usually wrong, and the mechanism is worth seeing clearly.

Replicate the aSAH control group 20 times over (the cases stay as they are), which drops prevalence from 36.3% to 2.8%:

	Original data	After replicating controls
Total people	113	1481
Prevalence	36.3%	2.8%
AUC	0.7314	0.7314
Average precision (area under the PR curve)	0.686	0.343
(No-information baseline)	0.363	0.028

Left panel: two ROC curves lying exactly on top of each other. Right panel: two precision-recall curves far apart, the low-prevalence one shifted downward along its whole length, with two horizontal dashed lines marking the baseline for each prevalence. — Left: the ROC curves are identical, and the AUCs agree to four decimal places. Right: the precision-recall curves for the very same scores — the low-prevalence one drops along its entire length, because precision is PPV and PPV follows prevalence.Plotting script `figures/scripts/B4-04-roc-auc.R`

The mechanism, stated plainly: AUC is not “broken” — the two plots answer different questions.

Both axes of the ROC plot — sensitivity and 1 − specificity — are proportions computed within a disease status (see the passage on which direction you divide in on the 2×2 table page). Replicating controls changes no proportion that is computed down a column, so the ROC curve cannot move, by definition.

The vertical axis of the PR curve is precision, which is PPV, and PPV is computed across a row. It takes prevalence as an input, so it moves.

The correct version of “AUC is misleading under imbalance” is therefore: AUC does not tell you — and never set out to tell you — how many of the people you flag by working down the score are genuine cases. For that you need PPV or a PR curve, and those values hold only at the prevalence they were computed at.

One detail that is easy to get wrong: average precision falls from 0.686 to 0.343, which looks like the model got worse — but the no-information baseline fell from 0.363 to 0.028 as well. Relative to its own baseline — as a ratio — the second one is actually further ahead. So not even PR numbers can be compared across prevalences at face value; read them alongside their baseline.

Running it yourself

library(pROC)
data(aSAH, package = "pROC")

r1 <- roc(aSAH$outcome, aSAH$s100b,
          levels = c("Good", "Poor"), direction = "<", quiet = TRUE)
r2 <- roc(aSAH$outcome, aSAH$ndka,
          levels = c("Good", "Poor"), direction = "<", quiet = TRUE)

auc(r1); ci.auc(r1, method = "delong")
plot(r1); plot(r2, add = TRUE, col = "steelblue")

# The 2x2 table behind any point on the curve
coords(r1, x = c(0.05, 0.10, 0.20, 0.40, 0.60), input = "threshold",
       ret = c("threshold", "sensitivity", "specificity", "tp", "fp", "fn", "tn"),
       transpose = FALSE)

# The probabilistic reading of AUC, counted out by hand
case <- aSAH$s100b[aSAH$outcome == "Poor"]
ctrl <- aSAH$s100b[aSAH$outcome == "Good"]
(sum(outer(case, ctrl, ">")) + 0.5 * sum(outer(case, ctrl, "=="))) /
  (length(case) * length(ctrl))

# Same ranking => same AUC
auc(roc(aSAH$outcome, log(aSAH$s100b), levels = c("Good", "Poor"),
        direction = "<", quiet = TRUE))

# Exported for the Python block below
write.csv(aSAH, "aSAH.csv", row.names = FALSE)

Verified with R 4.6.0 and pROC 1.19.0.1. It is worth writing levels and direction out explicitly in roc(); otherwise it guesses the direction for you and prints a message about it.

import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score, roc_curve, average_precision_score

d = pd.read_csv("aSAH.csv")            # exported by the R block above
y = (d["outcome"] == "Poor").astype(int)

print(roc_auc_score(y, d["s100b"]))
print(roc_auc_score(y, d["ndka"]))

fpr, tpr, thr = roc_curve(y, d["s100b"])

# The probabilistic reading
case = d.loc[y == 1, "s100b"].to_numpy()
ctrl = d.loc[y == 0, "s100b"].to_numpy()
cmp = case[:, None] > ctrl[None, :]
tie = case[:, None] == ctrl[None, :]
print((cmp.sum() + 0.5 * tie.sum()) / (len(case) * len(ctrl)))

# Area under the PR curve, and its baseline
print(average_precision_score(y, d["s100b"]), y.mean())

sklearn's roc_auc_score agrees with pROC (it also counts ties as half); average_precision_score corresponds to the area under the PR curve.

Four questions to ask when you read an AUC in a paper

Is there a confidence interval? The S100β AUC on this page is 0.731, interval 0.630–0.833, more than 0.2 wide. Diagnostic studies are often small, and a bare point estimate leaves the reader overestimating how precise it is.
Which data was this AUC computed on? An AUC computed on the same data the model was fitted to is optimistic; you want the value after internal validation (bootstrap or cross-validation), or one from external validation. The next page measures that optimism on this very dataset.
Is calibration reported? A prediction model paper carrying only an AUC has answered half the question.
How much of an AUC gain is clinically meaningful? Adding a new marker to an existing model often moves AUC by a few thousandths. Whether that is worth another tube of blood and another bill is not a question AUC can answer. Answering it needs decision-analytic tools (net benefit, decision curve analysis), or a direct look at how many people are reclassified near the clinical decision threshold.

Common misuses

Misuse	Why it is wrong
Treating AUC as the single answer to “is this model any good”	AUC measures discrimination only and is blind to calibration
Comparing AUCs from two different papers head to head	Populations, prevalence and case mix differ; AUC is not comparable across studies
Claiming one marker is better because the two AUCs look far apart	Two curves from the same patients are correlated; use a paired DeLong test
Reporting an AUC with no confidence interval	In a small sample the interval is wide, and the point estimate misleads
Reporting an AUC on the data the model was fitted to without saying so	Optimism bias; correct it with internal or external validation
Believing AUC deteriorates on imbalanced data	Both ROC axes are computed within disease status, so by definition prevalence does not enter
Comparing areas under PR curves across different prevalences	The PR baseline is the prevalence itself; read the area alongside its baseline
Deciding whether to add a new biomarker on AUC alone	A small AUC gain has no fixed relationship to clinical value; that needs decision analysis
Picking the best-looking point on the ROC curve as the recommended cut-off	That point was chosen on the same data, and “best-looking” is itself a cost assumption — one the prevalence of that dataset ends up making for you
Describing AUC as “diagnostic accuracy”	It is the probability of ranking a pair correctly, not the accuracy of any one cut-off
Teaching ROC with the Ct values from covid_testing	The Ct value comes from the very PCR result it is being validated against — circular

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B4-04-roc-auc.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

The AUC for this page's biomarker is 0.731. What does that number mean?

Show the answer and why

Correct answer: Draw one poor-outcome and one good-outcome patient at random; the probability that the first has the higher marker value is 0.731, counting ties as half over 2952 pairs

The AUC equals one very specific probability: draw a poor-outcome and a good-outcome patient at random and ask how often the first has the higher marker value, with ties counted as half. That is an identity rather than an analogy, and this page counts it out over 2952 pairs. So the AUC sees only the ordering: take logs of every marker value or multiply them all by ten and it does not move. 0.634 is the sensitivity at the cut-off of 0.20, which belongs to one operating point, whereas the AUC summarises the whole curve; calling the AUC a diagnostic accuracy suggests it is tied to some threshold. 0.775 is the AUC of the logistic model later on this page, and however high an AUC gets it still says nothing about whether the predicted probabilities themselves are right - that is calibration.

A set of predicted probabilities is shifted down by 1.5 on the log-odds scale, and the AUC goes from 0.7751 to 0.7751, not moving at all. Does that make the shifted model just as good?

Show the answer and why

Correct answer: Not as good. After the shift the mean predicted probability is 0.1591 while the observed rate is 0.3628 - every patient's risk is now systematically understated

The shift changes how large each prediction is but not the order they come in, and the AUC sees only order, which is why it is identical to four decimal places. Calibration is the other question: after the shift the mean prediction is 0.1591 against an observed rate of 0.3628, so the model understates risk systematically. The Brier score does worsen, from 0.1741 to 0.2218, and that direction is not wrong: it absorbs discrimination and calibration at once, so a calibration failure shows up in it. What is wrong is reading that deterioration as lost discrimination. Not one pair of predictions swapped places, the AUC is identical to four decimal places, and every bit of the deterioration is calibration. A Brier score tells you the model got worse; telling you which part got worse takes the AUC beside the gap between the mean prediction and the observed rate. Whenever a decision hangs on an absolute probability threshold, a miscalibrated model puts patients on the wrong side systematically, and the AUC cannot see any of it.

The controls are duplicated 20 times over, taking prevalence from 36.3% down to 2.8%. What happens to the AUC and to the average precision of the PR curve?

Show the answer and why

Correct answer: The AUC does not move, and average precision falls to 0.3426, because precision is PPV and PPV takes in prevalence

Both axes of the ROC curve - sensitivity and one minus specificity - are proportions computed inside a disease group, and duplicating controls changes neither, so the AUC cannot move by definition and stays 0.7314. The vertical axis of the PR curve is precision, which is PPV and divides along a row, so it falls from 0.6856 to 0.3426. That does not mean the model got worse: the no-information baseline fell from 36.3% to 2.8% as well, and as a multiple of its own baseline the second one is further ahead. So PR numbers cannot be compared across prevalences either, and have to be read alongside their own baseline.

This page's biomarker has an AUC of 0.731 (95% CI 0.630 to 0.833) and NDKA has 0.612 (0.501 to 0.723). Can you write that the former outperforms NDKA?

Show the answer and why

Correct answer: No. The lower bound for the former of 0.630 sits below the upper bound for NDKA, the intervals overlap substantially, and both curves come from the same patients

The point estimates look quite far apart, but the lower bound for the former of 0.630 sits well below the upper bound for NDKA of 0.723 and the intervals overlap substantially. More importantly, both markers were measured in the same patients, so the two AUC estimates are correlated and cannot be compared as if they came from independent samples - the right procedure is a paired DeLong test, which is on the cut-off page. The lower bound for NDKA of 0.501 belongs to a different question, namely whether NDKA discriminates at all, not whether the two differ. Until the paired test is done, do not write that one outperforms the other.

In the ROC table, the row for the cut-off of 0.05 has a sensitivity of 0.976 and one minus specificity of 0.931. Does that row show this is a good test?

Show the answer and why

Correct answer: It does not. The false-positive rate in the same row is 0.931, so this point sits almost in the top-right corner

The top-right corner of an ROC curve is the cut-off set so low that everyone is called positive. Every test has that point and it says nothing about how good the test is. The coordinates for the cut-off of 0.05 are a sensitivity of 0.976 against a false-positive rate of 0.931, essentially in that corner, and the high sensitivity was bought by letting almost nobody through. The row with a false-positive rate of 0.000 is the opposite extreme, near the bottom-left corner, and its sensitivity is barely above one fifth, so it is no more inherently right. Every point on the curve is a trade-off, and which one is right depends on the cost of a false positive against the cost of a false negative, not on which point looks best on the plot.

The ROC curve is redrawn after taking logs of every patient's marker value. What does the AUC become?

Show the answer and why

Correct answer: Still 0.7314, because taking logs does not reorder any pair of patients and the AUC sees only order

The AUC is the probability that a randomly drawn poor-outcome patient has a higher marker value than a good-outcome one, so it is decided entirely by the ordering. Any strictly increasing transformation - logs, square roots, multiplying by ten - leaves every pair in the same order, so the AUC does not move at all and stays 0.7314. The same property explains why the AUC is equivalent to the Mann-Whitney U statistic. 0.7751 is the AUC of the three-variable logistic model on this page and 0.6120 is the AUC of the other marker, NDKA; both are real numbers and neither is what this transformation would produce. Read the other way, the fact that the AUC does not move is also its limit: shift the whole set of predicted probabilities and it will not move either.

Chapters that use this method

Watch next

ENStatQuest with Josh Starmer· 16 minAnimates the act of sliding the cut-off. Watch it before the first section and the way the curve is generated becomes concrete.

ENTerry Shaneyfelt· 5 minA five-minute clinical overview, good for returning to the diagnostic setting after StatQuest.

ENData School· 14 minFills in the probabilistic reading of AUC and how it relates to choosing a threshold — the material of the fourth section here.

繁中EDMAN MURMURS· 13 minFirst episode of a six-part series in Traditional Chinese; useful if you need the Chinese wording for these terms.

繁中EDMAN MURMURS· 13 minIn Traditional Chinese, and it lands exactly on the two blind spots covered in sections five and six here.

Sources and licences

This page is original writing