Diagnostic accuracy study

What this chapter covers, and what the method pages cover

How sensitivity, specificity, likelihood ratios and ROC curves are calculated and read belongs to the method pages — the B4 family.

This chapter asks the design-level questions instead: which people did these numbers come from, who was left out, and can the reference standard itself be trusted. Because the most awkward thing about diagnostic accuracy is this:

The reference standard: the ground the whole study stands on

A diagnostic accuracy study compares the index test against a reference standard (formerly “gold standard”). Every number it produces means something only if the reference standard is right.

In practice that premise never holds completely:

Verification bias: what happens when only the positives get verified

The most common design compromise is this: the reference standard is too invasive or too expensive, so it is applied only to those who tested positive. Everyone who tested negative is written down as disease-free. That is verification bias, also called work-up bias.

Its direction can be derived, and you can watch it happen by simulating on one dataset. The table below lowers the proportion of test-negative people who receive the reference standard, from everyone down to one in ten, and reports the apparent sensitivity and specificity each time:

(The count column is an expected value, not the result of one draw — the script multiplies the proportion by the number of test-negatives, which is why fractions appear.)

The truth is fixed at sensitivity 0.634 and specificity 0.806. Verify only half the test-negatives and the apparent sensitivity climbs to 0.776 while the specificity falls to 0.674.

That diagram looks like this:

Each index test result splits into performed and not performed. Of the 214 positives, 205 were verified; of the 590 negatives, only 236 were, leaving 354 in the red box. The leftmost column of the table above — the proportion of test-negatives verified — is the ratio between that box and the one to its left. In this diagram it is 40%, which sits between the 50% and 30% rows of the table.

Notice also that the 2×2 hangs only off the performed boxes — and not even off all of them. Of the 822 who took the index test, 455 received the reference standard, but the 2×2 is built on 441: the 14 verified inconclusive results have no cell to sit in, because an inconclusive index test is neither a positive nor a negative call, and STARD asks for them to be reported separately rather than folded into either column. That is the whole point: the denominator behind the sensitivity and specificity a paper prints is 441, and it is the reader who has to find it and subtract to see who was left outside the table — the 367 never verified, plus the 14 verified but uncountable.

A softer variant is differential verification: test-positives get the invasive reference standard while test-negatives get a weaker one — clinical follow-up, say, or a second imaging study. Nobody is left unverified, so the flow diagram looks complete, but the two groups were not held to the same standard. This is not the same bias as partial verification, and its direction depends on how the weaker standard misclassifies: a follow-up that misses slowly progressing disease will file those patients as true negatives and inflate sensitivity, while one that over-calls will do the reverse. The question that catches it is the one QUADAS-2 asks outright — did every patient receive the same reference standard?

The spectrum effect: one test, a threefold difference in sensitivity

Split the same dataset by disease severity (WFNS grade). Same test, same cut-off:

Sensitivity in the milder stratum is 0.286; in the severe stratum it is 0.815 — close to a threefold difference, and the two confidence intervals do not overlap. Specificity moves the other way.

Why would specificity fall? One plausible reading: in the higher-WFNS stratum, even the patients with a good outcome have sustained more brain injury, so their S100β runs higher, so more of them cross the same cut-off and are called false positives — a larger share of the disease-free group is misclassified, and specificity drops. That is a plausible explanation built from the relationship between stratum and marker concentration, not a causal mechanism these data can confirm: confirming it would mean comparing the S100β distribution of the good-outcome patients across the two strata directly, rather than looking only at accuracy indices.

The two extra columns in the table are a reminder of where each denominator comes from: the precision of sensitivity is governed only by the number of diseased patients, and the precision of specificity only by the number of disease-free ones. The severe stratum contains just 15 disease-free patients, which is why its specificity interval is wide enough to carry almost no information — how much of that drop is real and how much is sampling noise is something these data cannot separate.

This is the spectrum effect, and the reason is blunt: severely ill patients have higher biomarker concentrations and are easier to detect, and if the “healthy controls” are people with no symptoms at all, they are easier to call negative.

Sample size: the confidence interval tells you whether you had enough

The table above demonstrates something else. The sensitivity interval in the milder stratum runs from 0.12 to 0.55 — wide enough to be nearly uninformative. That stratum contains only 14 diseased patients, and that number is the denominator of its sensitivity.

Sample size in a diagnostic accuracy study is set by the number of cases and the number of controls, not by the number of people enrolled. A study of 1000 participants containing 20 confirmed cases has a sensitivity whose precision is decided by those 20.

The design choices available

Blinding

Blinding is needed in three directions:

1. Whoever reads the index test does not know the reference standard result
2. Whoever reads the reference standard does not know the index test result
3. Neither knows the other clinical information — unless that information would be available in practice anyway, in which case say so

For tests requiring human interpretation, such as imaging and pathology, this matters more than anywhere else, and inter-reader agreement has to be reported — usually kappa, which has a property that catches people out: two datasets with the same observed agreement can differ threefold in kappa purely because the abnormal reading is rarer in one of them.

Common misuses

Where the numbers on this page come from

Every figure quoted here comes from the analysis scripts behind method pages B4-01 and B4-02:

/opt/homebrew/bin/Rscript figures/scripts/B4-01-sens-spec.R
/opt/homebrew/bin/Rscript figures/scripts/B4-02-ppv-npv.R
/opt/homebrew/bin/Rscript figures/scripts/D4-stard.R

The third script produces only the STARD diagram and its figures/out/D4-stard.stats.json. That file is marked hypothetical: true, because the counts in the diagram were invented to show the structure rather than measured from pROC::aSAH — in that dataset every patient received the reference standard, so it has no unverified branch to draw.

Watch next

▶Cohort Studies: A Brief Overview

Report a problem on this page