ExpertIndependently reviewed, not yet spot-checked by a human

Time-dependent ROC and cumulative-dynamic AUC(t)

Harrell's C compresses discrimination at every horizon into a single number, but clinical decisions happen at particular horizons — what you tell a patient is their five-year risk. This page unfolds one breast cancer Cox model's discrimination into a curve over time: how a survival dataset is split into cases and controls at a horizon t, why censored patients force inverse probability of censoring weighting, why AUC(t) sits systematically above Harrell's C, and how to read the curve when it turns out flat — the flatness is itself the conclusion, because you cannot know in advance that it will be flat.

One C-statistic cannot show you what it left out

The prediction model chapter and the three pages under it — internal validation, external validation and calibration — all rest on the same Cox model, and the discrimination column carries exactly one number throughout: Harrell’s C.

That number is built like this. Take every pair of patients whose event times can be ordered, check whether the model put the one who failed first ahead of the other, and average over all such pairs. The trouble is in the word all. Discrimination at two years, discrimination at five years, and a pair of patients who both survived into the sixth year before relapsing are all flattened into the same average.

Clinical decisions do not happen at the average over comparable pairs. What you tell a woman who has just had surgery is her probability of recurrence or death within five years, or within two. That is a question about a particular horizon. When a reviewer asks what the five-year AUC is, they are asking the same thing.

This page unfolds that one number into a curve.

AUC(t) asks a question about one horizon

Pin a horizon $t$ — say five years. Once it is pinned, every patient has one of two identities:

case — the event (here recurrence or death, whichever came first) has already happened by $t$
control — still event-free at $t$

With cases and controls you have an ordinary binary classification problem, and therefore the whole apparatus of the ROC curve and AUC. This particular version is called the cumulative/dynamic AUC: cases have accumulated up to $t$ (cumulative), controls are those event-free at that instant (dynamic). As a probability:

\mathrm{AUC}(t) = P\bigl(\hat r_i > \hat r_j \;\big|\; T_i \le t,\; T_j > t\bigr)

where $\hat r$ is the model’s risk score — for a Cox model, the linear predictor. This is the same sentence as the probability interpretation on that page. The one difference is that who counts as a case now depends on $t$ : the same patient can be a control at two years and a case at five.

Censored patients are stuck in between, and that is what forces the weighting.

A patient is lost to follow-up in year three and never heard from again. When you compute AUC(5 years), is she a case or a control? You do not know. She may have relapsed in year four (case) or still be well in year six (control). Such patients are not a fringe group in a real cohort: in the validation cohort on this page only 36 patients are still at risk by 6.0 years, and everyone else has either had the event or been censored.

Three common ways of handling them, of which only one is right:

Count censored patients as controls — this assumes everyone lost to follow-up stayed well, and inflates the AUC.
Drop them — the remainder no longer represents the original cohort, and being lost to follow-up is usually related to prognosis in the first place.
Inverse probability of censoring weighting (IPCW) — keep them, but weight each patient still uncensored at $t$ by $1/\hat G(t)$ , where $\hat G$ is the Kaplan-Meier estimate of the censoring distribution (fit once with the roles of event and censoring swapped). The intuition: every patient observed all the way to $t$ has to speak for the comparable patients who dropped out early.

The timeROC call on this page takes the third route, with marginal (Kaplan-Meier) censoring weights. IPCW is valid under the assumption that censoring is independent of the outcome given the covariates — the same assumption discussed on the censoring page. Changing the estimator does not make it go away; it puts it to work.

The model is the one from the external validation page

Not introducing a new dataset is deliberate. The whole argument here is one curve versus one number, and the comparison is only honest if the number is the same Harrell’s C the reader already met on the external validation page.

Development is survival::rotterdam (2982 patients, 1713 recurrence-free survival events); validation is survival::gbsg (686 patients, 299 events, longest follow-up 7.28 years). The four covariates are:

age -- age at surgery, years, continuous
size_mm -- tumour size in millimetres; rotterdam's bands (<=20, 20-50, >50) mapped to their midpoints 15, 35, 60 to match gbsg's millimetres
nodes -- number of positive lymph nodes, continuous
grade3 -- 1 if tumour grade >= 3, else 0; collapsed because the two cohorts do not grade with the same detail

Unfolding discrimination over time

A line chart. The horizontal axis is the prediction horizon, from 1.0 to 6.0 years. The vertical axis is the time-dependent AUC(t) and spans the full range 0.50 to 1.00 on purpose, rather than being cropped to the narrow band the curve actually occupies, because cropping would turn a flat curve into a dramatic one. A dark blue line with round markers gives AUC(t) at 11 horizons; it drifts gently between 0.669 and 0.711, lowest at 4.5 years and highest at 6.0 years. A pale blue 95% confidence band wraps the line, wider at both ends than in the middle and widest at the right (0.630 to 0.792 at 6.0 years). A red dashed horizontal line marks the same model's Harrell's C at 0.642; the blue line lies above it at every horizon, though the confidence band still covers it at most of them. A brown dotted horizontal line sits at 0.50, labelled at the lower right as a coin flip, meaning no ranking ability at all. A three-item legend sits at the top right, and a subtitle under the title states the range and where the extremes fall. — Discrimination of one model in an external cohort, estimated horizon by horizon with IPCW. The red dashed line is that same model's Harrell's C — one number, spanning every t. Drawing the axis all the way from 0.50 to 1.00 is deliberate: cropping the axis makes every curve look eventful.Plotting script `figures/scripts/B5-08-time-dependent-auc.R`

Why the whole curve sits above Harrell’s C

Most readers’ first reaction to this figure is to ask which of the two numbers is wrong. Of the 11 horizons on the grid, 11 have an AUC(t) point estimate above the same model’s Harrell’s C (0.642); even the lowest point on the curve, at 4.5 years, is 0.028 above it, and the largest gap anywhere is 0.069.

Nothing is wrong. The two quantities are not estimating the same thing.

AUC(t) asks one question only: does a patient who failed before $t$ have a higher score than a patient still event-free at $t$ ? It never compares two patients who both failed before $t$ — at that horizon they belong to the same class.
Harrell’s C does compare them. It averages over every orderable pair, including pairs like one relapse in year five and another in year six. Ranking two patients whose prognosis is equally poor and whose event times differ by months is close to a coin flip, and in a long-follow-up cohort there are many such pairs, pulling the average down towards 0.5.

There is a second source: Harrell’s C depends on the censoring distribution, because which pairs count as comparable depends on who was censored when, whereas the AUC(t) here has already corrected for censoring through IPCW.

A consistent direction is not the same as a detectable gap at each horizon, though: at 8 of the 11 horizons the 95% confidence interval still covers Harrell’s C. The defensible sentence is therefore that AUC(t) sits systematically above Harrell’s C — not that every horizon differs from it by a statistically distinguishable amount, which this data cannot support.

So which one should be reported?

Not one or the other. A C-statistic is still the cheapest summary of whether the model ranks patients correctly overall, and both reviewers and readers recognise it. AUC(t) answers whether it ranks them correctly at the horizon where the decision is actually made. A prediction model paper can report both; the extra cost is one figure.

If what you want is a single number that does not depend on the censoring distribution, that thing is Uno’s C, also built on IPCW. This site has no page for it. The criterion for deciding whether you need it is direct: a high proportion censored, or two cohorts whose follow-up lengths differ a lot. Those are the situations in which Harrell’s C becomes hard to compare across cohorts, and they are exactly what it is criticised for.

The flatness is the conclusion

Now face the shape of this curve honestly.

For this model, in this external cohort, the 11 horizons give AUC(t) values between 0.669 (at 4.5 years) and 0.711 (at 6.0 years) — a range of 0.042 and a standard deviation of 0.014.

Quoting the range on its own overstates how much the curve moves. The 95% confidence intervals at the two extremes overlap (0.620–0.719 against 0.630–0.792), which means the visible wobble lies inside the curve’s own uncertainty: no variation of AUC(t) over time was detected in this data. That is not the same statement as AUC(t) being constant. None of the 11 intervals is narrow, so a moderate amount of variation cannot be ruled out either.

The three horizons an oncology paper would actually quote are closer still:

Horizon	AUC(t)	95% CI	Still at risk	Cumulative events
2.0 years	0.674	0.624–0.723	458	165
3.0 years	0.694	0.649–0.739	331	224
5.0 years	0.675	0.620–0.731	121	285

Those three differ by 0.020.

Then why not just report one Harrell’s C?

Because you cannot know in advance that the curve is flat.

Its flatness is a property of this model, these four covariates and this external cohort — not a property of cumulative/dynamic AUC. Change the covariates, or move the same model to a different cohort, and the curve can slope down (the model only predicts short-term events well), slope up (early events driven by acute factors the model never recorded), or bulge in the middle.

Compressing discrimination into one number means accepting whatever differences get flattened. The problem is not the flattening itself — it is that you only learn what the flattening cost you after you have looked at the curve. Here that cost happens to be small: the 11 horizons span 0.042, and the three clinically quoted ones span 0.020. That is something the analysis told us, not something we saved by skipping it.

Why the confidence band flares at the right

Laying the whole grid out shows where the range comes from:

Horizon	AUC(t)	95% CI	CI width	Still at risk	Cumulative events
1.0 years	0.684	0.607–0.761	0.154	602	56
1.5 years	0.687	0.632–0.742	0.110	530	117
2.0 years	0.674	0.624–0.723	0.098	458	165
2.5 years	0.681	0.634–0.727	0.093	383	205
3.0 years	0.694	0.649–0.739	0.090	331	224
3.5 years	0.689	0.644–0.735	0.091	277	245
4.0 years	0.673	0.626–0.721	0.095	228	263
4.5 years	0.669	0.620–0.719	0.099	183	274
5.0 years	0.675	0.620–0.731	0.111	121	285
5.5 years	0.708	0.647–0.770	0.123	74	290
6.0 years	0.711	0.630–0.792	0.162	36	296

The last two columns explain the flare on the right of the figure. The further out the horizon, the fewer patients remain at risk — from 602 down to 36 — so there are fewer patients available on the control side, while the IPCW weights simultaneously grow larger. Both make the estimate less stable. The widest interval on the whole curve is at 6.0 years, spanning 0.162, several times the range of 0.042 quoted above.

And 6.0 years is exactly where the global maximum sits. A good part of that range is therefore noise at a single point rather than the model genuinely improving late. The first habit to build when reading an AUC(t) curve is to check how many patients remain at that horizon before reading anything into its height.

What the ROC curve at one horizon looks like

AUC(t) is an area, and underneath the area there is a curve. Pinned at 5.0 years:

The horizon goes in the main title rather than only in the caption: a time-dependent ROC with no time attached is the most common way this plot gets misread. The steps are fine because 285 cumulative events support this slice, and the 121 patients still at risk make up the control side.Plotting script figures/scripts/B5-08-time-dependent-auc.R

The plot looks identical to an ordinary ROC curve, but three things about the data underneath it differ, and all three change how it should be read:

	Ordinary ROC	Time-dependent ROC
Who is a case	Known and fixed	Depends on `t`; the same patient can switch sides between horizons
Patients with unknown outcome	Usually excluded, and the count should be stated	Kept, carried by IPCW weights
Is one plot enough	Yes — that is all the information there is	No; every `t` has its own

A second kind of time-dependent ROC, not to be confused with this one.

This page draws the cumulative/dynamic version: cases have accumulated up to $t$ . The other definition is incident/dynamic, where cases are only those failing exactly at $t$ , with controls still defined as event-free at $t$ . The summary quantity corresponding to that version is Harrell’s C, so the two are not old and new versions of one idea — they are two different questions.

Deciding which you need is straightforward. If your question is whether the event happens before $t$ — screening, pre-operative risk counselling, whether to add treatment within some deadline — you want cumulative/dynamic. Only if you are asking when risk is highest, an instantaneous question, do you need incident/dynamic. The overwhelming majority of clinical prediction model papers report the former.

Running it yourself

library(survival)
library(timeROC)
data(cancer, package = "survival")

# Cohort harmonisation, character for character as on the external validation page
rot <- rotterdam
rot$rfs_time  <- pmin(rot$rtime, rot$dtime)
rot$rfs_event <- as.integer(rot$recur == 1 | rot$death == 1)
rot$size_mm   <- c("<=20" = 15, "20-50" = 35, ">50" = 60)[as.character(rot$size)]
rot$grade3    <- as.integer(rot$grade >= 3)

ext <- gbsg
ext$rfs_time  <- ext$rfstime
ext$rfs_event <- ext$status
ext$size_mm   <- ext$size
ext$grade3    <- as.integer(ext$grade >= 3)

v   <- c("age", "size_mm", "nodes", "grade3")
fit <- coxph(Surv(rfs_time, rfs_event) ~ age + size_mm + nodes + grade3, data = rot)

# Risk score as X %*% beta rather than the centred predict(), matching B5-05
lp <- as.numeric(as.matrix(ext[, v]) %*% coef(fit))

# One number
concordance(Surv(ext$rfs_time, ext$rfs_event) ~ lp, reverse = TRUE)

# One curve. weighting = "marginal" is the Kaplan-Meier censoring weight (IPCW)
grid <- seq(1, 6, by = 0.5) * 365.25
roc  <- timeROC(T = ext$rfs_time, delta = ext$rfs_event, marker = lp,
                cause = 1, times = grid, weighting = "marginal", iid = TRUE)

auc <- as.numeric(roc$AUC)
se  <- as.numeric(roc$inference$vect_sd_1)   # only present with iid = TRUE
cbind(years = grid / 365.25, auc = auc,
      lcl = auc - 1.96 * se, ucl = auc + 1.96 * se)

# How many patients remain at each horizon -- read this before reading the curve
sapply(grid, function(t) sum(ext$rfs_time >= t))

# ROC coordinates at one horizon, to plot yourself
k <- which(roc$times == 5 * 365.25)
plot(roc$FP[, k], roc$TP[, k], type = "l", xlim = c(0, 1), ylim = c(0, 1))
abline(0, 1, lty = 2)

Verified with R 4.6.0, survival 3.8.6 and timeROC 0.4.1. The iid = TRUE argument is not optional here: without it timeROC returns no standard errors, and there is no confidence band to draw. cause = 1 refers to the event code, not to the first event type in a competing risks analysis (for AUC(t) under competing risks, delta has to be coded with several event types).

import numpy as np
from sksurv.util import Surv
from sksurv.metrics import cumulative_dynamic_auc, concordance_index_ipcw
import pandas as pd
from lifelines import CoxPHFitter

# The same cohort harmonisation and model fit as B5-05, spelled out so
# this block runs on its own.
RD  = "https://vincentarelbundock.github.io/Rdatasets/csv/"
rot = pd.read_csv(RD + "survival/rotterdam.csv")
ext = pd.read_csv(RD + "survival/gbsg.csv")

rot["rfs_time"]  = rot[["rtime", "dtime"]].min(axis=1)
rot["rfs_event"] = ((rot["recur"] == 1) | (rot["death"] == 1)).astype(int)
rot["size_mm"]   = rot["size"].map({"<=20": 15, "20-50": 35, ">50": 60})
rot["grade3"]    = (rot["grade"] >= 3).astype(int)
ext["rfs_time"]  = ext["rfstime"]
ext["rfs_event"] = ext["status"]
ext["size_mm"]   = ext["size"]
ext["grade3"]    = (ext["grade"] >= 3).astype(int)

v    = ["age", "size_mm", "nodes", "grade3"]
beta = (CoxPHFitter()
        .fit(rot[v + ["rfs_time", "rfs_event"]], "rfs_time", "rfs_event")
        .params_[v].to_numpy())

# scikit-survival takes structured arrays: (event as bool, time)
y_train = Surv.from_arrays(event=rot["rfs_event"].astype(bool), time=rot["rfs_time"])
y_test  = Surv.from_arrays(event=ext["rfs_event"].astype(bool), time=ext["rfs_time"])

v  = ["age", "size_mm", "nodes", "grade3"]
lp = ext[v].to_numpy() @ beta      # beta from any fitted Cox model; higher score = higher risk

times = np.arange(1, 6.5, 0.5) * 365.25   # horizons must lie inside the test cohort's follow-up
auc, mean_auc = cumulative_dynamic_auc(y_train, y_test, lp, times)

# Uno's C: the single-number summary that also uses IPCW
print(concordance_index_ipcw(y_train, y_test, lp))

scikit-survival's cumulative_dynamic_auc does the same job, but it estimates the censoring distribution from the training data while timeROC's marginal weighting uses the validation cohort's own Kaplan-Meier, so the two will not agree to the last digit. Every number on this page comes from the R above.

How this page connects

How Harrell’s C is computed, and why apparent performance is always too high — see internal validation. This page only spreads the same quantity along the time axis; none of the optimism problem goes away. Computing AUC(t) in the development data is just as optimistic.
Where the model, the cohorts and that Harrell’s C come from — see external validation, which also shows that three validation designs cannot be told apart on discrimination at all — what separates them is calibration.
Good discrimination does not mean the probabilities are right — see calibration. AUC(t), like Harrell’s C, only sees the ranking: multiply everyone’s predicted risk by a constant and this page’s curve does not move a digit, while every predicted probability is now wrong. A handsome AUC(t) curve cannot vouch for calibration.
The ordinary ROC and AUC — see ROC curves and AUC. The probability interpretation is the same sentence; what differs is whether case and control status changes with time.
What censoring is, and what independent censoring assumes — see censoring and follow-up time. IPCW puts that assumption to work inside the estimator; it does not remove it.
The clinical setting for the whole chapter — see prediction model studies.

Four questions to ask of an AUC(t) in a paper

Which horizons, and why those? Reporting the horizon whose AUC came out prettiest is the same problem as picking a threshold after the fact (see choosing a cut-off). Horizons should be set by the clinical question.
How many patients remain at risk there? Papers rarely say, yet it determines the width of the interval. Reading a conclusion off the rightmost point without a numbers-at-risk row is the most common misreading of this figure.
How was censoring handled? Look for IPCW or an equivalent weighting. A methods section saying only that ROC analysis was performed usually means censored patients were counted as controls or deleted.
Was calibration reported alongside? An external validation reporting only AUC(t) still covers only one of the three things a validation study has to report.

Common misuses

Misuse	Why it is wrong
Reporting a time-dependent AUC without stating the horizon	AUC(t) is undefined without t; nobody can reproduce or compare it
Counting censored patients as controls	Assumes everyone lost to follow-up stayed well, inflating discrimination
Deleting censored patients instead	The remainder no longer represents the cohort, and dropout tracks prognosis
Choosing the horizon with the highest AUC after seeing them	Same problem as picking a cut-off after the fact; the reported value is no longer unbiased
Drawing conclusions from the far end of the follow-up	Few patients remain at risk there; that stretch is noise, not signal
Reading a flat curve as proof that AUC(t) does not vary with time	Only that no variation was detected; the intervals are not narrow enough to rule out moderate variation
Deciding the curve is unnecessary because one model’s was flat	Flatness is something the analysis reveals, not something to assume beforehand
Cross-checking AUC(t) against Harrell’s C as if they were the same number	They estimate different quantities; disagreement is expected
Cropping the vertical axis to the range the curve occupies	Any flat curve then looks eventful
Skipping calibration because the AUC(t) curve looks good	Discrimination is completely immune to shifting or rescaling the risk scale
Computing AUC(t) in the development data and calling it validation	The optimism is still there; changing the metric does not remove it

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B5-08-time-dependent-auc.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

The whole AUC(t) curve sits above the same model's Harrell's C, with a largest gap of 0.069. Which reading is right?

Show the answer and why

Correct answer: Harrell's C is 0.642, and it compares two patients who both had their event, which AUC(t) never does - the two are not the same quantity

AUC(t) asks one thing: does a patient who had the event before the horizon score higher than one who was still event-free at it. It never compares two patients who have both already had the event, and Harrell's C does - those pairs are close to a coin flip, they are numerous in a long-followed cohort, and they pull the average down. So the gap between 0.642 and the curve is not an arithmetic error, it is two different quantities; Harrell's C is also affected by the censoring distribution, while this page's AUC(t) has already corrected for censoring with IPCW. As for the upper bound 0.675, most of the curve runs above it, but sitting above it does not mean the two are separable: at several grid points the AUC(t) interval still contains Harrell's C, so the sentence to write is that AUC(t) runs systematically higher, not that every point differs distinguishably. 0.686 is the curve's own mean.

Across the grid, AUC(t) has a range of only 0.042 and a standard deviation of 0.014. What does that mean?

Show the answer and why

Correct answer: The widest confidence interval on the same curve is 0.162 on its own, several times the range - the curve's wobble sits inside its own uncertainty

The range is a difference between two point estimates, and every point estimate carries its own uncertainty. The widest interval on this curve is 0.162, several times the range of 0.042, and the intervals at the minimum and the maximum overlap, so what to write is that no change in AUC(t) over time was detected in these data - not that it moves but only a little, which treats an undetected difference as a measured one. The maximum 0.711 does fall at the last time point, but that point has the widest interval on the curve, so using it to claim better long-term ranking is taking noise for signal. Equally, none of this means AUC(t) certainly does not move: no interval here is narrow, and a moderate change cannot be ruled out either. 0.669 is the minimum.

The widest confidence interval on the curve, 0.162, is at the last time point, and so is the global maximum. What should you do first when reading this figure?

Show the answer and why

Correct answer: Check how many people are left at that time point - only 36 remain at the last one - because a good part of that maximum is noise

The later the time, the fewer people remain in the risk set - from 602 down to 36 - so there are fewer controls to pair with, while the IPCW weights grow at the same time, and both make the estimate less stable. That is why the interval at the last time point is 0.162 wide, several times the range of the whole curve, and why the global maximum happens to land there. The cumulative event count of 296 is indeed the largest on the curve, but AUC(t) pairs an event with someone still event-free, and with only 36 left on the control side no number of events can make up for it. The first habit in reading an AUC(t) curve is to check how many people are left at a time point before reading its height.

The AUC(t) grid stops at six years and is not drawn out to eight. Why?

Show the answer and why

Correct answer: The longest follow-up in this external cohort is 7.28 years, so an eight-year horizon is not observable in these patients at all

The grid stops at six years because anything beyond it is extrapolation rather than estimation: the longest follow-up in this external cohort is 7.28 years, so an eight-year horizon is not observable in these patients. 4.50 years is the median follow-up, and a median cannot set the upper limit - it has not even seen half the cohort out, while the limit is a question about whether anyone is left in the data. 0.16 is the interval width at six years; pushing an IPCW estimate past the data does not raise an error, it just hands you a number whose confidence interval is too wide to argue with, which is exactly what this family of pages exists to prevent.

Fix the time point at five years and draw the ROC curve, then look at the area under it. What is that area saying?

Show the answer and why

Correct answer: 0.675 is the probability that a patient with an event by five years scores higher than one still event-free at five years

All three options hold the same number; what is being chosen is what it means. It is a probability: pick at random one patient who had the event before the horizon and one who was still event-free at it, and the chance the model scores the first higher is 0.675. Reading it as accuracy is wrong - accuracy needs a cut-off first, so that every patient is called positive or negative, whereas an ROC curve is what you get by drawing every cut-off at once, and talking about a share classified correctly before any cut-off has been chosen asks a question this curve does not answer. Reading it as calibration is wrong too: the area sees only the ordering, so halve every predicted risk and nothing about who ranks above whom changes, the area does not move at all, and the calibration has collapsed. Ranking and calibration are two questions, one model can rank tolerably and still be badly calibrated, and that is exactly what the calibration page measured on this same model.

Chapters that use this method

Prediction model study

Sources and licences

This page is original writing