AdvancedIndependently reviewed, not yet spot-checked by a human

Internal validation and optimism

Performance measured on the same data the model was fitted to is always flattering, and the gap has a name — optimism. This page carves a 200-person development set out of a large cohort, runs four internal validation methods on it, and checks every answer against the thousands of people left over. It ends with the most expensive mistake in this family — selecting variables on all the data and then cross-validating only the fit.

Why apparent performance is always flattering

Fitting a model is the act of picking the coefficients that make this particular dataset look as good as possible. Turning round afterwards and asking “how well do those coefficients do on this dataset?” is asking the person who wrote the exam to sit it.

The gap has a name — optimism:

\text{optimism} = \text{apparent performance} - \text{performance on new data}

It is not an occasional accident. It is guaranteed in expectation. The only open question is how large it is, and that depends on how much model you fitted relative to how much data you had (see the page on shrinkage).

Internal validation exists to answer exactly that question: using nothing but the data in front of you, estimate the optimism and subtract it off.

How this page is built: a setting where the answer is known

Comparing internal validation methods has a built-in difficulty — normally you have no idea what the right answer is. survival::rotterdam lets us get round that:

Draw 200 people out of 2982 to serve as the development set (115 events, 8 predictors, so 14.4 events per variable)
Every internal validation method is allowed to see only those 200 people
The remaining 2782 are the truth: same cohort, same measurements, never seen by the model

The truth is a C-index of 0.648; apparent performance is 0.691 — a gap of 0.043. That gap is the quantity every method below is trying to recover.

Four methods, all marked against the same answer

Left: what five approaches report on one development set; the red dashed line is the truth measured on the held-out remainder. The dots beside the split-sample bar are 20 different random splits of the same data. Right: what happens when variable selection sits inside versus outside the resampling loop — the subject of the last section.Plotting script figures/scripts/B5-04-internal-validation.R

Approach	C-index reported	SD across reruns	Distance from truth
Apparent (no validation at all)	0.6908	—	+0.0432
One 50/50 split	0.6465	0.0298	-0.0011
10-fold cross-validation	0.6551	—	+0.0075
Repeated 10-fold cross-validation	0.6463	0.0154	-0.0013
Bootstrap optimism correction	0.6623	—	+0.0146

Bootstrap optimism correction, step by step

This is Harrell’s procedure, and the logic behind validate() in the rms package. Its appeal is that nothing has to be held back — all 200 people go into the model, and optimism is estimated separately.

Repeat 200 times:

Draw a sample of the same size with replacement from the original data — one bootstrap sample
Refit the model from scratch on that bootstrap sample (including every selection step and every tuning step)
Measure that model’s performance on the bootstrap sample itself → the optimistic value
Measure the same model on the original data → the more honest value
Subtract → the optimism from this replication

Average those, and subtract the average from apparent performance:

\hat{C}_{\text{corrected}} = \hat{C}_{\text{apparent}} - \overline{\text{optimism}}

On this dataset: apparent 0.691 − optimism 0.0286 = 0.662 (truth 0.648).

Split sample, k-fold, repeated k-fold

Approach	How it works	What goes wrong
Split sample	Divide at random into a training half and a test half, each used once	Both halves shrink: the model is fitted worse than it needed to be, and the test is imprecise. The answer also swings widely with the split
k-fold cross-validation	Divide into k parts; each part serves as the test set once	Everyone gets used, but the partition is still random, so a single run still moves
Repeated k-fold	Run the whole k-fold procedure many times and average	Cost multiplied by the number of repeats — but this is the most direct way to cut the noise
Bootstrap optimism correction	As in the previous section	The model always uses all the data, but every resample has to rerun the entire modelling process

A single 10-fold cross-validation gives 0.655 on this data. Run the whole thing 20 times and the answers fall between 0.608 and 0.668, mean 0.646, standard deviation 0.0154.

The point of this page: selection has to live inside the loop

This is the most expensive mistake in prediction-model research, and the hardest one to catch in your own work.

The workflow looks like this, and a great many published papers follow it exactly:

Run variable selection on the whole development set to obtain “the final model”
Cross-validate that final model
Report the cross-validated C-index and state that internal validation was performed

Step 2 validates the fitting of coefficients and nothing else. It never validates the choosing of variables — and choosing is where the degrees of freedom actually go.

So let us measure it. To the same 200 people, alongside the 8 real predictors, add 10 columns of pure random noise (18 candidates in total). Backward selection by AIC kept 6, of which 2 were noise.

Approach	C-index
Apparent, after selection	0.678
Select once on all the data, then cross-validate the fit	0.655
Selection rerun inside every fold	0.600
The held-out remainder of the cohort (truth)	0.635

Another honest detail: once selection moves inside the loop, the answer (0.600) comes out below the truth (0.635).

That is not an error; the two are answering different questions. Fully-looped cross-validation estimates “given data of this size and this candidate list, and letting the algorithm choose for itself, how good a model do I get on average”, and each fold trains on only nine tenths of the people, which makes it slightly conservative. The truth row estimates “how good is the one specific model that actually got chosen this time” — and this time the algorithm happened to choose a little better than average.

Which should you believe in practice? If you intend to publish this specific model, what you need is external validation. Internal validation tells you roughly what your modelling process is worth; it does not tell you how lucky this particular draw was.

The rule extends well beyond variable selection. Any decision that looked at the outcome has to move inside the loop:

Variable selection (this section)
Choosing a cut-point (see the page on cut-points)
Cross-validating a penalty parameter (see the page on shrinkage)
Using the outcome to decide whether to include an interaction term or a spline
Dropping outliers on the basis of the outcome

What may stay outside are the steps that never saw the outcome: unit conversions, dropping variables that clinical knowledge says will not be available, standardising a predictor using its own distribution.

Run it yourself

library(survival); library(MASS)
data(cancer, package = "survival")

rot <- rotterdam
rot$rfs_time  <- pmin(rot$rtime, rot$dtime)
rot$rfs_event <- as.integer(rot$recur == 1 | rot$death == 1)
rot$size_mm   <- c("<=20" = 15, "20-50" = 35, ">50" = 60)[as.character(rot$size)]
rot$grade3    <- as.integer(rot$grade >= 3)
rot$log_pgr   <- log1p(rot$pgr); rot$log_er <- log1p(rot$er)
v <- c("age", "meno", "size_mm", "grade3", "nodes", "log_pgr", "log_er", "hormon")
f <- as.formula(paste("Surv(rfs_time, rfs_event) ~", paste(v, collapse = " + ")))

set.seed(1)
i   <- sample(nrow(rot), 200)
dev <- rot[i, ]; rest <- rot[-i, ]

fit <- coxph(f, data = dev)
apparent <- summary(fit)$concordance[1]

cidx <- function(lp, d)
  concordance(Surv(d$rfs_time, d$rfs_event) ~ lp, reverse = TRUE)$concordance

# Bootstrap optimism correction, the five steps exactly as listed above
opt <- replicate(200, {
  bd <- dev[sample(nrow(dev), replace = TRUE), ]
  bf <- coxph(f, data = bd)                                  # 2. refit from scratch
  c_boot <- summary(bf)$concordance[1]                       # 3. on the bootstrap sample
  c_orig <- cidx(as.matrix(dev[, v]) %*% coef(bf), dev)      # 4. on the original data
  c_boot - c_orig                                            # 5. subtract
})
apparent - mean(opt)

# Truth: the people the model never saw
cidx(as.matrix(rest[, v]) %*% coef(fit), rest)

# ⚠️ Selection belongs INSIDE the loop. This loop reruns stepAIC in every fold
folds <- sample(rep(1:10, length.out = nrow(dev)))
lp <- rep(NA, nrow(dev))
for (k in 1:10) {
  tr <- dev[folds != k, ]
  ff <- f; environment(ff) <- environment()
  st <- stepAIC(coxph(ff, data = tr), direction = "backward", trace = 0)
  vk <- names(coef(st))
  lp[folds == k] <- as.matrix(dev[folds == k, vk, drop = FALSE]) %*% coef(st)
}
cidx(lp, dev)

Verified with R 4.6.0 + survival 3.8.6 + MASS 7.3.65. In practice, validate(fit, method = "boot", B = 200) from the rms package does bootstrap optimism correction in one line — but it also only validates the model you wrote in the formula, so any selection step still has to be wrapped in yourself. The five steps are spelled out here so that they are visible.

import numpy as np, pandas as pd
from sklearn.model_selection import RepeatedKFold
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest, f_classif
from lifelines import CoxPHFitter
from lifelines.utils import concordance_index

RD = "https://vincentarelbundock.github.io/Rdatasets/csv/"
d = pd.read_csv(RD + "survival/rotterdam.csv")

# The same derived columns as the R code above. rotterdam ships none of
# them, and without this every line below is a KeyError.
d["rfs_time"]  = d[["rtime", "dtime"]].min(axis=1)
d["rfs_event"] = ((d["recur"] == 1) | (d["death"] == 1)).astype(int)
d["size_mm"]   = d["size"].map({"<=20": 15, "20-50": 35, ">50": 60})
d["grade3"]    = (d["grade"] >= 3).astype(int)
d["log_pgr"]   = np.log1p(d["pgr"]); d["log_er"] = np.log1p(d["er"])
v = ["age", "meno", "size_mm", "grade3", "nodes", "log_pgr", "log_er", "hormon"]

rng = np.random.default_rng(1)
i   = rng.choice(len(d), 200, replace=False)
dev, rest = d.iloc[i], d.drop(d.index[i])

# ⚠️ Everything depends on where the selection step lives. Inside the Pipeline it
#    gets rerun by cross_val; running SelectKBest once on all of dev and then
#    feeding the chosen columns to cross_val_score is the mistake this page ends on.
sel = Pipeline([("select", SelectKBest(f_classif, k=4))])

lp = np.full(len(dev), np.nan)
for tr_i, te_i in RepeatedKFold(n_splits=10, n_repeats=1, random_state=0).split(dev):
    tr, te = dev.iloc[tr_i], dev.iloc[te_i]
    sel.fit(tr[v], tr["rfs_event"])                 # selection sees the training folds only
    cols = [c for c, keep in zip(v, sel.named_steps["select"].get_support()) if keep]
    m = CoxPHFitter().fit(tr[cols + ["rfs_time", "rfs_event"]], "rfs_time", "rfs_event")
    lp[te_i] = te[cols].to_numpy() @ m.params_[cols].to_numpy()

print(concordance_index(dev["rfs_time"], -lp, dev["rfs_event"]))

scikit-learn's cross_val_score only validates what is inside the pipeline, so the correct approach is to make the selection step a stage of the pipeline. Leaving it outside the pipeline is exactly the mistake described in the last section of this page.

Five questions to ask of any paper

Which internal validation method was used? “The model was validated” without naming a method says nothing.
How many times was the cross-validation repeated? The second decimal place of a single k-fold run is not stable.
Was variable selection inside the resampling? The Methods section will rarely say so directly. Look at the order instead: if the paper reports “the final model included X variables” and only then says “this model was cross-validated”, the order is the answer.
Internal validation passed — and then what? Internal validation deals with optimism. It does nothing at all about differences between populations. That is the job of external validation.
Discrimination at which time point? The C-index compresses every time point into one number, and clinical decisions are made at particular ones. To see how it moves across time, see time-dependent AUC.

Common misuses

Misuse	Why it is wrong
Reporting the apparent C-index as the model’s performance	It is high in expectation, always
Selecting variables on all the data, then cross-validating that model	Every fold’s test set took part in the selection; the optimism is still there
Using one split sample as the internal validation	The answer swings widely with the split, and both training and testing are shrunk
Reporting the second decimal place from a single k-fold run	The partition is random; rerunning gives a different answer
Reusing the variables chosen on the original data inside the bootstrap	That is not the same modelling process, and optimism comes out too small
Calling a split sample “external validation”	It is internal validation; external requires different people
Declaring a model usable because internal validation passed	Internal validation is blind to population differences and calibration drift
Cross-validating discrimination and never reporting calibration	The calibration slope is the most direct internal-validation signal of overfitting
Excluding outliers on the basis of the outcome before resampling begins	That decision used the outcome too, so it belongs inside the loop

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B5-04-internal-validation.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

On the same two-hundred-person development set the apparent C-index is 0.691, while the two thousand and more people the model never saw give 0.648. What is that gap called, and where does it come from?

Show the answer and why

Correct answer: 0.691 was measured on the very people whose data chose the coefficients, which is letting the author of the exam sit it - the gap is optimism

0.691 minus 0.648 is the quantity this page sets out to estimate. The hold-out sample is carved out of the same cohort, with the same measurements and the same inclusion rules, which switches off the population-difference explanation and leaves only the fact that the coefficients were chosen to flatter this particular batch of people. 0.029 is the optimism the bootstrap correction estimated, and it not matching the true gap is unremarkable: it is an estimate with its own error, and the whole design of this page exists to see how well the various methods estimate it. The thing to remember is that optimism is not occasional - in expectation it always happens, and the only question is how large.

A single split-sample run gives a C-index point estimate almost exactly on the truth, apparently the most accurate of the five approaches. Does that make split-sample a good method?

Show the answer and why

Correct answer: No. Rerunning the same data with different random splits gives answers with a standard deviation of 0.0298, so hitting the truth this once was luck; whether a method is good depends on how much it moves when it is rerun

0.0298 is the standard deviation across twenty different splits, against 0.0154 for repeated cross-validation, about half. How close a method landed this once and how dependable it is are two different things - change the random seed and the split-sample answer wanders across a wide span, a range of 0.1379. The remedy in the second option sounds reasonable, but splitting several times and averaging is what cross-validation already does, and split-sample trains on half the people and tests on the other half each time, whereas cross-validation lets everybody serve as a test case once, so the cost is not higher. The third option conflates unbiased with precise: an estimate can be right on average and wide of the mark every single time. The real cost of splitting is that it makes the model worse and the evaluation blurrier at the same time - and it is the most intuitive and most commonly chosen option.

A paper reports a C-index of 0.655 after 10-fold cross-validation, without saying how many times it was repeated. Which number on this page should make you doubt that second decimal?

Show the answer and why

Correct answer: Rerunning the same data with only the folds reshuffled sends single-run answers as high as 0.668 and as low as 0.608, a swing wider than the model differences most people want to compare

0.668 and 0.608 are the two ends produced by the same data and the same code with nothing but the random folds reshuffled. That span is wider than the model differences most papers want to claim, so an unrepeated 0.655 does not point at a stable quantity. 0.646 is the mean after repetition, and it is lower than this particular single run, but always too high does not follow - a single run can land anywhere in the span. 0.015 is indeed the standard deviation across repeats, yet reading it as steady enough misjudges the scale: it is already enough to produce a span from 0.608 to 0.668, and that span is wider than the differences most papers argue over.

The second step of bootstrap optimism correction says to refit the model from scratch in the bootstrap sample. What happens if variable selection is done once on the original data and then fixed?

Show the answer and why

Correct answer: Selecting once on all the data and then cross-validating only the fit gives 0.655; the selection never entered the loop, so every fold's test set took part in it and the optimism is still inside

0.655 and 0.600 are far apart: in the first, every fold's test set took part in the one selection, so those test sets are no longer data the model has not seen; in the second, the selection is rerun inside each fold, which finally charges the cost of selecting. The nine tenths of people in each training fold mentioned by the second option is real, and it does make the in-loop version slightly conservative, but it cannot account for a gap this size - ten pure random variables were added to the candidate list and backward AIC kept two of them, and they were kept because they happened to correlate a little in these two hundred people. 0.678 is apparent performance after selection, measured on the same people, the most optimistic of the three rather than the underestimated one. And the rule extends past variable selection: choosing a cut-off, cross-validating a penalty, excluding outliers on the basis of the outcome - any decision that has seen the outcome must go inside the loop.

With selection inside every fold, cross-validation returns 0.600, below the hold-out truth of 0.635. Is that an error?

Show the answer and why

Correct answer: No. 0.635 estimates how good the particular model actually selected is, whereas the fully in-loop cross-validation estimates how good a model the algorithm produces on average - two different questions

0.635 is how the model selected this time performs in the two thousand and more people who were not drawn; 0.600 is what the same modelling procedure is worth on average, and each fold trains on nine tenths of the people, so it is slightly conservative as well. Two different questions, and different answers do not constitute an error. The double penalty in the first option does not exist - each fold reruns the selection using only that fold's training data, and no variable is charged twice. The 0.655 that the third option picks is precisely the mistake this page is demonstrating: every one of its test sets took part in the selection, and the number is higher not because it is balanced but because it has not paid for the selection; it also sits above the held-out truth of 0.635, so it is not a middle value in any sense. Which to trust in practice? If you intend to publish this particular model, what you need is external validation - internal validation can tell you roughly what your modelling procedure is worth, not how lucky this draw was.

In the two-hundred-person development set, ten pure random variables are added alongside the eight real ones, and backward AIC keeps six out of the eighteen candidates. Which statement is right?

Show the answer and why

Correct answer: 2 of the six kept are pure noise; they were selected because they happened to correlate a little in these two hundred people

Two of the six variables kept are noise - they happened to correlate a little in these two hundred people, and that is all. That real variables outnumber noise ones does not show that the procedure broadly tells true from false: a noise variable's true contribution is zero, so any one of them reaching the final model means the procedure is already learning this batch's noise, and unless the validation reruns the selection too, that noise gets counted as the model's own ability. More candidates than survivors is not conservatism either: screening consumes degrees of freedom, so the more you cut the more you have spent, and those discarded candidates still count towards the number of parameters in a sample-size calculation.

Chapters that use this method

Prediction model study

Watch next

簡中Bessie Hiram· 13 minIn Simplified Chinese. Walks through the mechanics of internal validation end to end, and is the easiest place to pick up the Chinese equivalents of the terms on this page.

繁中Colon & Code· 10 minIn Traditional Chinese. Its section on data leakage is the same mechanism as the last section here — any step that touches all the data has to move inside the resampling loop.

Sources and licences

This page is original writing