Bland-Altman analysis and the limits of agreement

What this page is here to answer

Results sections often read like this: the new method correlated strongly with the reference method, demonstrating good agreement. The first half is a statistic; the second half is a wish, and no inference connects them. The data on this page will make that correlation coefficient look entirely defensible, and then show you the same people’s differences.

Agreement asks something very concrete: if I switch this patient from method A to method B, how different will the number be, and is that difference big enough to change what I do. A correlation coefficient cannot answer it, because it measures how similarly the two methods rank patients, not how far apart two readings on one patient are.

After this page, four questions should be available to you whenever you meet a method-comparison paper:

1. How large is the bias, and in which direction—does the new method read higher or lower than the reference, and by how much.
2. How wide are the limits of agreement—converted into clinical units: how many clinically material differences wide is that.
3. How uncertain are the limits themselves—the limits are estimates with their own confidence intervals, and mishandled replicates usually break this, not the limits.
4. Do the differences change with the size of the measurement—if they do, one fixed pair of limits cannot serve the whole range.

The data: Bland and Altman’s own blood pressure set

This page uses MethComp::sbp: 765 readings from 85 subjects, each measured 3 times by each of 3 methods, in mmHg. It is the teaching dataset Bland and Altman used themselves, and the three methods are:

This page compares human observer J against semi-automatic machine S, and the difference is always defined as S − J: a positive value means the machine read higher than the human. That gives 255 replicate-level pairs, or 85 pairs of per-subject means.

The seduction: the scatter plot first, then the same people’s differences

The left panel gives r = 0.817 (95% CI 0.732 to 0.878, p < 0.0001), so r² = 0.668. Computed on all 255 replicate-level pairs instead, it is 0.796.

No clinical paper would refuse to call that a strong correlation—the p value is not remotely in question and the lower confidence bound still sits above 0.73. And it contributes nothing to the question of whether this machine can replace manual measurement.

Why the x axis is the mean of the two methods

The x axis of the right panel is the mean of the two methods, not the reference method. This is not a matter of taste: the difference is negatively correlated with the reference method by construction (whenever the reference reading happens to be high, the difference is pulled down), so plotting difference against reference manufactures a downward slope and makes you see proportional bias that is not there. The mean of the two is the best available approximation to the truth, and it flattens that artefact out.

The three lines on the difference plot come from: the mean difference, that is a bias of +15.62 mmHg (95% CI +11.54 to +19.70); and the bias plus and minus 1.96 times the SD of the differences, 20.95 mmHg, giving -25.4 and +56.7. That is a span of 82.1 mmHg.

Which means: swap this machine in, and an individual patient’s reading may come out 25 mmHg below the manual measurement or 57 mmHg above it—and that is still only the range holding 95% of people.

How wide is too wide: three ways to translate the limits

The limits are a statistic; whether they are good enough is not a statistical question. Three routes turn them into something judgeable, and this dataset supports all three.

First, convert to a clinically material difference. For systolic pressure, 10 mmHg is the usual order of magnitude that changes management. The span of the limits equals 8.2 such differences. The raw distribution of the replicate pairs says the same thing: 62.7% of pairs differ by more than 10 mmHg, 23.5% by more than twice that, and the largest single pair differs by 109 mmHg.

Second, ask whether it flips decisions. Take 140 mmHg as the line. Observer J puts 24 people above it, machine S puts 36. Compared person by person, 14 subjects (16.5%) cross that line purely because of which method measured them—13 of them because the machine called them high and the human did not, 1 in the other direction. How a cut-point is chosen, and how to read misclassification near it, is on choosing a cut-point.

Third, find an external regulatory anchor. ISO 81060-2 criterion 1 requires a non-invasive blood pressure device to differ from the reference by no more than 5 mmHg on average, with an SD of differences no greater than 8 mmHg. This dataset observes a mean difference of 15.62 and an SD of 20.95 mmHg, failing both criteria by a wide margin.

The control condition: same people, same occasion, two human observers

The dataset carries a third method: a second human observer, R. Pair J with R and nothing changes except who the other party is—same sample, same occasion, same arithmetic:

Absolute-agreement ICC between the two human observers is 0.997, and their limits span 8.9 mmHg—roughly 11% of the human-versus-machine span, or put the other way, the machine pair is about 9 times wider.

That control rules out a whole class of excuses. The limits are not wide because blood pressure fluctuates, nor because subjects were uncooperative, nor because 85 people is too few, nor because Bland-Altman is harsh on everyone. Same people, same afternoon, same formulas: two humans reading the same sphygmomanometer differ with an SD of only 2.26 mmHg. What is wide is this machine.

Replicate measurements: the precision collapses, the limits do not

Every subject was measured 3 times, so the question of how many observations there are has three answers, and the three answers give three different sets of limits:

All values in mmHg. C is what this page uses. Put the three rows side by side and the conclusion runs against most people’s intuition:

Treating all 255 pairs as independent barely moves the limits at all. A span of 79.84 against the correct 82.12 is 2.77% narrow, or 2.28 mmHg in absolute terms. Drawn on a plot, the two pairs of dashed lines are not distinguishable by eye.

What collapses is the precision of the limits. The 95% confidence interval of each limit shrinks from the correct 6.60 mmHg either side to 4.30 mmHg either side, 34.9% narrower. The reason is direct: the effective sample size is inflated 3-fold—there are only 85 independent people in the data, but the standard errors are computed as though there were 255 independent observations.

The approach that really is badly too narrow is the third one. Take each subject’s mean of 3 readings, difference those, and apply no correction: the span is 74.22 mmHg, 9.62% narrow, 7.90 mmHg missing. This is what section 5.2 of Bland and Altman (1999) was warning about: averaging shrinks each method’s own measurement error, so the limits you report describe a mean of three against a mean of three, while a patient in clinic is measured once.

Why the limits did not collapse too

Because J and S read the same subject at the same time. Whatever the true pressure was at that instant is shared by both methods, so it subtracts out within each occasion’s difference. The variance of the differences therefore contains almost nothing of where this person’s pressure happened to be, and repeating three times does not conjure any of it back—so treating the replicates as independent estimates an SD close to the right answer.

The correct procedure splits the variance of the difference and rebuilds it. With $m$ replicates per subject, $\bar d$ the per-subject mean difference, and $s_w^2$ each method’s own within-subject variance:

For this dataset: $m$ is 3, so the correction factor is 0.667; the SD of the per-subject mean differences is 18.93 mmHg; observer J’s own within-subject SD across three readings is 6.12 mmHg and machine S’s is 9.12 mmHg (on 170 degrees of freedom each). Rebuilt, that is 20.95 mmHg, which is row C above.

Notice the relationship between those two within-subject SDs in passing: the machine repeating itself three times varies more than the human does (9.12 against 6.12 mmHg). A device sold on removing observer error is not, on its own repeatability, steadier than the observer.

Proportional bias: checked, and none detected

A single pair of limits carries an implicit premise: the size of the difference is unrelated to the size of the measurement. If people with higher pressures differ more, that premise fails, and one pair of limits stretched across the whole range is too wide at the low end and too narrow at the high end.

The check is a linear regression of the difference on the mean, read off the slope. On per-subject means the slope here is +0.0356 mmHg/mmHg (95% CI −0.1028 to +0.1739, p = 0.610); on all 255 replicate pairs it is +0.0514 (p = 0.226). The point estimate says that a 10 mmHg rise in the measurement shifts the difference by +0.36 mmHg—negligible next to 10 mmHg.

So the finding on this page is: no proportional bias was detected in these data, and the limits are reported as one fixed pair.

The right panel demonstrates the other common move: take logs of both methods and repeat. That is the standard handling when differences scale with the magnitude, and its advantage is that the limits come back as ratios rather than absolute amounts. Here the log-scale slope is −0.0640 (p = 0.338), again with no trend detected; translated back, the machine reads 12.6% higher on average, with limits at ratios of 0.851 to 1.491, that is -14.9% to +49.1%.

Running it yourself

data(sbp, package = "MethComp")

# The data are long: meth (J/R/S), item (subject), repl (which reading), y (mmHg).
# Assert one row per (meth, item, repl) first — otherwise the reshape below
# silently keeps the first match instead of erroring. This line is load-bearing.
stopifnot(!anyDuplicated(sbp[c("meth", "item", "repl")]))

w <- reshape(sbp[sbp$meth %in% c("J", "S"), ],
             direction = "wide", idvar = c("item", "repl"),
             timevar = "meth")
w$diff <- w$y.S - w$y.J          # difference is machine minus human
w$avg  <- (w$y.S + w$y.J) / 2

# Per-subject means (85 subjects)
m <- aggregate(cbind(diff, avg, y.J, y.S) ~ item, data = w, FUN = mean)

# 1. The seduction: the correlation. High, and unrelated to agreement.
cor.test(m$y.J, m$y.S)

# 2. Limits of agreement with the Bland & Altman (1999) replicate correction.
#    Taking sd(w$diff) over the 255 pairs treats 85 people as 255 independent
#    observations: the limits come out only a few percent narrow, but the
#    confidence interval of each limit comes out a third too narrow.
mm <- 3
within_var <- function(v) {
  # each method's own within-subject variance, pooled by degrees of freedom
  a <- aggregate(v ~ item, data = w, FUN = function(x) c(var(x), length(x) - 1))
  sum(a$v[, 1] * a$v[, 2]) / sum(a$v[, 2])
}
n      <- nrow(m)
df_w   <- n * (mm - 1)
k      <- 1 - 1 / mm
vJ     <- within_var(w$y.J)
vS     <- within_var(w$y.S)
v_dbar <- var(m$diff)
s2     <- v_dbar + k * (vJ + vS)
bias   <- mean(m$diff)
z      <- qnorm(0.975)
loa    <- bias + c(-1, 1) * z * sqrt(s2)

# 3. Confidence intervals for the limits. s2 is a sum of three independent
#    components, so its variance is the sum of three Var(s2) = 2 s2^2 / df
#    terms, and the limit's variance is then
#    Var(LoA) = Var(bias) + z^2 * Var(s), with Var(s) = Var(s2) / (4 s2).
#    The textbook sqrt(s2 * (1/n + z^2 / (2(n-1)))) assumes every observation
#    is independent, which is exactly what is false here: it comes out about a
#    millimetre of mercury wider than the table above. The denominator is
#    subjects, not pairs.
var_s2 <- 2 * v_dbar^2 / (n - 1) + k^2 * 2 * vJ^2 / df_w + k^2 * 2 * vS^2 / df_w
se     <- sqrt(v_dbar / n + z^2 * var_s2 / (4 * s2))
tq     <- qt(0.975, n - 1)
rbind(lower = loa[1] + c(-1, 1) * tq * se,
      upper = loa[2] + c(-1, 1) * tq * se)

# 4. Proportional bias: regress difference on mean. The x variable must be the
#    mean of the two — plotting against the reference manufactures a slope.
summary(lm(diff ~ avg, data = m))

# 5. The log-scale version: the limits come back as ratios, not mmHg. Fit it on
#    the per-subject means, as in step 4 — run over the 255 single pairs it
#    returns a different slope, which is not the one the table reports.
lm(log(y.S) - log(y.J) ~ I((log(y.S) + log(y.J)) / 2), data = m)

# Exported for the Python block below
write.csv(sbp, "sbp.csv", row.names = FALSE)

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
from scipy import stats

# MethComp is not in the Rdatasets index, so get_rdataset("sbp", "MethComp")
# returns a 404. The R block above already writes it out.
sbp = pd.read_csv("sbp.csv")
assert not sbp.duplicated(["meth", "item", "repl"]).any()

w = (sbp[sbp.meth.isin(["J", "S"])]
     .pivot(index=["item", "repl"], columns="meth", values="y")
     .reset_index())
w["diff"] = w["S"] - w["J"]
w["avg"] = (w["S"] + w["J"]) / 2
m = w.groupby("item")[["diff", "avg", "J", "S"]].mean()

print(stats.pearsonr(m["J"], m["S"]))

mm = 3
def within_var(col):
    g = w.groupby("item")[col]
    return (g.var(ddof=1) * (g.count() - 1)).sum() / (g.count() - 1).sum()

n = len(m)
df_w = n * (mm - 1)
k = 1 - 1 / mm
vJ, vS = within_var("J"), within_var("S")
v_dbar = m["diff"].var(ddof=1)
s2 = v_dbar + k * (vJ + vS)
bias = m["diff"].mean()
z = stats.norm.ppf(0.975)
loa = bias + np.array([-1, 1]) * z * np.sqrt(s2)

var_s2 = 2 * v_dbar**2 / (n - 1) + k**2 * 2 * vJ**2 / df_w + k**2 * 2 * vS**2 / df_w
se = np.sqrt(v_dbar / n + z**2 * var_s2 / (4 * s2))
tq = stats.t.ppf(0.975, n - 1)
print(loa, loa[:, None] + np.array([-1, 1]) * tq * se)

print(smf.ols("diff ~ avg", data=m).fit().summary())

lg = m.assign(ld=np.log(m["S"]) - np.log(m["J"]),
              la=(np.log(m["S"]) + np.log(m["J"])) / 2)
print(smf.ols("ld ~ la", data=lg).fit().params)

Reading the report

A method-comparison paper usually gives you one Bland-Altman plot and a paragraph. Six places are worth checking one by one:

1. Is the direction of the difference stated. A minus B and B minus A have opposite signs for the bias. The y axis label should say who is subtracted from whom; a plot labelled only difference cannot be quoted.
2. Do the limits come with confidence intervals. The limits are estimates. The upper limit here is +56.7 and its 95% CI reaches +63.3—if the authors want to argue that the limits fall within an acceptable range, the outer edge of that interval is what matters, not the point estimate.
3. How were the replicates handled. Replicates present and n reported as the number of pairs (here that would be 255 rather than 85) means the confidence intervals of the limits are too narrow; per-subject means with no correction means the limits themselves are too narrow. A Methods section that says nothing leaves both possible.
4. Was proportional bias checked, and how. There should be a regression of difference on the mean of the two methods. If it was regressed on the reference method, that slope cannot be trusted.
5. Is there a pre-specified acceptable difference. Without that number, acceptable limits is just the authors’ impression. The best version cites an external standard, as with ISO 81060-2 criterion 1 here.
6. Does the sample cover enough of the range. Limits extrapolate only as far as the data go. A cohort clustered in the normal range says nothing about how the machine behaves above 140 mmHg.

Common misuses

How this page relates to the others

• The reliability side of the same data—see the intraclass correlation coefficient. That page uses the J against R pair this one deliberately avoids, and together they show that agreement and reliability are two different questions asked of one set of numbers.
• Agreement for categorical outcomes—see kappa and weighted kappa. When the outcome is a category rather than a quantity, a difference has no meaning and the tooling changes.
• The downstream consequences of measurement error—see measurement error. The spread of differences measured here is what that page discusses, seen before it enters a regression.
• That regression of difference on mean—see linear regression. The proportional-bias check is just that model applied to a particular pair of variables.
• Dependence between repeated measurements—see clustered data. The correction formula here is a special case of that page’s variance decomposition.
• Cut-points and flipped decisions—see choosing a cut-point. The second translation route above uses that page’s framing.
• Clinical context—see the design chapter diagnostic accuracy studies.

Regenerating every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B9-03-bland-altman.R

Report a problem on this page