Intraclass correlation and reliability

This ICC and the one in the clustered-data chapter are the same ICC

This site has already taught the ICC once. Clustered and repeated measurements defines it as the correlation between any two observations on the same patient, and uses it to build a design effect — the penalty for treating repeated measurements as independent.

The ICC on this page is the same variance split, written as the same ratio: a between-unit variance over a total variance. What differs is the question.

• The clustered-data chapter asks how alike two observations inside one cluster are, and a large answer is bad news — it is a nuisance to be corrected for.
• This page asks how alike two measurements of the same subject are, and a large answer is the point — it is the result being reported.

One number, a nuisance in one place and the deliverable in the other. The collision is historical, not a misreading on your part.

The unnamed reliability on the measurement-error page is this number

Measurement error defines a quantity while explaining regression dilution: reliability, the variance of the true value over the variance of the observed value. Under classical error the regression coefficient is multiplied by that constant. That page uses the quantity without saying where it comes from.

It comes from here. The variance of the true value is the between-subject variance — people really do differ — and the variance of the observed value adds the measurement variance on top. The reliability of a single measurement is the single-measurement ICC.

There is a practical consequence. Regression calibration needs a repeat-measurement substudy to estimate reliability, and the statistic that substudy has to compute is the one on this page. Pick the wrong ICC form there and the correction factor is wrong — while the corrected coefficient goes on looking perfectly respectable.

Six forms, not eight

Three axes decide which ICC you computed.

1. Model: the rater effect is not in the model (one-way) or in the model (two-way).
2. Type: systematic differences between raters are excluded from the denominator (consistency) or kept in it (absolute agreement).
3. Unit: the reliability of one rating, or of the mean of k ratings.

Three axes should give eight combinations. Only six exist, and the next section explains where the other two went.

The lowest is 0.175, the highest 0.453, a gap of 0.278 and a ratio of 2.59. All six are legal, and all six come from the same untouched dataset.

Why six and not eight: the one-way branch never looks at type

The label the package prints collides with a different model

Here are the three naming systems side by side. Each row is one and the same form:

So ICC = 0.85 on its own is not a result

When a paper gives one ICC and no model, type or unit, you cannot tell which cell it came from. On this dataset the cells run from 0.175 to 0.453; on data with larger rater differences the span only widens.

The three axes are not academic taste either. Each maps onto a different clinical question.

• Absolute agreement or consistency: will the measurement be compared against a fixed cut-point, the way a systolic pressure is compared against a hypertension threshold? Then it has to be absolute agreement — a rater who reads systematically two units high moves every patient across the line. If you only need the ranking, consistency is enough.
• Single or average: will one person read the scan in practice, or will three read it and agree a consensus? Reporting the average form when the clinic uses a single reading inflates the reliability.
• One-way or two-way: is every subject assessed by the same set of raters (two-way), or by whoever was on duty that day (one-way)?

What each axis does to the denominator

All three numerators are called the between-subject variance, but the one-way one is not the same number as the two-way one: a one-way model has no rater term, so rater differences are absorbed into the within-subject error and its between-subject variance comes out 0.056 smaller. The two two-way forms genuinely share a numerator; there, only the denominator moves.

Start with the variances themselves. The two-way split gives between-subject 0.399, between-rater 0.168 and residual 1.448, which is 19.8%, 8.4% and 71.8% of the total.

The three single-measurement denominators are assembled like this:

Notice that the first row’s numerator, 0.343, is smaller than the 0.399 of the other two, by 0.056. How does the between-subject variance shrink on identical data? Because a one-way layout cannot identify the rater effect, that variance is washed into the error term, and the numerator is pulled down with it. A one-way ICC is not merely a laxer denominator; it is a model that cannot estimate a rater effect.

The average forms do something else: every term in the denominator except the numerator itself is divided by k. More raters, a steadier mean, a higher reliability — by construction, not because the measurement got better.

Do the raters actually differ on this dataset

They do, and the difference reaches statistical significance, if narrowly. The F test of the rater main effect gives F(2, 38) = 3.33 with p = 0.0467. The three rater means are 3.15, 3.15, 2.30 — the first two are identical and the third sits systematically almost a whole point lower.

That point is exactly what a consistency form discards and an absolute-agreement form keeps.

An honest limitation: on this dataset the other axis is doing the work

To make the second axis visible, the script performs a deliberate synthetic manipulation: it adds a constant to every score given by rater 1 and recomputes. That section of the stats file is labelled SYNTHETIC MANIPULATION, not a finding, which is there to stop anyone reading the table as a finding about anxiety ratings.

Three things happen at once, and every one of them is algebraically forced.

1. The consistency column does not move at all. Not nearly — exactly. Consistency forms are invariant to shifting a whole rater by a constant, and the script asserts that with stopifnot() rather than eyeballing it.
2. The agreement column falls steadily, from 0.198 to 0.068. It counts the offset as disagreement, which is what the offset is.
3. The one-way column falls fastest and turns negative once the constant reaches 2. A one-way layout has nowhere to put a systematic rater difference except the error term, and that term grows past the between-subject variance.

A high ICC does not mean a precise measurement

A different dataset. data(sbp, package = "MethComp") holds systolic pressures for 85 people, each measured 3 times by each of 3 methods. This page uses the two human observers (J and R), because that is an inter-observer reliability question with no device in it; the human-versus-machine pair belongs to Bland-Altman agreement.

Computed on each person’s mean of three readings, the two-way absolute-agreement single-measure ICC is 0.9991. That is about as good as a number gets.

The same people and the same readings, described the other way round: the differences between the two observers have a mean of 0.09 mmHg and limits of agreement running from -4.34 to 4.52 mmHg, a width of 8.86 mmHg.

So these two observers, measuring the same person on the same occasion, can differ by four or five mmHg, and the ICC for that comparison is 0.9973.

That claim is testable, so the script tests it. It trims the 85 people symmetrically around their median mean pressure, into progressively narrower bands. Neither observer changes and not one reading changes. All that changes is how unlike each other the measured people are.

As the spread of the people falls from 30.63 to 4.09 mmHg, the ICC falls from 0.9991 to 0.9609. The limits of agreement go from 5.24 to 4.60 mmHg — a change of 0.64 mmHg, and not even monotone: the width narrows down the first three rows and then turns back up on the narrowest band.

The sentence worth carrying away: an ICC is a property of the measurement and of the population it was estimated in; the limits of agreement are a property of the measurement alone. That is why a published ICC cannot be transported to your cohort and published limits of agreement can — and it is the division of labour between this page and Bland-Altman, which answers the same question in mmHg instead of as a ratio.

When to use kappa and when to use the ICC

The line between this page and kappa and rater agreement is one thing only: whether the outcome is continuous or categorical.

• Categorical (present or absent, a grade, benign or malignant) goes to the kappa family, which asks how much agreement is left after subtracting what raters would hit by chance.
• Continuous (a score, a concentration, a length, a blood pressure) goes to the ICC, which asks how much of the total variance is genuinely due to subjects differing.

They are not two options to choose between; the data type chooses. One boundary case is worth remembering. Ordered categories — a five-level pain score, say — support both, and a weighted kappa with quadratic weights and an absolute-agreement ICC give closely related answers. There the choice is about how you intend to describe the result, not about which number looks better.

The division between the ICC and Bland-Altman is not about data type at all. It is about whether you want a ratio or a distance: a reviewer asking whether the two observers agree wants the ICC, a clinician asking whether one observer’s reading can be substituted for the other’s wants limits in mmHg. Reporting both is normal, not redundant.

Run it yourself

library(irr)
data(anxiety, package = "irr")   # 20 subjects x 3 raters, fully crossed

# The six legal forms. Note that `type` does nothing in the oneway branch:
# icc(x, "oneway", "agreement", ...) returns the consistency value, silently.
icc(anxiety, model = "oneway", type = "consistency", unit = "single")
icc(anxiety, model = "twoway", type = "consistency", unit = "single")
icc(anxiety, model = "twoway", type = "agreement",   unit = "single")
icc(anxiety, model = "oneway", type = "consistency", unit = "average")
icc(anxiety, model = "twoway", type = "consistency", unit = "average")
icc(anxiety, model = "twoway", type = "agreement",   unit = "average")

# Do not copy $icc.name into a manuscript. For oneway + average the label is
# paste("ICC(", number_of_raters, ")"), so three raters print ICC(3) -- which
# reads as Shrout-Fleiss ICC(3,1), a two-way consistency form.
icc(anxiety, "oneway", "consistency", "average")$icc.name

# The same thing built by hand. Across the two two-way forms the numerator
# really is one number and only the denominator moves; the oneway call above is
# not one of them -- a one-way layout re-estimates the numerator, and lower.
A  <- as.matrix(anxiety); ns <- nrow(A); nr <- ncol(A)
long <- data.frame(y = as.vector(A),
                   subj = factor(rep(seq_len(ns), nr)),
                   rater = factor(rep(seq_len(nr), each = ns)))
a  <- anova(lm(y ~ subj + rater, data = long))
MSs <- a["subj", "Mean Sq"]; MSr <- a["rater", "Mean Sq"]; MSe <- a["Residuals", "Mean Sq"]

vs <- (MSs - MSe) / nr                      # between subjects
vr <- (MSr - MSe) / ns                      # between raters
ve <- MSe                                   # residual
c(consistency = vs / (vs + ve),             # rater variance excluded
  agreement   = vs / (vs + vr + ve))        # rater variance kept

# The same machinery as the clustered-data page: on a balanced fully crossed
# design REML reproduces the moment estimates above exactly.
lme4::VarCorr(lme4::lmer(y ~ 1 + (1 | subj) + (1 | rater), data = long))

# Second dataset: the ICC is a ratio, the limits of agreement are a distance.
data(sbp, package = "MethComp")
sb <- sbp[sbp$meth %in% c("J", "R"), ]
mJ <- tapply(sb$y[sb$meth == "J"], sb$item[sb$meth == "J"], mean)
mR <- tapply(sb$y[sb$meth == "R"], sb$item[sb$meth == "R"], mean)
M  <- cbind(J = as.numeric(mJ), R = as.numeric(mR))
icc(M, "twoway", "agreement", "single")$value
d <- M[, "J"] - M[, "R"]; mean(d) + c(-1.96, 1.96) * sd(d)   # mmHg, not a ratio

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

# irr does not ship a Rdatasets CSV, so get_rdataset("anxiety", "irr") 404s.
# The table is 20 x 3 integers; it is written out here so the block runs with
# no R installation. Column order is rater1, rater2, rater3.
A = np.array([
    [3, 3, 2], [3, 6, 1], [3, 4, 4], [4, 6, 4], [5, 2, 3],
    [5, 4, 2], [2, 2, 1], [3, 4, 6], [5, 3, 1], [2, 3, 1],
    [2, 2, 1], [6, 3, 2], [1, 3, 3], [5, 3, 3], [2, 2, 1],
    [2, 2, 1], [1, 1, 3], [2, 3, 3], [4, 3, 2], [3, 4, 2],
], dtype=float)
ns, nr = A.shape

long = pd.DataFrame({
    "y": A.ravel(order="F"),
    "subj": np.tile(np.arange(ns), nr),
    "rater": np.repeat(np.arange(nr), ns),
})

fit = smf.ols("y ~ C(subj) + C(rater)", data=long).fit()
tab = sm.stats.anova_lm(fit, typ=1)
MSs = tab.loc["C(subj)", "sum_sq"] / tab.loc["C(subj)", "df"]
MSr = tab.loc["C(rater)", "sum_sq"] / tab.loc["C(rater)", "df"]
MSe = tab.loc["Residual", "sum_sq"] / tab.loc["Residual", "df"]

vs = (MSs - MSe) / nr      # the two-way subject variance
vr = (MSr - MSe) / ns
ve = MSe

# The one-way model has no rater term: rater variance and residual collapse
# into one within-subject mean square, and the subject variance has to be net
# of MSw, not MSe. Using vs here makes oneway equal to agreement exactly.
MSw = ((tab.loc["C(rater)", "sum_sq"] + tab.loc["Residual", "sum_sq"])
       / (tab.loc["C(rater)", "df"] + tab.loc["Residual", "df"]))
vs_ow = (MSs - MSw) / nr

# Three single-measurement forms. Consistency and agreement differ by one
# line in the denominator; the one-way form has no rater term, so both its
# numerator and its denominator change.
print({
    "oneway":      vs_ow / (vs_ow + MSw),
    "consistency": vs / (vs + ve),
    "agreement":   vs / (vs + vr + ve),
})

# pingouin.intraclass_corr(data=long, targets="subj", raters="rater", ratings="y")
# returns all six forms with their intervals, which is a useful cross-check.

How to read the report

Work down this list against the paper. If any item is missing, the number cannot be quoted and cannot be compared with anyone else’s.

1. state the model: one-way, two-way random, or two-way mixed
2. state the type: consistency or absolute agreement
3. state the unit: a single measurement or the mean of k, and give k
4. state how many subjects and how many raters, since the confidence interval is driven by both
5. report the confidence interval, not the point estimate alone
6. describe the spread of the subjects, because the ICC rises with it and does not transfer to a narrower population
7. for a measurement with units, report the limits of agreement alongside the ICC

The last item is the one most often dropped. A unitless ratio cannot answer whether a measurement is precise enough to act on. The blood-pressure section above is the counterexample: an ICC of 0.9973 alongside limits of agreement 8.86 mmHg wide, on the same people and the same readings.

As for the familiar cut-points for what counts as good reliability — they are conventions, not properties. Since six forms of one dataset can differ by a factor of 2.59, any fixed threshold loses its meaning the moment the form is unstated. A threshold is only useful once the form is pinned down.

Common misuses

How this page relates to the others

• The same variance split, put to different use — see clustered and repeated measurements. There the ICC is a nuisance to correct for, here it is the result to report, and the model is the same one.
• Where reliability comes from — see measurement error. The constant that regression dilution multiplies by is the single-measurement ICC on this page.
• The categorical version — see kappa and rater agreement.
• Agreement in the original units — see Bland-Altman agreement: the same blood-pressure data, asked a different way.
• Where rater reliability sits in a diagnostic study — see the design chapter on diagnostic accuracy. An index test that cannot be read reliably has its accuracy capped by its reliability.
• The analysis of variance underneath — see t tests and analysis of variance. The three mean squares on this page are that table.

Rerunning every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B9-02-icc.R

Report a problem on this page