Agreement and the kappa family

This page fills a gap the rest of the site leaves open

Diagnostic accuracy studies require inter-rater agreement to be reported, and systematic reviews require the agreement between the two screeners — and both chapters dispatch it in a single sentence saying to report a kappa. This page is that kappa, in full.

There are two common ways to land here. One is reading a Methods section that says inter-rater agreement was substantial, with a kappa beside it, and wanting to know what that number is guarding against. The other is that agreement is your own first study: two people each read the films, or the charts, or the abstracts, and something has to be reported. The second case needs this page more, because kappa has several places where it quietly hands you a number that looks perfectly ordinary and means something else.

By the end you should be able to ask four things: does the scale have an order (which decides whether to weight), are there two raters or more (Cohen or Fleiss), how common is the target finding (which decides whether kappa gets crushed), and who computed that confidence interval.

Observed agreement, minus the part that guessing would produce

The first dataset is irr::diagnoses: 30 patients, 6 raters, and every patient read once by all six. The five diagnostic categories are Depression, Personality Disorder, Schizophrenia, Neurosis and Other.

Start with two of the raters, rater1 and rater2. Their cross-tabulation:

The diagonal sums to 22, so observed agreement is 22/30 = 73.3%. That figure on its own is useless, and the reason is simple: two people filling the form blindfolded, at the same category frequencies, would still land on the same cell some of the time. Multiplying these two raters’ marginals gives 23.6% from chance alone.

Kappa subtracts that first and then asks how much of what is left got filled in:

The denominator $1 - p_e$ is the room chance cannot explain, the part left for skill: 76.4% here. The numerator $p_o - p_e$ is the agreement actually achieved beyond chance: 49.8%. Dividing gives κ = 0.651.

So kappa reads like this: beyond what chance can account for, these two raters filled in 65.1% of the room that was left. It is not an agreement rate, and it does not convert into one — the paradox section below shows one observed agreement mapping onto kappas more than three times apart.

Is 0.65 good? The Landis-Koch cut points have no test behind them

Almost every paper reporting a kappa attaches an adjective, on the authority of the Landis and Koch table from 1977: cut points at 0.2, 0.4, 0.6 and 0.8 slicing kappa into five bands.

That table is a convention the two authors proposed so the numbers would be easier to talk about, not something derived from data. There is no test behind it, no distribution, and no calibration against clinical consequence. The same κ = 0.651 means very different things for two radiologists deciding whether a bone is fractured and for two screeners deciding whether an abstract goes to full text, and that table says the same word about both.

The figure on this page deliberately draws those four cut points on the colour scale without printing the five adjectives, because printing them would be an endorsement. In practice: report the point estimate and its interval, and let the reader judge; if you use an adjective, say that it is the Landis-Koch convention.

Weighted kappa: first ask whether the scale has an order

Unweighted kappa treats every disagreement as equally bad: BI-RADS 4 against 5 loses exactly as much as 1 against 5. On an ordered scale that is obviously wrong, which is what weighted kappa is for — every cell gets a weight $w_{ij}$ between 0 and 1, full agreement scoring 1 and the furthest possible disagreement scoring 0.

Both common weight sets depend only on how many steps apart the two readings are (write $d = |i-j|$ on a $k$-point scale): linear weights are $1 - d/(k-1)$, quadratic weights are $1 - d^2/(k-1)^2$. Quadratic weights forgive a one-step miss almost entirely and punish a distant miss harder than linear ones do.

All of which requires that being one step apart mean something. The next two sections run identical arithmetic on two kinds of scale, and reach opposite conclusions.

The counter-example: these five diagnoses have no order

Linear-weighted kappa (0.633) comes out lower than the unweighted one (0.651), and that is not an arithmetic slip. The first two columns explain it: weighting lifts observed agreement from 0.733 to 0.858, which looks like a gain, but it lifts expected agreement from 0.236 to 0.614 at the same time — expected agreement rises further than observed agreement does (0.378 against 0.125). Numerator and denominator both shrink, and the numerator shrinks by slightly more. Handing partial credit to disagreements that are not really near misses gives more away to chance than it gives to skill.

So this set of numbers is a counter-example. There is no distance between these five diagnoses: schizophrenia does not sit halfway between personality disorder and neurosis. Weighting nominal categories produces a number, and that number corresponds to no clinical idea at all.

The real case: a 6-point anxiety scale, where weighting earns its keep

A different dataset: irr::anxiety, in which 20 subjects were each rated by 3 raters on a 6-point anxiety scale. On a scale like this, one step apart is a real quantity, so weighting means something.

This pair (rater1 and rater2) agree exactly on only 6 of 20 subjects, an observed agreement of 30.0%. But of the 14 disagreements, 7 are one point apart — half of all the disagreements there are, and unweighted kappa charges that half the full penalty.

So the three kappas rise monotonically, in the opposite direction to the nominal set:

The quadratic point estimate is 2.48 times the unweighted one.

Weighting multiplied the estimate by two and a half, and changed nothing

Three raters or more: Fleiss kappa

Cohen’s kappa takes exactly two raters at a time. Real studies routinely have three, or six, or a different pair of readers per case.

The laziest and most common response is to average all the pairwise kappas. First, look at what is being averaged:

These 15 pairs run from 0.081 (rater1 against rater6) to 0.857 (rater4 against rater5), with a median of 0.439. Averaging them into one number asserts that all 15 are measuring the same thing, and plainly they are not: R1 against R2 is 0.651, ranking 3 of 15, yet R1 sits far lower against each of the other four and has the lowest row average of the six.

Putting raw agreement and kappa side by side for each pair shows a second thing:

The two rank columns are not the same order: setting the 15 pairs against each other gives 105 head-to-head comparisons, and in 4 of them raw agreement puts one pair above another while kappa reverses them. The cause is always the same — each pair of raters has its own marginals, and therefore its own expected agreement, so kappa subtracts a different amount from each. This is also why raw agreement and kappa cannot be converted into each other.

The right move is an estimator that takes all the raters at once. Fleiss kappa has the same shape, $(p_o - p_e)/(1 - p_e)$, with $p_o$ redefined as the average, over patients, of the proportion of rater pairs agreeing on that patient, and $p_e$ as the sum of squared category shares across all raters pooled. It needs no pairing; the price is that it pools every rater into one common category distribution — that definition of $p_e$ is itself the assumption that the six raters are interchangeable.

For this dataset Fleiss kappa is 0.430 (subject-level bootstrap percentile interval, 0.315 to 0.529), with observed agreement 55.6% and expected agreement 22.0%.

This dataset happens to test that assumption directly: rater1 gives 43.3% of its readings to Depression and rater6 never gives that reading at all. The estimator that keeps each rater’s own marginal is Conger (1980). Observed agreement is identical (55.6%, it is literally the same quantity); expected agreement is 0.204 rather than the Fleiss 0.220, so kappa becomes 0.442 (bootstrap interval 0.338 to 0.533), higher than Fleiss by 0.012.

The gap is small but its direction is fixed: pooling raters with different habits into one distribution can only push expected agreement up, so kappa is charged a little more. Reporting Fleiss means accepting that assumption; a reader unwilling to accept it wants Conger.

One overall number still hides something. Split the kappa by diagnostic category:

The overall 0.430 covers up something more clinically useful: the six raters diverge most on depression and personality disorder, both at 0.245, while the loosest-sounding category, Other, is where they agree best at 0.566. What needs fixing is the definition of those first two categories, not another training session for everybody. Per-category kappas cost almost nothing and are almost never reported.

The kappa paradox: identical agreement, kappas three times apart

Observed agreement is 0.900 in both, identical to the last decimal place. Kappa nonetheless falls from 0.800 to 0.232 — the first is 3.45 times the second.

The reason is the expected agreement. On the left each rater calls half the readings abnormal, so chance alone lands them on the same cell 0.500 of the time. On the right both raters give 93% of their readings to abnormal, which means that two people who never looked at a single film, each ticking abnormal at random with their own 93% rate, would still agree 0.870 of the time. The denominator $1 - p_e$ is squeezed almost flat, so there is very little left to prove, and kappa is duly small.

This is not kappa going wrong. It is kappa answering, exactly as defined, a demanding question: how much is left once the agreement produced by both raters leaning the same way has been taken out. For a rare finding — or a near-universal one — very little is left by construction.

Three practical consequences:

1. A low kappa does not necessarily mean the raters are poor. Where the target is lopsided (a 2% screening positivity rate, a rare histological pattern, the small fraction of abstracts that genuinely belong in a review), kappa is low by nature. Judging rater quality means looking at the prevalence too.
2. Kappas from different studies are not directly comparable unless the prevalences are similar. The same raters on a different case mix will produce a different kappa.
3. Report the 2x2 table (or the $k \times k$ table) alongside the kappa. With the table, a reader can compute PABAK, the prevalence index and the bias index themselves; from a lone kappa nothing can be recovered.

As for PABAK, the prevalence-adjusted bias-adjusted kappa, it forces the marginals flat and so returns the same 0.800 for both tables. It is a useful companion number — it shows how much of a low kappa is prevalence — but it is not a correction, because what it throws away is the real information about how common the finding is in these patients. Report both; do not let one replace the other.

Running it yourself

library(irr)
data(diagnoses, package = "irr")

# rater6 never uses "1. Depression", so its factor carries only 4 levels.
# as.integer() would shift every one of its codes down by one and get the
# whole matrix wrong in silence. Map by LABEL with match() instead.
CATS <- c("1. Depression", "2. Personality Disorder", "3. Schizophrenia",
          "4. Neurosis", "5. Other")
M <- sapply(diagnoses[paste0("rater", 1:6)],
            function(x) match(as.character(x), CATS))

kappa2(M[, c(1, 2)])              # two raters: Cohen
kappam.fleiss(M)                  # six raters: Fleiss
kappam.fleiss(M, detail = TRUE)   # plus the per-category kappas

# -- an ordinal scale, and a place that bites quietly ----------------------
data(anxiety, package = "irr")
A <- as.matrix(anxiety[c("rater1", "rater2", "rater3")])

# kappa2() builds its category list from the ratings it OBSERVES. Neither
# rater2 nor rater3 ever says 5, so that rung vanishes and 4 and 6 become
# adjacent.
kappa2(A[, c(2, 3)], weight = "equal")   # returns 0.146, on a silently 5-point scale

# Passing factor(x, levels = 1:6) does not rescue it: kappa2() calls
# as.matrix() first and the levels are gone by then (checked, still 0.146).
# R has no labels argument here, so build the k x k table yourself.
kw <- function(x, y, k, type) {
  tt <- table(factor(x, levels = 1:k), factor(y, levels = 1:k))
  p <- tt / sum(tt)
  d <- abs(outer(1:k, 1:k, "-"))
  w <- switch(type, unweighted = (d == 0) * 1,
                    linear     = 1 - d / (k - 1),
                    quadratic  = 1 - d^2 / (k - 1)^2)
  po <- sum(w * p); pe <- sum(w * outer(rowSums(p), colSums(p)))
  (po - pe) / (1 - pe)
}
kw(A[, 2], A[, 3], k = 6, type = "linear")   # 0.126, the value on the 6-point scale

# Before anything else, ask whether the two raters between them used the
# whole scale:
sort(unique(c(A[, 2], A[, 3])))   # a rung missing means kappa2() cannot be trusted

# Intervals: the z and p that irr prints come from the standard error under
# kappa = 0 — right for the test, wrong for an interval. Compute the
# unrestricted asymptotic SE, or bootstrap over subjects.
set.seed(20260823)
b <- replicate(4000, { i <- sample.int(nrow(A), nrow(A), replace = TRUE)
                       kw(A[i, 2], A[i, 3], 6, "linear") })
quantile(b, c(0.025, 0.975))

import numpy as np
import pandas as pd
from sklearn.metrics import cohen_kappa_score
from statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters

# irr::anxiety, 20 x 3. Small enough to carry here, so this block runs alone.
anx = pd.DataFrame(
    [[3, 3, 2], [3, 6, 1], [3, 4, 4], [4, 6, 4], [5, 2, 3],
     [5, 4, 2], [2, 2, 1], [3, 4, 6], [5, 3, 1], [2, 3, 1],
     [2, 2, 1], [6, 3, 2], [1, 3, 3], [5, 3, 3], [2, 2, 1],
     [2, 2, 1], [1, 1, 3], [2, 3, 3], [4, 3, 2], [3, 4, 2]],
    columns=["rater1", "rater2", "rater3"],
)
r2, r3 = anx["rater2"].to_numpy(), anx["rater3"].to_numpy()

# The same trap: with no labels argument the category list is built from the
# ratings observed, and the unused rung disappears.
cohen_kappa_score(r2, r3, weights="linear")                       # 0.146
cohen_kappa_score(r2, r3, labels=range(1, 7), weights="linear")   # 0.126

# Python does have a labels argument, so one line pins the scale down — but
# you have to remember to write it. The default quietly returns a different
# number whenever some rung of the scale went unused.

# Three raters or more: fleiss_kappa wants a subject-by-category count
# matrix, and aggregate_raters is what converts the raw ratings into one.
table, _ = aggregate_raters(anx.to_numpy())
fleiss_kappa(table, method="fleiss")

Six things to settle before your own first agreement study

1. Does the scale have an order? If it does, use weighted kappa and fix linear or quadratic in the protocol. If it does not, do not weight.
2. How many raters? Two means Cohen, three or more means Fleiss, and never average the pairwise kappas.
3. How common is the target finding? A lopsided prevalence means a low kappa should be expected, so report the 2x2 table and let readers compute PABAK for themselves.
4. Who computed the confidence interval? The p-value a package prints tests κ = 0; it is not an interval. Check where the interval came from.
5. Did the raters between them use the whole scale? If not, the package will silently shorten it — the one trap on this page that is invisible without reading the code.
6. Should this be a different statistic entirely? Kappa is for categorical readings. Read the same irr::anxiety scores as a continuum and the right tool is the ICC, not kappa; how far wrong a rater goes can also be framed as measurement error. And agreement is not accuracy — two people can be very consistently wrong, which is a question for diagnostic accuracy, not for this page.

Report a problem on this page