AdvancedIndependently reviewed, not yet spot-checked by a human

Measurement error and misclassification

"Non-differential misclassification biases towards the null" is the most widely taught rule in this area, and the most widely over-extended. This page simulates it holding for a binary exposure, then reversing for a three-level one, and lays out regression dilution in continuous variables, error in the outcome, and the residual confounding left behind when a confounder is measured crudely.

Two problems that need separating first

“This variable was not measured accurately” is one sentence in a paper and two different things statistically:

Binary or categorical variableContinuous variable
What it is calledMisclassificationMeasurement error
How it is describedSensitivity and specificityStandard deviation of the error, reliability
Typical consequenceThe effect estimate is pushed towards the null, with exceptionsThe regression coefficient is systematically shrunk

Both have a second dimension: is the error even-handed, or does it have a direction? If the probability of measuring wrongly is unrelated to the other variable, the error is non-differential; if cases recall an exposure more readily than controls, or the exposed group is followed more attentively, it is differential. This dimension does not decide how severe the problem is — it decides whether the direction is predictable.

Every number on this page comes from a simulation whose truth is hard-coded: exposure prevalence 30%, risk in the unexposed 10%, true odds ratio 2.0. Measured without any error at all, the estimate comes out at 1.975; that is the scale of the simulation’s own noise, and every number below should be compared with it rather than with 2.0.

Non-differential misclassification: towards the null, and specificity hurts more

SensitivitySpecificityApparent exposure prevalenceObserved odds ratio
1.001.0029.9%1.975
0.951.0028.4%1.935
0.901.0026.9%1.911
0.801.0023.9%1.850
1.000.9831.3%1.924
1.000.9533.4%1.871
1.000.9037.0%1.775
0.900.9034.0%1.682
0.800.8038.0%1.472

Every row is smaller than the first. The direction is predictable: towards 1. That is what makes the rule useful — when you see a positive result contaminated by non-differential misclassification, you at least know the truth can only be further from the null, never closer to it.

The two parameters are not equivalent either: for an equal loss, specificity does the greater damage. With sensitivity down at 0.90 the odds ratio is 1.911; with specificity down at the same 0.90 it is already 1.775.

Left panel: the horizontal axis is the sensitivity or specificity of the measurement instrument, from 0.6 to 1.0, and the vertical axis is the observed odds ratio. Both curves lie below the horizontal dashed line marking the true value of 2.0, the specificity curve (triangles) sits clearly below the sensitivity curve (circles), and where the axis reaches 1.0 the two meet at 1.975, the estimate obtained under perfect measurement, which is itself slightly below the dashed line. Right panel: the horizontal axis is the standard deviation of the measurement error, from 0 to 2, and the vertical axis is the estimated regression slope; the simulated points fall exactly on the theoretical curve of reliability times the true slope, descending from the true value of 1.5 to about 0.3.
Left: for a binary exposure whose sensitivity and specificity sum to more than 1, non-differential misclassification compresses the odds ratio towards 1, and an equal loss of specificity does more damage than an equal loss of sensitivity, because the unexposed are the larger group and so generate more false positives in absolute terms. Neither the compression nor that ordering survives a three-level exposure, as the table further down shows. Right: for a continuous variable the dilution is not only predictable in direction but has a closed-form magnitude.Plotting script figures/scripts/B8-04-measurement-error.R

Differential misclassification: the direction is no longer predictable

The same data again, but now the probability of misclassification depends on the outcome — the structure of recall bias, where cases who have been questioned repeatedly are likelier to remember an exposure than controls are.

ScenarioSensitivity / specificity in casesSensitivity / specificity in controlsObserved odds ratio
Cases recall better0.90 / 0.950.70 / 0.952.392
Controls recall better0.70 / 0.950.90 / 0.951.227
Cases over-report0.95 / 0.850.95 / 0.952.294
Non-differential, same average0.80 / 0.950.80 / 0.951.695

The first row inflates a true value of 2.0 to 2.392; the second compresses it to 1.227 — one truth, one average quality of instrument, two opposite directions. The third row shows that a modest loss of specificity in the cases alone, that is, over-reporting, is enough to produce 2.294.

The fourth row is the comparison: fix sensitivity at the unweighted midpoint of the first two rows and make it identical in both groups, and the result is 1.695, dutifully biased towards the null. (“Unweighted” because cases are a minority, so the sensitivity averaged over people is not this value — the row exists as a same-parameters, differentiality-removed comparison, not as a strict average.) What separates differential from non-differential error is not how badly you measured, but how evenly.

With three categories or more, “towards the null” stops being a guarantee

The rule is almost always taught without its premise: the exposure has to be binary. An ordered three-level exposure carries no such guarantee.

The three exposure levels are distributed 50% / 30% / 20%, with true odds ratios of 1, 2 and 4 respectively. Both confusion matrices below are non-differential — cases and controls are misclassified with the same probabilities — and their overall correct-classification rates were deliberately matched (both 77%), so the only thing that separates them is the direction of the error:

True levelSymmetric: classified as (low / middle / high)Asymmetric: classified as (low / middle / high)
Low80%20%0%90%10%0%
Middle15%70%15%15%70%15%
High0%20%80%0%45%55%

The symmetric version leaks each level evenly into its neighbours. The asymmetric version pushes 45% of the highest level down into the middle one, so the middle level ends up salted with a large number of genuinely high-exposure people.

ComparisonTruthMeasured perfectlyAfter symmetric misclassificationAfter asymmetric misclassification
middle vs low2.02.0281.7162.134
high vs low4.04.0253.2003.105

The symmetric column behaves as advertised: both rows are compressed towards 1. In the asymmetric column the first row is 2.134, above the 2.028 the same comparison returns under perfect measurement — non-differential misclassification has pushed the middle-level estimate away from the null, and past its true value of 2.0 as well.

Continuous variables: regression dilution, with a magnitude you can compute

A single blood pressure reading, a salt intake estimated from one questionnaire, a biomarker from one blood draw — all carry measurement error. If the error is classical, meaning independent of the true value and zero on average, the regression coefficient is multiplied by a constant smaller than 1. That constant is the reliability: the variance of the true value divided by the variance of the observed value.

That quantity has a name, and you will meet it in papers: it is the intraclass correlation coefficient (ICC) in its reliability sense. The repeat-measurement sub-study described below is exactly how it gets estimated. The ICC comes in six forms, and a paper that reports one number without saying which form cannot be interpreted; see measurement reliability and the ICC.

Error SDReliabilityEstimated slopeReliability × true slope
0.001.0001.5001.500
0.250.9411.4121.412
0.500.8001.2011.200
0.750.6400.9610.960
1.000.5000.7540.750
1.500.3080.4610.462
2.000.2000.3000.300

The last two columns are all but identical. This is not an empirical regularity; it is algebra. When the error SD equals the SD of the true value (fifth row), exactly half the slope survives.

Error in the outcome: specificity does most of the damage

Everything so far has been about the exposure. Error in the outcome has a different structure, and for rare outcomes it has a counter-intuitive property. In the table below the true risk ratio is 2.0 and the risk in the unexposed is only 2%.

Sensitivity of outcome ascertainmentSpecificity of outcome ascertainmentObserved risk ratio
1.001.001.979
0.901.001.999
0.701.001.985
1.000.991.654
1.000.981.494
0.900.981.466

Top three rows: with specificity perfect, sensitivity can fall to 0.70 without moving the risk ratio at all (1.985). The events that are missed are missed in the same proportion in both groups, so the ratio is untouched.

Bottom three rows: specificity slips to 0.99 and the risk ratio falls from 1.979 to 1.654; at 0.98 it is 1.494.

The reason is that the baseline risk is only 2%: a false positive rate of 2% applied to the 98% of people who are healthy generates spurious events of the same order of magnitude as the real ones, and both groups get diluted down into the same background noise.

A crudely measured confounder: residual confounding

The last kind, and the easiest to overlook. In this simulation the exposure has no effect on the outcome whatsoever — the true odds ratio is 1 — and the entire apparent association between them comes from a single confounder.

SD of measurement error in the confounder used for adjustmentAdjusted odds ratio for the exposure
0.001.017
0.251.086
0.501.280
1.001.676
1.501.938
2.002.090

With no adjustment at all, the odds ratio for the exposure is 2.380. Adjusting for the perfectly measured confounder brings it back to 1.017 — correct. Adjusting for a noisy version removes only part of it: when the error SD equals the confounder’s own SD (fourth row), a residual odds ratio of 1.676 is still standing.

What to look for when reading a paper

  1. When the exposure was measured, and by whom. An exposure asked about after the outcome has occurred requires a discussion of differential misclassification by default.
  2. Whether there is a validation sub-study. With no estimate of sensitivity, specificity or reliability, any defence along the lines of “misclassification would only make us underestimate” is an assertion, not an argument.
  3. How the outcome was defined. Diagnosis codes, self-report and adjudication committees differ enormously in specificity, and specificity dominates everything once the outcome is rare.
  4. How many categories the exposure has. With three or more, “biased towards the null” no longer applies.
  5. The granularity of the confounders. A continuous confounder recorded as yes/no is an announcement that residual confounding will remain.
  6. Whether any quantitative bias analysis was done. “Measurement error may be present” is a description, not an analysis; substituting plausible sensitivity and specificity values and recomputing is.

Run it yourself

set.seed(20260824)
n <- 200000

# Truth: 30% exposed, risk 0.10 unexposed, odds ratio 2
E <- rbinom(n, 1, 0.3)
Y <- rbinom(n, 1, ifelse(E == 1, 2 / 11, 0.10))

or <- function(e, y) {
  t <- matrix(as.numeric(table(e, y)), 2, 2)   # doubles: integers overflow here
  (t[2, 2] * t[1, 1]) / (t[2, 1] * t[1, 2])
}
or(E, Y)

# Non-differential: same sensitivity and specificity in cases and non-cases
mis <- function(e, sens, spec) rbinom(length(e), 1, ifelse(e == 1, sens, 1 - spec))
or(mis(E, 0.90, 0.90), Y)                   # pulled toward 1

# Differential: cases recall the exposure better than non-cases
sens <- ifelse(Y == 1, 0.90, 0.70)
or(rbinom(n, 1, ifelse(E == 1, sens, 0.05)), Y)   # can go either way

# Regression dilution for a continuous exposure
x  <- rnorm(n); y <- 2 + 1.5 * x + rnorm(n, 0, 2)
xs <- x + rnorm(n, 0, 1)                    # reliability = 1 / (1 + 1^2) = 0.5
coef(lm(y ~ xs))[2]                         # about half of 1.5

Verified with R 4.6.0. Every number on this page is simulated, with the seed fixed at 20260824. The snippet below is a quick-to-rerun sketch. It draws in a different order from the simulation behind the tables on this page, so its numbers land in the same ballpark without matching digit for digit.

Common misuses

MisuseWhy it is wrong
Defending any positive result with “non-differential misclassification only biases towards the null”The premises are a binary exposure, non-differential error, independence between the exposure’s misclassification and the outcome’s, and sensitivity plus specificity above 1 — and even then it only holds in expectation; with three or more categories it can reverse
Treating a recalled exposure as non-differentially measuredExposure information collected after the outcome is known is differential by default
Reporting only the sensitivity of outcome ascertainmentFor a rare outcome the risk ratio is crushed by specificity, not sensitivity
Saying “adjusted for smoking” without saying how smoking was measuredA crudely measured confounder leaves residual confounding that is indistinguishable from a real effect
Running a dose-response analysis on a single measurement without mentioning dilutionThe slope is multiplied by the reliability, so the trend is systematically understated
Reading a diluted non-significant result as “there is no association”Measurement quality systematically reduces power; that is a design problem, not a biological conclusion
Writing “measurement error may be present” in the limitations and stopping thereThat is a description, not an analysis; substituting sensitivity and specificity and recomputing is quantitative bias analysis
Defining the exposure and the outcome from the same source without discussing correlated errorIf the two errors are correlated, even the direction becomes unpredictable

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B8-04-measurement-error.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

In the non-differential misclassification table, perfect measurement gives an odds ratio of 1.975. Sensitivity alone falling to 0.90 gives 1.911. What does specificity alone falling to 0.90 give, and why?

Show the answer and why

Correct answer: 1.775. For the same drop, specificity does more damage, because the unexposed are the majority and false positives are manufactured out of the larger group

1.775 sits below the 1.911 from the sensitivity row, so in this simulation an equal drop in specificity really does cost more. The reason is prevalence: exposure runs at three tenths, the unexposed are the majority, and a percentage point off specificity manufactures false positives out of the larger group while a percentage point off sensitivity misses true positives in the smaller one. 1.472 is the row where sensitivity and specificity both fall further, and the claim attached to it - that the ordering holds regardless of prevalence - is wrong, because the ordering reverses when exposure is common. 1.904 comes from the separate curve that varies sensitivity only, and calling the two parameters symmetric is the opposite of what this section demonstrates. What does not depend on prevalence is the direction, not which parameter hurts more, and even the direction carries three preconditions: a binary exposure, misclassification of exposure independent of misclassification of outcome, and sensitivity plus specificity above 1.

All four rows of the recall-bias table have a true odds ratio of 2. The first row, cases recalling better, pushes it to 2.392. What does the second row, controls recalling better, give, and what does that show?

Show the answer and why

Correct answer: 1.227. One true value, one average quality of instrument, and the direction can reverse - what makes differential misclassification hard is not how wrong it is but how unevenly

1.227 pushes the truth down to less than half, while the first row's 2.392 exaggerates the same truth - the two rows use the same sensitivities and specificities, with only the question of who is more accurate swapped. Differential misclassification therefore has no predictable direction, which is what makes it harder to live with than the non-differential kind: you cannot even fall back on saying the truth must lie further from the null. 1.695 is the fourth row, which deliberately sets both groups to the same parameters and so removes the differential part, which is why it obediently biases towards the null; taking it as the general rule for differential misclassification reverses the section's conclusion. 2.294 is the third row, where the specificity among cases slips a little through over-reporting, and it does exaggerate - but it is one of four possibilities, not the rule. The practical test is when the exposure information was collected: an exposure asked about after the outcome is known makes differential misclassification a default topic.

For a three-category exposure, two confusion matrices are given the same overall correct-classification rate on purpose. Under symmetric misclassification the middle-versus-low odds ratio is 1.716, and perfect measurement gives 2.028. What does the asymmetric matrix give?

Show the answer and why

Correct answer: 2.134, higher than perfect measurement. Non-differential misclassification has pushed the middle stratum away from the null, so the bias-towards-the-null rule fails once there are more than two categories

2.134 exceeds the 2.028 from perfect measurement and exceeds this comparison's own truth as well - a non-differential error has pushed the estimate past the null on the far side. The mechanism is that the asymmetric matrix moves a large share of the top stratum into the middle one, so the middle stratum fills up with genuinely high-exposure people and its risk rises. 3.105 and 3.200 both belong to the high-versus-low row rather than the one asked about, and the claim attached to 3.200 - that a matching correct-classification rate forces a matching answer - is exactly what this table refutes, since the two matrices were given identical accuracy on purpose and differ only in the direction of the error. 1.716 is the symmetric column, which does follow the rule. The practical consequence lands on dose-response: a tidy staircase may owe part of its shape to the top stratum being under-read, so observing a dose-response trend and therefore inferring causation deserves a discount whenever the exposure categories are coarse or self-reported.

In the regression-dilution table for a continuous exposure the true slope is 1.5. What slope is estimated in the row where the error standard deviation equals the true standard deviation, giving a reliability of 0.5?

Show the answer and why

Correct answer: 0.754, about half the true slope. Classical measurement error multiplies the coefficient by the reliability, and that amount is algebra rather than an empirical regularity

0.754 is almost exactly half of 1.5, and the reliability is 0.5 - the coefficient is multiplied by the reliability, a closed form rather than an empirical regularity, and the near-identical estimated-slope and reliability-times-true-slope columns are the evidence. 1.201 belongs to a row with a smaller error standard deviation, and treating dilution as proportional to the size of the error badly understates the damage once the error grows. 0.300 is the row with the largest error, where reliability is only two tenths; mapping a reliability of 0.5 onto a fall to a fifth confuses two rows. Two practical consequences: a repeat-measurement substudy lets the reliability be estimated and the coefficient divided back out, which is regression calibration, and without that substudy there is nothing to calibrate against. And because dilution shrinks the test statistic along with the point estimate, a poorly measured exposure systematically makes a real association harder to detect, so reading such a result as no association turns a measurement problem into a biological conclusion.

The outcome-error table has a true risk ratio of 2 and an unexposed risk of only 2%. With outcome sensitivity down to 0.70 and perfect specificity the observed risk ratio is 1.985. What happens at specificity 0.98 with perfect sensitivity?

Show the answer and why

Correct answer: 1.494. The baseline risk is only 2%, so a false-positive rate of two percentage points acting on the overwhelmingly healthy majority manufactures spurious events on the same scale as the real ones

1.494 is the specificity 0.98 row, and it falls far further than the sensitivity 0.70 row at 1.985. This is peculiar to rare outcomes: real events are scarce to begin with, so a false-positive rate acting on the overwhelmingly healthy majority creates spurious events on the same scale as the real ones, and both arms are diluted down towards the same background noise. 1.999 comes from the row with perfect specificity and sensitivity alone reduced - the first three rows really do barely move, but that is a statement about sensitivity, since missed events are proportional across the arms and the ratio survives, and carrying it over to specificity reverses the conclusion. 1.654 is the row where specificity drops by one point only, and it sits above 1.494, so the table refutes the claim that a two point drop hurts less. The practical consequence is for database studies that define outcomes with diagnosis codes: the usual worry is that the codes miss patients, but for a rare outcome what destroys the estimate is false positives, so a validation study should report positive predictive value and specificity.

In this simulation the exposure has no effect on the outcome at all and the whole association comes from one confounder. With no adjustment the exposure odds ratio is 2.380, and adjusting for a perfectly measured confounder returns 1.017. What happens when the confounder is measured with an error standard deviation equal to its own?

Show the answer and why

Correct answer: 1.676. Adjustment removes only part of the confounding, and what remains is indistinguishable from a real effect

1.676 is still a long way from the truth, and it is an adjusted number - most of the adjustment was eaten by measurement error in the confounder. 1.086 belongs to the row with very little error, and taking it as the general case yields in the model is enough, which is the opposite of what this table says row after row. 2.090 is the row with the most error; it does approach the unadjusted 2.380 without reaching it, because this column varies continuously rather than switching between adjusted and unadjusted. The practical version: adjusted for smoking is not the same as smoking is no longer a confounder, because smoking recorded as a yes-or-no box carries no information about pack-years, and the confounding that survives is indistinguishable from a real effect. So the limitations paragraph should not say that all known confounders were adjusted for; it should say how those confounders were measured and roughly how much confounding is likely to remain.

Besides the odds ratio, the non-differential table carries a column for the exposure prevalence as it appears after misclassification. The table prints it as a percentage; written as a proportion the truth is 0.30, and the row where sensitivity alone falls to 0.90 reads 0.269. What does the row where specificity alone falls to 0.90 read, and what is that column good for?

Show the answer and why

Correct answer: It reads 0.370, above the truth. Losing sensitivity pushes the apparent prevalence down and losing specificity pushes it up, and the upward push is the larger of the two

0.370 sits above the true 0.30, while the same drop in sensitivity pushes it down to 0.269: the same ten points move it up more than twice as far as they move it down. The reason is the one behind the odds ratio column - only three in ten are exposed, so the false positives created by imperfect specificity come out of the large unexposed group, while the true positives missed by imperfect sensitivity can only come out of the smaller one. 0.239 is a row further down the sensitivity-only block, and it too sits below the truth, but using it to claim that misclassification always pulls the prevalence down ignores the false-positive route, which is what the lower half of the same table is doing. 0.313 is the row where specificity slips by two points only, and reading it as the answer for a ten-point drop is how the symmetric conclusion arises, which this entire column refutes. The practical use is direct: when a study reports an exposure prevalence that will not line up with a registry or a national survey, ask how exposure was measured - too high points at specificity, too low at sensitivity.

Sources and licences

This page is original writing

Report a content problem

The statistics on this site are written by AI and reviewed by AI; a human only spot-checks. What you can see may be what we cannot.

The more specific, the more fixable — e.g. which sentence disagrees with which textbook or paper.

Needed only if you want a reply; reports without it are still read.

Sent along with your report

These are attached automatically. You can drop any of them.