Measurement error and misclassification
"Non-differential misclassification biases towards the null" is the most widely taught rule in this area, and the most widely over-extended. This page simulates it holding for a binary exposure, then reversing for a three-level one, and lays out regression dilution in continuous variables, error in the outcome, and the residual confounding left behind when a confounder is measured crudely.
Two problems that need separating first
“This variable was not measured accurately” is one sentence in a paper and two different things statistically:
| Binary or categorical variable | Continuous variable | |
|---|---|---|
| What it is called | Misclassification | Measurement error |
| How it is described | Sensitivity and specificity | Standard deviation of the error, reliability |
| Typical consequence | The effect estimate is pushed towards the null, with exceptions | The regression coefficient is systematically shrunk |
Both have a second dimension: is the error even-handed, or does it have a direction? If the probability of measuring wrongly is unrelated to the other variable, the error is non-differential; if cases recall an exposure more readily than controls, or the exposed group is followed more attentively, it is differential. This dimension does not decide how severe the problem is — it decides whether the direction is predictable.
Every number on this page comes from a simulation whose truth is hard-coded: exposure prevalence 30%, risk in the unexposed 10%, true odds ratio 2.0. Measured without any error at all, the estimate comes out at 1.975; that is the scale of the simulation’s own noise, and every number below should be compared with it rather than with 2.0.
Non-differential misclassification: towards the null, and specificity hurts more
| Sensitivity | Specificity | Apparent exposure prevalence | Observed odds ratio |
|---|---|---|---|
| 1.00 | 1.00 | 29.9% | 1.975 |
| 0.95 | 1.00 | 28.4% | 1.935 |
| 0.90 | 1.00 | 26.9% | 1.911 |
| 0.80 | 1.00 | 23.9% | 1.850 |
| 1.00 | 0.98 | 31.3% | 1.924 |
| 1.00 | 0.95 | 33.4% | 1.871 |
| 1.00 | 0.90 | 37.0% | 1.775 |
| 0.90 | 0.90 | 34.0% | 1.682 |
| 0.80 | 0.80 | 38.0% | 1.472 |
Every row is smaller than the first. The direction is predictable: towards 1. That is what makes the rule useful — when you see a positive result contaminated by non-differential misclassification, you at least know the truth can only be further from the null, never closer to it.
The two parameters are not equivalent either: for an equal loss, specificity does the greater damage. With sensitivity down at 0.90 the odds ratio is 1.911; with specificity down at the same 0.90 it is already 1.775.
figures/scripts/B8-04-measurement-error.RDifferential misclassification: the direction is no longer predictable
The same data again, but now the probability of misclassification depends on the outcome — the structure of recall bias, where cases who have been questioned repeatedly are likelier to remember an exposure than controls are.
| Scenario | Sensitivity / specificity in cases | Sensitivity / specificity in controls | Observed odds ratio |
|---|---|---|---|
| Cases recall better | 0.90 / 0.95 | 0.70 / 0.95 | 2.392 |
| Controls recall better | 0.70 / 0.95 | 0.90 / 0.95 | 1.227 |
| Cases over-report | 0.95 / 0.85 | 0.95 / 0.95 | 2.294 |
| Non-differential, same average | 0.80 / 0.95 | 0.80 / 0.95 | 1.695 |
The first row inflates a true value of 2.0 to 2.392; the second compresses it to 1.227 — one truth, one average quality of instrument, two opposite directions. The third row shows that a modest loss of specificity in the cases alone, that is, over-reporting, is enough to produce 2.294.
The fourth row is the comparison: fix sensitivity at the unweighted midpoint of the first two rows and make it identical in both groups, and the result is 1.695, dutifully biased towards the null. (“Unweighted” because cases are a minority, so the sensitivity averaged over people is not this value — the row exists as a same-parameters, differentiality-removed comparison, not as a strict average.) What separates differential from non-differential error is not how badly you measured, but how evenly.
With three categories or more, “towards the null” stops being a guarantee
The rule is almost always taught without its premise: the exposure has to be binary. An ordered three-level exposure carries no such guarantee.
The three exposure levels are distributed 50% / 30% / 20%, with true odds ratios of 1, 2 and 4 respectively. Both confusion matrices below are non-differential — cases and controls are misclassified with the same probabilities — and their overall correct-classification rates were deliberately matched (both 77%), so the only thing that separates them is the direction of the error:
| True level | Symmetric: classified as (low / middle / high) | Asymmetric: classified as (low / middle / high) | ||||
|---|---|---|---|---|---|---|
| Low | 80% | 20% | 0% | 90% | 10% | 0% |
| Middle | 15% | 70% | 15% | 15% | 70% | 15% |
| High | 0% | 20% | 80% | 0% | 45% | 55% |
The symmetric version leaks each level evenly into its neighbours. The asymmetric version pushes 45% of the highest level down into the middle one, so the middle level ends up salted with a large number of genuinely high-exposure people.
| Comparison | Truth | Measured perfectly | After symmetric misclassification | After asymmetric misclassification |
|---|---|---|---|---|
| middle vs low | 2.0 | 2.028 | 1.716 | 2.134 |
| high vs low | 4.0 | 4.025 | 3.200 | 3.105 |
The symmetric column behaves as advertised: both rows are compressed towards 1. In the asymmetric column the first row is 2.134, above the 2.028 the same comparison returns under perfect measurement — non-differential misclassification has pushed the middle-level estimate away from the null, and past its true value of 2.0 as well.
Continuous variables: regression dilution, with a magnitude you can compute
A single blood pressure reading, a salt intake estimated from one questionnaire, a biomarker from one blood draw — all carry measurement error. If the error is classical, meaning independent of the true value and zero on average, the regression coefficient is multiplied by a constant smaller than 1. That constant is the reliability: the variance of the true value divided by the variance of the observed value.
That quantity has a name, and you will meet it in papers: it is the intraclass correlation coefficient (ICC) in its reliability sense. The repeat-measurement sub-study described below is exactly how it gets estimated. The ICC comes in six forms, and a paper that reports one number without saying which form cannot be interpreted; see measurement reliability and the ICC.
| Error SD | Reliability | Estimated slope | Reliability × true slope |
|---|---|---|---|
| 0.00 | 1.000 | 1.500 | 1.500 |
| 0.25 | 0.941 | 1.412 | 1.412 |
| 0.50 | 0.800 | 1.201 | 1.200 |
| 0.75 | 0.640 | 0.961 | 0.960 |
| 1.00 | 0.500 | 0.754 | 0.750 |
| 1.50 | 0.308 | 0.461 | 0.462 |
| 2.00 | 0.200 | 0.300 | 0.300 |
The last two columns are all but identical. This is not an empirical regularity; it is algebra. When the error SD equals the SD of the true value (fifth row), exactly half the slope survives.
Error in the outcome: specificity does most of the damage
Everything so far has been about the exposure. Error in the outcome has a different structure, and for rare outcomes it has a counter-intuitive property. In the table below the true risk ratio is 2.0 and the risk in the unexposed is only 2%.
| Sensitivity of outcome ascertainment | Specificity of outcome ascertainment | Observed risk ratio |
|---|---|---|
| 1.00 | 1.00 | 1.979 |
| 0.90 | 1.00 | 1.999 |
| 0.70 | 1.00 | 1.985 |
| 1.00 | 0.99 | 1.654 |
| 1.00 | 0.98 | 1.494 |
| 0.90 | 0.98 | 1.466 |
Top three rows: with specificity perfect, sensitivity can fall to 0.70 without moving the risk ratio at all (1.985). The events that are missed are missed in the same proportion in both groups, so the ratio is untouched.
Bottom three rows: specificity slips to 0.99 and the risk ratio falls from 1.979 to 1.654; at 0.98 it is 1.494.
The reason is that the baseline risk is only 2%: a false positive rate of 2% applied to the 98% of people who are healthy generates spurious events of the same order of magnitude as the real ones, and both groups get diluted down into the same background noise.
A crudely measured confounder: residual confounding
The last kind, and the easiest to overlook. In this simulation the exposure has no effect on the outcome whatsoever — the true odds ratio is 1 — and the entire apparent association between them comes from a single confounder.
| SD of measurement error in the confounder used for adjustment | Adjusted odds ratio for the exposure |
|---|---|
| 0.00 | 1.017 |
| 0.25 | 1.086 |
| 0.50 | 1.280 |
| 1.00 | 1.676 |
| 1.50 | 1.938 |
| 2.00 | 2.090 |
With no adjustment at all, the odds ratio for the exposure is 2.380. Adjusting for the perfectly measured confounder brings it back to 1.017 — correct. Adjusting for a noisy version removes only part of it: when the error SD equals the confounder’s own SD (fourth row), a residual odds ratio of 1.676 is still standing.
What to look for when reading a paper
- When the exposure was measured, and by whom. An exposure asked about after the outcome has occurred requires a discussion of differential misclassification by default.
- Whether there is a validation sub-study. With no estimate of sensitivity, specificity or reliability, any defence along the lines of “misclassification would only make us underestimate” is an assertion, not an argument.
- How the outcome was defined. Diagnosis codes, self-report and adjudication committees differ enormously in specificity, and specificity dominates everything once the outcome is rare.
- How many categories the exposure has. With three or more, “biased towards the null” no longer applies.
- The granularity of the confounders. A continuous confounder recorded as yes/no is an announcement that residual confounding will remain.
- Whether any quantitative bias analysis was done. “Measurement error may be present” is a description, not an analysis; substituting plausible sensitivity and specificity values and recomputing is.
Run it yourself
set.seed(20260824)
n <- 200000
# Truth: 30% exposed, risk 0.10 unexposed, odds ratio 2
E <- rbinom(n, 1, 0.3)
Y <- rbinom(n, 1, ifelse(E == 1, 2 / 11, 0.10))
or <- function(e, y) {
t <- matrix(as.numeric(table(e, y)), 2, 2) # doubles: integers overflow here
(t[2, 2] * t[1, 1]) / (t[2, 1] * t[1, 2])
}
or(E, Y)
# Non-differential: same sensitivity and specificity in cases and non-cases
mis <- function(e, sens, spec) rbinom(length(e), 1, ifelse(e == 1, sens, 1 - spec))
or(mis(E, 0.90, 0.90), Y) # pulled toward 1
# Differential: cases recall the exposure better than non-cases
sens <- ifelse(Y == 1, 0.90, 0.70)
or(rbinom(n, 1, ifelse(E == 1, sens, 0.05)), Y) # can go either way
# Regression dilution for a continuous exposure
x <- rnorm(n); y <- 2 + 1.5 * x + rnorm(n, 0, 2)
xs <- x + rnorm(n, 0, 1) # reliability = 1 / (1 + 1^2) = 0.5
coef(lm(y ~ xs))[2] # about half of 1.5Verified with R 4.6.0. Every number on this page is simulated, with the seed fixed at 20260824. The snippet below is a quick-to-rerun sketch. It draws in a different order from the simulation behind the tables on this page, so its numbers land in the same ballpark without matching digit for digit.
import numpy as np
rng = np.random.default_rng(20260824)
n = 200_000
E = rng.binomial(1, 0.3, n)
Y = rng.binomial(1, np.where(E == 1, 2 / 11, 0.10))
def odds_ratio(e, y):
a = ((e == 1) & (y == 1)).sum(); b = ((e == 1) & (y == 0)).sum()
c = ((e == 0) & (y == 1)).sum(); d = ((e == 0) & (y == 0)).sum()
return (a * d) / (b * c)
print(odds_ratio(E, Y))
mis = rng.binomial(1, np.where(E == 1, 0.90, 0.10)) # sens .9, spec .9
print(odds_ratio(mis, Y)) # toward the null
x = rng.normal(size=n); y = 2 + 1.5 * x + rng.normal(0, 2, n)
xs = x + rng.normal(0, 1, n)
print(np.polyfit(xs, y, 1)[0]) # about 0.75numpy plus statsmodels is enough; the point is to construct the truth first, so there is something to compare against. This too is a sketch; it draws in a different order from the tables on this page.
Common misuses
| Misuse | Why it is wrong |
|---|---|
| Defending any positive result with “non-differential misclassification only biases towards the null” | The premises are a binary exposure, non-differential error, independence between the exposure’s misclassification and the outcome’s, and sensitivity plus specificity above 1 — and even then it only holds in expectation; with three or more categories it can reverse |
| Treating a recalled exposure as non-differentially measured | Exposure information collected after the outcome is known is differential by default |
| Reporting only the sensitivity of outcome ascertainment | For a rare outcome the risk ratio is crushed by specificity, not sensitivity |
| Saying “adjusted for smoking” without saying how smoking was measured | A crudely measured confounder leaves residual confounding that is indistinguishable from a real effect |
| Running a dose-response analysis on a single measurement without mentioning dilution | The slope is multiplied by the reliability, so the trend is systematically understated |
| Reading a diluted non-significant result as “there is no association” | Measurement quality systematically reduces power; that is a design problem, not a biological conclusion |
| Writing “measurement error may be present” in the limitations and stopping there | That is a description, not an analysis; substituting sensitivity and specificity and recomputing is quantitative bias analysis |
| Defining the exposure and the outcome from the same source without discussing correlated error | If the two errors are correlated, even the direction becomes unpredictable |
Reproducing every number on this page
/opt/homebrew/bin/Rscript figures/scripts/B8-04-measurement-error.RRead the figure
The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.
In the non-differential misclassification table, perfect measurement gives an odds ratio of 1.975. Sensitivity alone falling to 0.90 gives 1.911. What does specificity alone falling to 0.90 give, and why?
Show the answer and why
Correct answer: 1.775. For the same drop, specificity does more damage, because the unexposed are the majority and false positives are manufactured out of the larger group
1.775 sits below the 1.911 from the sensitivity row, so in this simulation an equal drop in specificity really does cost more. The reason is prevalence: exposure runs at three tenths, the unexposed are the majority, and a percentage point off specificity manufactures false positives out of the larger group while a percentage point off sensitivity misses true positives in the smaller one. 1.472 is the row where sensitivity and specificity both fall further, and the claim attached to it - that the ordering holds regardless of prevalence - is wrong, because the ordering reverses when exposure is common. 1.904 comes from the separate curve that varies sensitivity only, and calling the two parameters symmetric is the opposite of what this section demonstrates. What does not depend on prevalence is the direction, not which parameter hurts more, and even the direction carries three preconditions: a binary exposure, misclassification of exposure independent of misclassification of outcome, and sensitivity plus specificity above 1.
All four rows of the recall-bias table have a true odds ratio of 2. The first row, cases recalling better, pushes it to 2.392. What does the second row, controls recalling better, give, and what does that show?
Show the answer and why
Correct answer: 1.227. One true value, one average quality of instrument, and the direction can reverse - what makes differential misclassification hard is not how wrong it is but how unevenly
1.227 pushes the truth down to less than half, while the first row's 2.392 exaggerates the same truth - the two rows use the same sensitivities and specificities, with only the question of who is more accurate swapped. Differential misclassification therefore has no predictable direction, which is what makes it harder to live with than the non-differential kind: you cannot even fall back on saying the truth must lie further from the null. 1.695 is the fourth row, which deliberately sets both groups to the same parameters and so removes the differential part, which is why it obediently biases towards the null; taking it as the general rule for differential misclassification reverses the section's conclusion. 2.294 is the third row, where the specificity among cases slips a little through over-reporting, and it does exaggerate - but it is one of four possibilities, not the rule. The practical test is when the exposure information was collected: an exposure asked about after the outcome is known makes differential misclassification a default topic.
For a three-category exposure, two confusion matrices are given the same overall correct-classification rate on purpose. Under symmetric misclassification the middle-versus-low odds ratio is 1.716, and perfect measurement gives 2.028. What does the asymmetric matrix give?
Show the answer and why
Correct answer: 2.134, higher than perfect measurement. Non-differential misclassification has pushed the middle stratum away from the null, so the bias-towards-the-null rule fails once there are more than two categories
2.134 exceeds the 2.028 from perfect measurement and exceeds this comparison's own truth as well - a non-differential error has pushed the estimate past the null on the far side. The mechanism is that the asymmetric matrix moves a large share of the top stratum into the middle one, so the middle stratum fills up with genuinely high-exposure people and its risk rises. 3.105 and 3.200 both belong to the high-versus-low row rather than the one asked about, and the claim attached to 3.200 - that a matching correct-classification rate forces a matching answer - is exactly what this table refutes, since the two matrices were given identical accuracy on purpose and differ only in the direction of the error. 1.716 is the symmetric column, which does follow the rule. The practical consequence lands on dose-response: a tidy staircase may owe part of its shape to the top stratum being under-read, so observing a dose-response trend and therefore inferring causation deserves a discount whenever the exposure categories are coarse or self-reported.
In the regression-dilution table for a continuous exposure the true slope is 1.5. What slope is estimated in the row where the error standard deviation equals the true standard deviation, giving a reliability of 0.5?
Show the answer and why
Correct answer: 0.754, about half the true slope. Classical measurement error multiplies the coefficient by the reliability, and that amount is algebra rather than an empirical regularity
0.754 is almost exactly half of 1.5, and the reliability is 0.5 - the coefficient is multiplied by the reliability, a closed form rather than an empirical regularity, and the near-identical estimated-slope and reliability-times-true-slope columns are the evidence. 1.201 belongs to a row with a smaller error standard deviation, and treating dilution as proportional to the size of the error badly understates the damage once the error grows. 0.300 is the row with the largest error, where reliability is only two tenths; mapping a reliability of 0.5 onto a fall to a fifth confuses two rows. Two practical consequences: a repeat-measurement substudy lets the reliability be estimated and the coefficient divided back out, which is regression calibration, and without that substudy there is nothing to calibrate against. And because dilution shrinks the test statistic along with the point estimate, a poorly measured exposure systematically makes a real association harder to detect, so reading such a result as no association turns a measurement problem into a biological conclusion.
The outcome-error table has a true risk ratio of 2 and an unexposed risk of only 2%. With outcome sensitivity down to 0.70 and perfect specificity the observed risk ratio is 1.985. What happens at specificity 0.98 with perfect sensitivity?
Show the answer and why
Correct answer: 1.494. The baseline risk is only 2%, so a false-positive rate of two percentage points acting on the overwhelmingly healthy majority manufactures spurious events on the same scale as the real ones
1.494 is the specificity 0.98 row, and it falls far further than the sensitivity 0.70 row at 1.985. This is peculiar to rare outcomes: real events are scarce to begin with, so a false-positive rate acting on the overwhelmingly healthy majority creates spurious events on the same scale as the real ones, and both arms are diluted down towards the same background noise. 1.999 comes from the row with perfect specificity and sensitivity alone reduced - the first three rows really do barely move, but that is a statement about sensitivity, since missed events are proportional across the arms and the ratio survives, and carrying it over to specificity reverses the conclusion. 1.654 is the row where specificity drops by one point only, and it sits above 1.494, so the table refutes the claim that a two point drop hurts less. The practical consequence is for database studies that define outcomes with diagnosis codes: the usual worry is that the codes miss patients, but for a rare outcome what destroys the estimate is false positives, so a validation study should report positive predictive value and specificity.
In this simulation the exposure has no effect on the outcome at all and the whole association comes from one confounder. With no adjustment the exposure odds ratio is 2.380, and adjusting for a perfectly measured confounder returns 1.017. What happens when the confounder is measured with an error standard deviation equal to its own?
Show the answer and why
Correct answer: 1.676. Adjustment removes only part of the confounding, and what remains is indistinguishable from a real effect
1.676 is still a long way from the truth, and it is an adjusted number - most of the adjustment was eaten by measurement error in the confounder. 1.086 belongs to the row with very little error, and taking it as the general case yields in the model is enough, which is the opposite of what this table says row after row. 2.090 is the row with the most error; it does approach the unadjusted 2.380 without reaching it, because this column varies continuously rather than switching between adjusted and unadjusted. The practical version: adjusted for smoking is not the same as smoking is no longer a confounder, because smoking recorded as a yes-or-no box carries no information about pack-years, and the confounding that survives is indistinguishable from a real effect. So the limitations paragraph should not say that all known confounders were adjusted for; it should say how those confounders were measured and roughly how much confounding is likely to remain.
Besides the odds ratio, the non-differential table carries a column for the exposure prevalence as it appears after misclassification. The table prints it as a percentage; written as a proportion the truth is 0.30, and the row where sensitivity alone falls to 0.90 reads 0.269. What does the row where specificity alone falls to 0.90 read, and what is that column good for?
Show the answer and why
Correct answer: It reads 0.370, above the truth. Losing sensitivity pushes the apparent prevalence down and losing specificity pushes it up, and the upward push is the larger of the two
0.370 sits above the true 0.30, while the same drop in sensitivity pushes it down to 0.269: the same ten points move it up more than twice as far as they move it down. The reason is the one behind the odds ratio column - only three in ten are exposed, so the false positives created by imperfect specificity come out of the large unexposed group, while the true positives missed by imperfect sensitivity can only come out of the smaller one. 0.239 is a row further down the sensitivity-only block, and it too sits below the truth, but using it to claim that misclassification always pulls the prevalence down ignores the false-positive route, which is what the lower half of the same table is doing. 0.313 is the row where specificity slips by two points only, and reading it as the answer for a ten-point drop is how the symmetric conclusion arises, which this entire column refutes. The practical use is direct: when a study reports an exposure prevalence that will not line up with a registry or a national survey, ask how exposure was measured - too high points at specificity, too low at sensitivity.
Chapters that use this method
Sources and licences
This page is original writing