Chi-square test and Fisher's exact test

What this method solves

Both variables are categorical (smoker or not, low birth weight or not) and the data collapse into a contingency table. The question is: is the pattern in this table far enough from what “the two variables are unrelated” would produce?

The chi-square test turns that into arithmetic. First compute how many people each cell should hold if the two variables were independent, by multiplying the margins back out:

Then add up the departures cell by cell:

That $E_{ij}$ in the denominator is what makes the statistic work: being five people out is nothing in a cell that expected 100, and a great deal in a cell that expected 6. Under the null hypothesis this statistic approximately follows a chi-square distribution with $(r-1)(c-1)$ degrees of freedom — note the word approximately, because every argument later on this page grows out of it.

The example here is MASS::birthwt: 189 mothers, asking whether maternal smoking and low birth weight (< 2500 g) are associated.

Running it yourself

library(MASS)
data(birthwt, package = "MASS")

birthwt$smoke_f <- factor(birthwt$smoke, levels = c(0, 1),
                          labels = c("Non-smoker", "Smoker"))
birthwt$low_f   <- factor(birthwt$low, levels = c(0, 1),
                          labels = c(">= 2500 g", "< 2500 g"))

tb <- table(birthwt$smoke_f, birthwt$low_f)
tb

cs <- chisq.test(tb)      # 2x2 carries Yates's correction by default
cs
cs$expected               # <- look at this before choosing a test
chisq.test(tb, correct = FALSE)

fisher.test(tb)           # exact test; also returns the OR and its CI

# Effect size: the chi-square statistic is not one.
# phi comes from the UNcorrected statistic, so do not reuse cs here.
sqrt(chisq.test(tb, correct = FALSE)$statistic / sum(tb))   # phi (Cramér's V for a 2x2)

# Paired categorical data needs a different method (same people, twice)
approval <- matrix(c(794, 86, 150, 570), nrow = 2)
mcnemar.test(approval)

import statsmodels.api as sm
import numpy as np
from scipy import stats
from statsmodels.stats.contingency_tables import mcnemar, Table2x2

bw = sm.datasets.get_rdataset("birthwt", "MASS").data
tb = np.array([[sum((bw.smoke == 0) & (bw.low == 0)), sum((bw.smoke == 0) & (bw.low == 1))],
               [sum((bw.smoke == 1) & (bw.low == 0)), sum((bw.smoke == 1) & (bw.low == 1))]])

chi2, p, dof, expected = stats.chi2_contingency(tb)      # correction=True by default
print(expected)                                          # <- expected counts first
stats.fisher_exact(tb)

t2 = Table2x2(tb)
print(t2.oddsratio, t2.oddsratio_confint())

mcnemar(np.array([[794, 150], [86, 570]]), exact=False, correction=True)

How to read this table

Among smokers 30/74 = 40.5% of infants were of low birth weight, against 29/115 = 25.2% among non-smokers.

Three effect measures, each with its own home:

• Risk ratio (RR): for cohort studies and trials. It is simply the ratio of two risks, and it is the easiest to interpret.
• Odds ratio (OR): a case-control study can only produce an OR, because the design fixes the number of cases and controls and no risk can be computed; logistic regression coefficients are ORs as well. The OR approximates the RR for rare outcomes, but for an outcome as common as the 25% here it clearly overstates it (2.02 against 1.61).
• Cramér’s V: for $r \times c$ tables, ranging from 0 to 1, with no direction. Useful for describing strength of association, hard to interpret clinically.

The expected-count-of-five line

The rule is usually remembered as “you cannot use a chi-square test if any expected count is below 5”. The original is more permissive: what Cochran recommended in the 1950s was that all expected counts be at least 1, and that no more than 20% of cells have an expected count below 5. For a 2×2 table, 20% is less than one cell, which is how the rule collapsed into “no cell below 5”.

Three things to keep hold of:

• It is a rule of thumb, not a theorem. There is no derivation behind the number 5; it comes from numerical checks on the quality of the chi-square approximation made in the 1950s.
• It governs the approximation, not the data. When expected counts are small, the distribution of the $\chi^2$ statistic is discrete and lumpy, and a continuous chi-square distribution approximates it badly. That has nothing to do with whether the data themselves are any good.
• It is about expected counts, not observed ones. A cell with zero observations is not automatically a violation, while a cell with an expected count of 4.2 violates the rule however many people happen to land in it. Which is why the first thing to do after chisq.test() is to print $expected.

The same dataset supplies a ready-made example. Restrict the analysis to the race == Black stratum (n = 26):

The smallest expected count falls to 4.2, breaking the rule. The three calculations give: uncorrected Pearson 0.149, Fisher’s exact 0.228, Yates-corrected 0.300 — on this table Yates’s correction is the most conservative of the three and the uncorrected Pearson the least. Note that Fisher is not guaranteed to be more conservative than Yates in every dataset; what the simulation in the next two sections shows is that both sit below the nominal level on average, not that one always follows the other. The confidence interval for the OR runs from 0.63 to 17.16. None of the three calculations reaches statistical significance here, but what actually needs saying is that this stratum has nowhere near enough patients to detect an association of even moderate size — the width of that confidence interval says so, and a p-value cannot.

When to use Fisher’s exact test

Fisher’s exact test makes no approximation. It holds both sets of margins fixed, enumerates every table those margins allow, and uses the hypergeometric distribution to compute the probability of a result at least this extreme. Because it never leans on a large-sample approximation, it remains valid however small the expected counts are.

Reach for it when:

• any expected count is below 5 (or the table total is simply small);
• the table contains a zero;
• the sample is small enough that the approximation worries you — a 2×2 Fisher test is essentially free on a modern computer, so there is no reason to accept an approximation to save computation.

The controversy about Yates’s correction

Yates’s continuity correction shrinks each cell’s departure by 0.5 before squaring it:

The intent is to compensate for approximating a discrete statistic with a continuous distribution. R’s chisq.test() applies it by default to 2×2 tables, which is why the same data can give you different p-values in R and in some other software (on this page: 0.026 against 0.040).

The dispute is that it overcorrects. Simulate all three tests in a setting where the null hypothesis is true (both groups carry a true risk of 20%, 3000 replicates per point) and see how often each one declares significance. The ideal value is 0.05:

The result is counter-intuitive but very consistent: the uncorrected Pearson chi-square already tracks 0.05 from about 15 per group upwards, while Yates and Fisher stay below the nominal value throughout — even at 200 per group they reach only 0.034 and 0.034. A type I error rate below the nominal level is not a bonus: it means you are giving away power you already paid for with your sample size.

This is one of the few places in statistics where the standard textbook advice and the simulation results do not line up. Pragmatically:

• State which version you used (uncorrected chi-square, Yates, or Fisher). Do not make the reader guess.
• When the sample is large enough (all expected counts at least 5), the uncorrected Pearson chi-square is a reasonable default.
• When the sample is small, Fisher is a safe choice — conservative, but conservative in the direction that does not manufacture false positives. At that point the real problem is usually a lack of power, not the choice of test.
• Whichever you pick, report the effect measure and its confidence interval. This whole argument moves the third decimal place of a p-value; it does not decide whether the OR is 2 or 20.

Paired categorical data needs McNemar

When the same people are measured twice (before and after treatment, two diagnostic tools compared, a matched case-control design), the data still look like a 2×2 table, but the cells mean something entirely different: each one now counts one person’s pair of results.

Take the classic example from R’s documentation (1600 respondents asked the same question at two time points):

The people on the diagonal — those who answered the same way twice — contribute nothing whatsoever to the question “did anything change”. All the information is in the two discordant cells: 150 people moved from approve to disapprove and 86 moved the other way. McNemar’s test uses only those two numbers:

which gives $\chi^2$ = 17.36, p = < 0.001. R’s mcnemar.test() applies a continuity correction by default — $(|b - c| - 1)^2 / (b + c)$ — so what it prints is $\chi^2$ = 16.82, p = < 0.001; the exact binomial version gives p = < 0.001. All three point the same way here, but the formula above is the uncorrected one, so check which version a paper is quoting before you try to reproduce it.

Hand the same table to a chi-square test as if the observations were independent and you get $\chi^2$ = 785.5, p = < 0.001 — an astronomically significant result, except that it tests a completely different null hypothesis (that a respondent’s first answer is independent of their second), and that hypothesis was absurd from the start: of course one person’s two answers are related. When a p-value comes back this extreme, asking “what is this test actually testing” usually finds the error faster than asking “how big is the difference”.

Common misuses

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B1-03-chi-square.R

Watch next

▶醫學統計 EP12 卡方檢定

▶醫學統計 EP15 RR vs OR

▶醫學統計 EP02 變數類型

Report a problem on this page