BasicIndependently reviewed, not yet spot-checked by a human

Linear regression

What "holding the other variables constant" actually holds constant, why a high R² does not mean the model is useful, why linearity in a continuous predictor is an assumption rather than a fact, and which numbers change when you switch the reference group — and which do not.

What this model is for

A large share of clinical questions have this shape: the outcome is a continuous number — birth weight, systolic blood pressure, eGFR, length of stay — and you want to know how strongly some factor is related to it, and specifically how much of that relationship is left after the other factors are taken out.

The t-test and ANOVA compare means across two or more groups, but only one grouping variable at a time, and that variable has to be categorical. Linear regression generalises the idea: put several variables in at once, continuous or categorical, and get a coefficient for each one, where the coefficient means “with the other variables held constant, a one-unit increase in this variable moves the outcome by this much on average”.

That sentence is the spine of this page. It sounds straightforward. It hides three traps, and the fourth section is about them.

The example on this page

MASS::birthwt comes from a 1986 study of low birth weight at a hospital in Springfield, Massachusetts. It has 189 mothers, with age, weight at the last menstrual period, race, smoking during pregnancy, history of hypertension, uterine irritability and more; the outcome is the newborn’s weight in grams. Every regression chapter on this site uses it, for two reasons: it is a standard teaching dataset available everywhere, and it is the designated replacement after the Pima diabetes data were banned site-wide.

Mean birth weight is 2,945 g (SD 729 g), and 74 of the mothers smoked during pregnancy.

library(MASS)
data(birthwt, package = "MASS")

bw <- birthwt
bw$race_f  <- factor(bw$race,  levels = 1:3, labels = c("White", "Black", "Other"))
bw$smoke_f <- factor(bw$smoke, levels = 0:1, labels = c("No", "Yes"))

# One predictor
fit_s <- lm(bwt ~ lwt, data = bw)
summary(fit_s)

# Several predictors
fit <- lm(bwt ~ age + lwt + smoke_f + race_f + ht + ui, data = bw)
summary(fit)          # coefficients, SE, t, p, R-squared, adjusted R-squared, overall F
confint(fit)          # 95% CI -- this is what a paper reports, not the p-value alone

# Change the reference group: the coefficients move, the model does not
bw$race_o <- relevel(bw$race_f, ref = "Other")
fit_o <- lm(bwt ~ age + lwt + smoke_f + race_o + ht + ui, data = bw)
all.equal(fitted(fit), fitted(fit_o))     # TRUE

# Linearity is an assumption; test it
anova(fit, update(fit, . ~ . + I(age^2)))

Verified with R 4.6.0 and MASS 7.3.65. birthwt ships with MASS, so there is nothing to download.

import statsmodels.api as sm
import statsmodels.formula.api as smf

bw = sm.datasets.get_rdataset("birthwt", "MASS").data
bw["race_f"]  = bw["race"].map({1: "White", 2: "Black", 3: "Other"})
bw["smoke_f"] = bw["smoke"].map({0: "No", 1: "Yes"})

fit = smf.ols("bwt ~ age + lwt + C(smoke_f) + C(race_f) + ht + ui", data=bw).fit()
print(fit.summary())
print(fit.conf_int())

# The reference group is set with Treatment(reference=...)
fit_o = smf.ols(
    'bwt ~ age + lwt + C(smoke_f) + C(race_f, Treatment(reference="Other")) + ht + ui',
    data=bw).fit()

# Linearity check
fit_q = smf.ols("bwt ~ age + I(age**2) + lwt + C(smoke_f) + C(race_f) + ht + ui",
                data=bw).fit()
print(fit.compare_f_test(fit_q))

The statsmodels OLS summary maps almost column-for-column onto R's, and the formula API keeps the model syntax the same too.

Start with one predictor

Ordinary least squares (OLS) does exactly one thing: it finds the line that minimises the sum of the squared vertical distances from the points to the line.

\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x, \qquad (\hat{\beta}_0, \hat{\beta}_1) = \arg\min \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i)^2

Using the mother’s weight (lwt, in pounds) to predict birth weight, the slope is 4.43 g per pound (95% CI 1.05–7.81, p 0.011). Per 10 pounds it is easier to picture: 44.3 g. A coefficient on a continuous variable cannot be interpreted without its unit, which is exactly the situation with the hazard ratio in a Cox model.

Two scatter plots. On the left, newborn birth weight against the mother's weight, with the least-squares line and a shaded confidence band. On the right, birth weight against the mother's age, with two fitted curves drawn together: a straight line and a quadratic curve that turns upward at both ends. — Left: the least-squares fit with one predictor; the shaded band is the 95% confidence band for the regression line. Right: the same data with age as the predictor, straight line and quadratic curve overlaid — the sixth section explains why the curve is statistically supported and why that support is fragile.Plotting script `figures/scripts/B2-01-linear-regression.R`

The R² of that line is only 0.034: the mother’s weight accounts for less than a twentieth of the variation in birth weight. That does not make the association spurious — the confidence interval for the slope does not cross 0, so there is statistical evidence for the association itself. What it says is that almost all of the variation in birth weight comes from somewhere else. The fifth section returns to this.

Reading a coefficient, and what “holding the others constant” holds

The full table with six predictors in the model (n = 189):

Variable	Coefficient (g)	95% CI	p
Age, per year	-4.7	-23.1–13.8	0.617
Mother's weight, per lb	4.4	1.0–7.8	0.011
Smoking during pregnancy	-360.7	-566.0–-155.4	< 0.001
Race: Black vs White	-490.6	-785.0–-196.3	0.001
Race: Other vs White	-356.6	-581.3–-132.0	0.002
History of hypertension	-590.0	-985.2–-194.8	0.004
Uterine irritability	-528.5	-795.1–-262.0	< 0.001

The smoking row reads: “among mothers of the same age, the same weight, the same race, the same history of hypertension and the same uterine irritability, those who smoked during pregnancy had newborns 361 g lighter on average.”

Compare the two groups’ raw means with no adjustment at all and the smoking group is lower by 284 g (95% CI 72.8–494.8 g lower). Adjustment made the gap larger, which is a common direction here: race and maternal weight are related to smoking in these data, so the crude comparison mixed their effects in and they partly cancelled out.

Three traps inside “holding the other variables constant”

Only the variables that are in the model are held. Anything you left out is not held at all. The phrase has never meant “all confounding removed”; it means “the ones I listed have been removed”. Which variables belong in the model is a question for a causal diagram (DAG), not for p-values.
It is a statistical condition, not an intervention you could perform. “Hold weight fixed and change race” has no real-world counterpart. The coefficient is a conditional contrast within the data; reading it as “if I got this patient to quit smoking, her baby would be 361 g heavier” is a jump from association to causation, and that jump needs a reason in the study design, not in the model output.
Change the adjustment set and the coefficient changes. Two papers reporting different coefficients may differ only in what they put in the model. A coefficient must always be read together with the model that produced it — exactly as with the hazard ratio from a Cox model.

There is one more that gets overlooked: do not adjust for a mediator. If smoking affects birth weight by reducing placental blood flow, then adjusting for placental blood flow subtracts part of the very effect you were trying to estimate.

R² and adjusted R²

R² is the share of the total variation the model accounts for:

R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}

For the multivariable model R² is 0.241 and adjusted R² is 0.212. The difference between them:

R² never goes down when you add a variable, whatever the variable is — including a column of dice rolls.
Adjusted R² charges a penalty for the number of parameters: $1 - (1-R^2)\frac{n-1}{n-p-1}$ . If an added variable does not buy enough explanation to cover the penalty, it falls.

Running the experiment 200 times makes it obvious. At each step one more column of pure standard-normal noise goes into the model:

A line chart. The horizontal axis is the number of pure-noise variables added, from 0 to 20. The red line, R squared, rises steadily throughout. The blue line, adjusted R squared, stays roughly flat and drifts slightly downward. — Each point is the average over 200 random draws. The added variables have nothing to do with birth weight, and R² still climbs from 0.241 to 0.325; adjusted R² goes from 0.212 to 0.212 and is not fooled.Plotting script `figures/scripts/B2-01-linear-regression.R`

A high R² does not mean the model is useful, and a low R² does not mean it is useless. Both misreadings are common:

A high R² may just mean you put the outcome back in. Predict “lowest haemoglobin during admission” from “haemoglobin at discharge” and R² will look beautiful, but that is not prediction — it is the same thing measured twice. In clinical prediction models this is called data leakage.
A low R² does not refute an association. The model on this page accounts for 24.1% of the variation, yet the confidence intervals for smoking, hypertension and uterine irritability all stay clear of 0. “This factor matters” and “this model can predict an individual patient” are two different questions, and R² answers the second. It is the same phenomenon as a Cox model with a tiny p-value and a merely moderate C-index.
R² is not comparable across different outcomes. Its denominator is the variation of that particular outcome.

The overall F test (8.23, df = 7 and 181, p < 0.001) asks whether all the coefficients are simultaneously 0. It is the counterpart of the global likelihood ratio test in a Cox model.

Linearity is an assumption, not a given

“Each one-unit increase moves the outcome by a fixed amount” is itself an assumption, and it is the one most likely to fail. It can also be tested: add a quadratic term and see whether the fit improves.

Variable	p for the linear term	F after adding the squared term	p for the squared term	Conclusion
Age	0.617	5.68	0.018	A straight line is not enough to describe it
Mother’s weight	0.011	0.11	0.741	No departure from linearity detected

The age row is the single most useful thing on this page. The linear age term is nowhere near significant (p 0.617), yet adding a squared age term produces a statistically detectable improvement in fit (p 0.018).

Reading only the first column and concluding “age is unrelated to birth weight” turns “the straight-line version of the relationship was not supported” into “there is no relationship”. Those are not the same statement.

But the support for that curve is fragile, and saying so is worth more than keeping a tidy conclusion.

The fitted parabola opens upward with its minimum near age 26.0 (this vertex comes from the six-variable multivariable model; the figure earlier on this page shows the single-predictor version, whose vertex sits further left). In other words the model claims that both the youngest and the oldest mothers have heavier newborns. Clinically that does not hold up.

The reason is in how the data are distributed. Age here ranges from 14 to 45, and only 3 mothers are over 35. Polynomials have no locality — those three points at the right-hand end bend the entire curve. Drop them and refit, and the p-value for the squared term becomes 0.373: the evidence for a departure from linearity is gone.

So the practical advice has two layers. Do check linearity rather than assuming it, but check it with restricted cubic splines rather than polynomials — a spline bends piecewise, so a handful of points in a sparse region cannot lever the whole curve. For how a spline is actually fitted, where the knots go, and how to read the curve and its confidence band, see non-linearity and restricted cubic splines. For how influence is measured, and how three observations come to dominate a model fitted on 189 of them, see regression diagnostics.

Changing the reference group: the coefficients move, the model does not

A categorical variable enters the model as dummy variables: $k$ categories produce $k-1$ columns, and the category left out as the baseline is the reference group. R takes the factor’s first level by default.

Same model, with race’s reference group switched from White to Other:

Reference	Contrast	Coefficient (g)	95% CI	p
White	Black vs White	-490.6	-785.0–-196.3	0.001
White	Other vs White	-356.6	-581.3–-132.0	0.002
Other	White vs Other	356.6	132.0–581.3	0.002
Other	Black vs Other	-134.0	-446.9–178.8	0.399

What changed across those four rows is not how much evidence there is, but which contrasts you are able to report. White versus Other appears in both versions with the sign flipped and the confidence intervals mirrored, and the p-value is identical (0.002 against 0.002). The real difference is the Black versus Other row (p 0.399), which only the second version reports directly; the first version’s table does not contain it at all. Meanwhile:

R²: 0.241 against 0.241
Residual standard deviation: 647.27 against 647.27
Fitted value for every mother: identical

It is the same model. Changing the reference group only re-describes the same plane in a different set of coordinates, the way converting Celsius to Fahrenheit changes every number without changing how warm the room is.

And a contrast that “does not appear” is still in the model. Black versus White is estimated directly in the first version, at -490.6 g; in the second version you recover it by subtraction, and you get -490.6 g. The same value.

How to choose a reference group in practice:

Start from what is clinically interpretable — placebo, standard of care, unexposed. Then the sign of each coefficient matches the reader’s intuition. This consideration comes first.
Group size comes second. Using the largest group as the reference usually makes the direct “each group versus reference” contrasts more precise. But note that this is not a way to shrink the standard error of a given contrast — re-expressing the same contrast only flips its sign, leaving the standard error and p-value untouched (White versus Other in the table above is the demonstration). What changes is which contrasts show up in the table.
Do not switch reference groups to make a p-value look better. Switching does not change the model, but it does change which contrasts you report; trying several and reporting the one you liked is undeclared multiple comparison. To compare all categories, use the overall (type III) F test, or report all pairwise contrasts with an appropriate correction.

Common misuses

Misuse	Why it is wrong
Fitting a binary outcome with linear regression	Fitted values escape the 0–1 range; use logistic regression
Fitting counts or rates with linear regression	The variance grows with the mean and fitted values can go negative; use Poisson or negative binomial
Fitting censored survival times with linear regression	Censored data are neither missing values nor event times; see censoring
Reporting a coefficient on a continuous variable without its unit	Per year, per decade and per standard deviation are entirely different numbers
Reading “holding the others constant” as “all confounding removed”	Only the variables in the model are held
Reading a coefficient as what would happen after an intervention	That is a causal claim and needs a reason in the design
Adjusting for a mediator as if it were a confounder	It subtracts part of the effect you wanted to estimate
Using R² to judge whether the model is “right”	R² measures how much variation is explained, not whether the model is correctly specified
Comparing R² between models with different outcomes	Different denominators, not comparable
Treating a rise in R² after adding a variable as improvement	R² rises for anything; look at adjusted R² at minimum, and external validation ideally
Declaring a variable irrelevant because its linear term is not significant	What failed to find support is the straight-line version, not the relationship
Handling non-linearity with high-order polynomials	They swing wildly where the data are sparse; use restricted cubic splines
Trying several reference groups and reporting the best-looking one	Undeclared multiple comparison
Writing a non-significant result as “the two do not differ”	Write “no difference was detected”, and give the confidence interval so the reader sees the range still compatible with the data

Reproducing every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B2-01-linear-regression.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

How should the "smoking during pregnancy" row of this multivariable coefficient table be read?

Show the answer and why

Correct answer: -360.7 grams - smokers' babies weigh this much less on average than non-smokers'

Smoking enters this model as a binary variable, so its coefficient of -360.7 is a difference against the reference group, not a slope per unit. -490.6 is in fact the race row, and 4.4 is the slope per pound of maternal weight. A single table mixes per-unit rows with versus-reference rows, and reading the second kind as the first produces sentences like "several hundred grams per extra unit smoked", which mean nothing. Before reading any coefficient, establish whether its variable is continuous or categorical.

A batch of purely random variables is added to the existing multivariable model. Which statement is right?

Show the answer and why

Correct answer: R-squared rises to 0.325 - it can only increase with more variables, so it cannot compare models of different sizes

R-squared goes from 0.241 to 0.325, and the added variables are pure random numbers. That is not a coincidence but arithmetic: one more variable can only shrink or hold the residual sum of squares, so R-squared can only rise. 0.212 is the adjusted R-squared after the addition, and it edges down instead - adjusted R-squared charges for the extra parameters, which is the whole reason it exists and why it, not R-squared, is what compares models of different sizes.

An overall F test is printed under the multivariable model. What question does it answer?

Show the answer and why

Correct answer: Numerator df 7 - whether these variables together explain anything

The numerator degrees of freedom equal the number of variables in the model, 7 here, and the F test asks exactly whether those 7 together explain anything. 181 is the denominator df and 189 is the sample size; the three sit within a line or two of each other and mean nothing alike. The test says nothing about any single variable - a model with a significant F test and not one significant coefficient is entirely possible, and usually looks like collinearity.

Chapters that use this method

Watch next

ENStatQuest with Josh Starmer· 9 minNine minutes on what least squares is actually doing. Watch it before the third section and "minimise the sum of squared residuals" stops being a slogan.

繁中EDMAN MURMURS· 13 minIn Mandarin, from a clinician's point of view. Useful if you need the Chinese terminology for coefficients and R² alongside the English.

繁中臺大開放式課程 NTU OCW· 22 minIn Mandarin. Lists the assumptions of simple linear regression one by one — the same list the sixth section of this page is checking.

繁中NYCU OCW· 143 minIn Mandarin. Two hours of full derivation. Watch it if you want the algebra behind the formulas; skip it if your goal is reading papers.

Sources and licences

This page is original writing