Interrupted time series and difference-in-differences

A policy went live. Now what?

Three settings in Taiwan produce this question shape more often than any other: National Health Insurance moving a drug or a test from out-of-pocket to reimbursed, a hospital rolling out an infection-control bundle, and the wholesale displacement of care-seeking during COVID. All three look the same on paper — one time point, one intervention, one monthly indicator, and then somebody asks how much it fell.

The cheapest answer is a before-after comparison: the mean before, the mean after, divided. On this dataset that is 873 against 571 casualties a month, a ratio of 0.654 (95% CI 0.642–0.665), and it looks tidy.

Its problem is not precision. It charges the whole of an already-moving trend to the policy. If the indicator was falling anyway, a before-after comparison books that fall as the policy’s achievement; if the indicator has a seasonal cycle and the intervention lands on the boundary between a high season and a low one, a before-after comparison can get the direction wrong.

The two methods on this page differ in exactly one thing: where the counterfactual comes from.

• Interrupted time series (ITS) extends the pre-intervention trend and treats it as where the indicator would have gone had the policy never arrived. The counterfactual is extrapolated.
• Difference-in-differences (DiD) finds a control series the policy cannot reach and uses its change over the same period as the counterfactual. The counterfactual is borrowed.

Neither conjures a control group from nothing. ITS bets that a trend continues; DiD bets that two series would have kept moving in parallel. This page places both bets on one dataset so you can see what the extra control series buys and what it costs.

The data on this page

datasets::Seatbelts ships with base R — data(Seatbelts) is the whole acquisition step. It holds 192 months of UK road-casualty data from 1969-01 to 1984-12, and it happens to carry both an intervention series and a natural control series:

• front — front-seat casualties. The 1983 compulsory seat belt law covered front seats.
• rear — rear-seat casualties. Same cars, same roads, same economy, but the law did not reach the back seat.

One more thing has to be handled first. Distance driven rose from 14463 to 18890 per month across the transition, an increase of 30.6%. Leave it out of the model and more driving, therefore more collisions gets folded into the effect of the law. Every model on this page therefore carries offset(log(kms)), which turns the outcome from casualties per month into casualties per unit distance — the offset described on the Poisson regression page.

The empirical-calibration section at the end of this page uses simulated data rather than Seatbelts; that section says so again at its start.

Interrupted time series: extending the pre-intervention trend

An ITS is a regression with two extra terms. Written as a Poisson model:

• $t$ is the month index and $\beta_1$ is the pre-intervention secular trend — the term a before-after comparison lacks.
• $\mathrm{law}_t$ is an indicator that switches on at the intervention, and $\beta_2$ is the level change: the step at the moment of the intervention.
• $\mathrm{tsl}_t$ counts months elapsed since the intervention and is 0 before it, so $\beta_3$ is the slope change: how much further the indicator moves each month relative to the old trend. In the code on this page tsl is already 1 in the intervention month itself, so $\exp(\beta_2)$ is the step read at $\mathrm{tsl} = 0$ while the ratio actually observed in the first post-intervention month is $\exp(\beta_2 + \beta_3)$. Reporting $\exp(\beta_2)$ as the level change is the usual convention, but the coding moves a small amount between the step and the slope, so the Methods have to say which one was used.
• $X_t$ is a time-varying covariate; here it is petrol price.

Level and slope answer two different clinical questions. A level change asks what changed the moment this happened — regulations, reimbursement rules and other switch-like interventions usually look like this. A slope change asks what this did to the trajectory afterwards — education, training and anything that diffuses gradually usually looks like that. One intervention can produce either, or both.

The fitted level change is RR 0.680 (95% CI 0.654–0.707, p < 0.0001) — at the moment the law took effect, front-seat casualties fell about 32.0% relative to the counterfactual. The slope change is RR 1.0123 per month (95% CI 1.010–1.015), which is above 1: the monthly trajectory after the law tilts upward relative to the one before it.

A before-after comparison is not an interrupted time series

Put the model beside the before-after ratio from the top of this page:

The two numbers are not far apart, which is luck rather than reassurance — the pre-intervention trend in this series happens to be shallow. The real difference is in the last column. A before-after ratio cannot tell you whether this was a step or a change of direction, and that is precisely what clinical readers want to know. A policy that knocks an indicator down one notch and then lets it climb again calls for different follow-up from a policy that bends the whole trajectory.

There is a second, invisible cost: a before-after ratio cannot be caught out by its own figure. An ITS plot carries a counterfactual line the reader can judge for plausibility. Two means offer nothing to disagree with.

Whatever you fail to subtract gets charged to the policy

This is the most practical section on the page. Same level change, adjustment terms removed one at a time:

The first thing worth noticing runs against intuition. Dropping petrol price and the distance offset moves the estimate only between 0.660 and 0.682, a spread of 3.4%, with heavily overlapping intervals. On this dataset, whether you adjusted for petrol price cannot change the conclusion. That point was originally planned as this page’s headline lesson and the numbers overturned it once the script actually ran.

What does change the conclusion is seasonality. Add calendar month and the level change moves from 0.680 to 0.739 (95% CI 0.711–0.768) — an interval that does not overlap the fully adjusted model’s 0.654–0.707. AIC falls at the same time from 5204 to 2941, a drop of 2263, which says the seasonal cycle was the largest piece of structure the other models were ignoring.

The reason is not mysterious. Road casualties have a strong within-year cycle — short winter days, worse weather, more casualties — and the law arrived in 1983-02, so the 23 post-intervention months do not cover the same mix of calendar months as the 169 months before them. Without a calendar-month term, that seasonal difference is charged to the law.

The placebo series: the line the law could not reach

The great weakness of an ITS is that it has no control group — its control is the same population’s own past. So anything else that changed the indicator in the same month (another policy, an oil shock, a change of coding rules) is indistinguishable from the intervention.

Seatbelts offers a clean check: run exactly the same model on rear. Rear-seat casualties share the season, the petrol price and the distance driven, but the law did not cover them. That makes rear-seat casualties a negative control outcome: if the model is right, no level change should appear here.

The fully adjusted placebo series gives a level RR of 0.963 with a 95% CI of 0.919–1.008, which crosses 1: not statistically significant. That row is the most valuable line on this page. The law really did reach only the front seats, and the reader can now see that as evidence in a table rather than take it as the author’s assertion.

The wording has to stay precise. That row says no level change was detected in these data, not rear-seat casualties did not change. The lower confidence limit reaches 0.919, so a genuine reduction of roughly 8.1% remains compatible with the data. Claiming that the rear seats were unaffected would require a pre-specified equivalence margin, not an interval that happens to cross 1.

The third row carries a bonus lesson: the same placebo series turns significant when it is left unadjusted (RR 0.953, 95% CI 0.910–0.998, p = 0.0400). A placebo test is itself sensitive to model specification. It is not free insurance — a badly specified placebo test hands out false reassurance as readily as false alarms.

The full taxonomy of negative controls — what a negative control exposure and a negative control outcome can each refute, and how to read one that is not null — is on the instrumental variables page. The last section here takes the same idea and scales it to dozens of outcomes.

Difference-in-differences: borrowing the counterfactual

With rear in hand, a placebo test is not the only thing available. Stack the two series into long format, add a group indicator, and the counterfactual switches from extrapolated to borrowed:

$\alpha_3$, the interaction, is the DiD estimate: the treated group’s change across the intervention, minus the control group’s change over the same period. Anything that hit both groups — petrol, weather, the economy, a change of coding rules — cancels in the subtraction. That cancellation is what the control series buys.

All three coefficients are worth unpacking, because papers routinely print only the interaction row and readers never learn what the other two were doing:

The group main effect of 2.182 says front-seat casualties were already more than twice rear-seat casualties before the law. DiD does not require the two groups to be at the same level, only that their trends would have moved in parallel. That is the sharpest difference between DiD and the matching and weighting methods, and the point most often misread.

Parallel trends: the one identifying assumption DiD needs

DiD trades extrapolation for a control series, and the price is a new assumption: parallel trends — that absent the policy, the gap between the two series would have stayed put.

The assumption cannot be verified, because it describes a world that did not happen. What can be done is a pre-trend check: split the pre-intervention period into bins, estimate a treated-versus-control value in each, and see whether they all sit near 1. That is an event-study plot.

The result is unambiguous, and for this page it is good news: 10 of the 13 non-baseline years have intervals that exclude 1, and the joint test that all of them are zero gives chi-squared 129.6 on 13 degrees of freedom, p < 0.0001. Across 1969 to 1972 the front-to-rear ratio ran at 1.143 to 1.174 times its baseline value; only after 1975 does it settle.

So the DiD estimate of 0.642 on this page cannot be presented as more trustworthy than the ITS estimate of 0.680. The honest formulation is that the pre-trend check refutes parallel trends over the full period and only the later stretch is defensible. A DiD here should either restrict the sample to that stretch or move to a specification that allows differential trends — and either decision belongs in the Methods.

How to compute it

All four models are base R glm() with no extra packages.

data(Seatbelts)
d <- as.data.frame(Seatbelts)
d$time <- seq_len(nrow(d))
d$mon  <- (d$time - 1) %% 12 + 1
d$month <- factor(d$mon)

# Derive the intervention index from the data; never hard-code a month. The
# effective date in a policy circular and the month the database actually
# turns are routinely one or two months apart.
law_idx <- which(d$law == 1)[1]

# Months elapsed since the intervention, 0 before it. This coding makes the
# `law` coefficient the step AT the transition and `tsl` the change in slope
# after it; shifting the origin by one month moves value between the two.
d$tsl <- pmax(0, d$time - law_idx + 1)

# --- 1. ITS on the front seats --------------------------------------------
its <- glm(front ~ time + law + tsl + PetrolPrice + offset(log(kms)),
           data = d, family = poisson())
exp(cbind(RR = coef(its), confint.default(its)))["law", ]

# --- 2. Seasonality: add calendar month. The term that moves the answer ----
its_season <- glm(front ~ time + law + tsl + PetrolPrice + month +
                    offset(log(kms)), data = d, family = poisson())
exp(coef(its_season)["law"])

# --- 3. Placebo series: the identical model on the rear seats -------------
# The law did not cover rear seats. A level change here means the model is
# picking up something other than the law.
placebo <- glm(rear ~ time + law + tsl + PetrolPrice + offset(log(kms)),
               data = d, family = poisson())
exp(cbind(RR = coef(placebo), confint.default(placebo)))["law", ]

# --- 4. DiD: stack the two series into long format ------------------------
long <- rbind(
  data.frame(y = d$front, grp = 1L, d[, c("time", "law", "kms", "PetrolPrice")]),
  data.frame(y = d$rear,  grp = 0L, d[, c("time", "law", "kms", "PetrolPrice")])
)
did <- glm(y ~ grp * law + time + PetrolPrice + offset(log(kms)),
           data = long, family = poisson())
exp(cbind(RR = coef(did), confint.default(did)))["grp:law", ]

# --- 5. The pre-trend check (event study) ---------------------------------
# Pre-intervention data only, one grp x year interaction per year.
pre <- subset(long, law == 0 & time < law_idx)
pre$year <- 1969 + (pre$time - 1) %/% 12
pre$yf <- relevel(factor(pre$year), ref = "1982")   # last complete pre-law year
es <- glm(y ~ grp + yf + grp:yf + PetrolPrice + offset(log(kms)),
          data = pre, family = poisson())

# Joint test that every pre-intervention interaction is zero. Rejecting it
# refutes parallel trends; failing to reject does NOT establish them.
es_null <- glm(y ~ grp + yf + PetrolPrice + offset(log(kms)),
               data = pre, family = poisson())
anova(es_null, es, test = "LRT")

import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

d = sm.datasets.get_rdataset("Seatbelts", "datasets").data.copy()
d["time"] = np.arange(1, len(d) + 1)
d["mon"] = (d["time"] - 1) % 12 + 1

law_idx = int(d.loc[d["law"] == 1, "time"].iloc[0])
d["tsl"] = np.maximum(0, d["time"] - law_idx + 1)

# In statsmodels the offset is an argument to the fit, not part of the formula.
its = smf.glm("front ~ time + law + tsl + PetrolPrice",
              data=d, family=sm.families.Poisson(),
              offset=np.log(d["kms"])).fit()
print(np.exp(its.params["law"]), np.exp(its.conf_int().loc["law"]))

# Seasonality: C(mon) is patsy's factor syntax, equivalent to R's factor()
its_season = smf.glm("front ~ time + law + tsl + PetrolPrice + C(mon)",
                     data=d, family=sm.families.Poisson(),
                     offset=np.log(d["kms"])).fit()

# DiD in long format; grp * law means the same thing in patsy as in R
long = pd.concat([
    d.assign(y=d["front"], grp=1),
    d.assign(y=d["rear"], grp=0),
], ignore_index=True)
did = smf.glm("y ~ grp * law + time + PetrolPrice",
              data=long, family=sm.families.Poisson(),
              offset=np.log(long["kms"])).fit()
print(np.exp(did.params["grp:law"]))

ITS and DiD side by side: what a control series buys

The two estimates differ by 0.038, with the DiD estimate the further from 1 — that is, the larger reduction (0.642 against 0.680). The source of the gap is visible: the DiD period main effect of 1.190 is above 1, so the borrowed counterfactual says the indicator should have risen over that window, and the front seats therefore have to fall further relative to it.

That does not make DiD the better answer here. It means the two methods disagree about what would have happened, and on this dataset the assumption behind the DiD answer failed its pre-test. What the control series buys is cancellation of common shocks; what it costs is a requirement that the relative trajectory of the two series be stable, and here it is not.

Common practice is to report both — one as the primary analysis and the other as a sensitivity analysis — and to say explicitly that their assumptions differ, so that agreement is support and disagreement is something to explain. Reporting only one, without mentioning the other, leaves the reader unable to judge how sensitive the conclusion is to the choice of method.

Negative controls and empirical calibration

A single negative control outcome — rear, above — answers whether there is bias. A panel of them answers a harder question: how large the bias is and in which direction.

The cloud in the left panel is the whole argument. All 39 of those outcomes have a true RR of 1, so an unbiased analysis would scatter them either side of 1. Instead the cloud sits to the right, with a fitted null-distribution centre of RR 1.316 — off 1 by 31.6% — and a spread (tau, on the log scale) of 0.117.

One number is sharper still: 34 of the 39 negative controls have a 95% CI that excludes 1. Read conventionally, those 34 significant associations would all be believed, and every one of them has a true RR of 1. Intervals labelled 95% achieve actual coverage of 12.8%.

The calibration itself is simple: on the log scale, subtract the fitted offset mu from every estimate and replace the standard error with the square root of the original variance plus tau squared. Three things then happen at once:

1. Coverage among the negative controls returns from 12.8% to 94.9%, which is what the nominal level promised.
2. The outcome of interest moves from 2.99 (95% CI 2.72–3.29, which misses the true value of 2.0) to 2.27 (95% CI 1.77–2.91, which covers it).
3. The price is an interval 2.6 times wider. Calibration does not make you more certain. It makes your certainty honest.

Questions to ask of a policy-evaluation paper

When a title says an indicator fell by so much after a policy, four questions dispose of most of the doubt:

1. Who is the control group? An ITS uses the same population’s own past; a DiD uses another group over the same period. A paper that does not say which cannot be evaluated. If it is a DiD, the reason the control group is beyond the policy’s reach has to be argued.
2. Was the parallel-trends assumption checked? Is there an event-study plot, or only a sentence saying the check was not significant? The second reads low power as proof.
3. Were seasonality and the secular trend subtracted? On this page, one calendar-month term moved the level change from 0.680 to 0.739, with intervals that do not overlap.
4. Is there a negative control or placebo series? An outcome or a group the policy could not reach is the cheapest and most convincing check available.

Two more details that get skipped: how the intervention month was defined (the circular’s date and the database’s turning point are routinely a month or two apart), and whether autocorrelation was handled (if not, the standard errors are optimistic and the intervals too narrow).

Common misuses

How this page relates to the others

• Where you meet it clinically — see database and claims-data studies. Any trend analysis across years has to check the reimbursement policy history, and a policy change is itself something the methods on this page can estimate.
• The underlying model — see Poisson regression and incidence rates. Every model here is a Poisson GLM with an offset, and each exponentiated coefficient is a rate ratio.
• Another design where people serve as their own controls — see self-controlled designs. It places the control window inside the same person’s own follow-up, which is the same family of idea as using a whole series’ past.
• The language of counterfactuals and identifying assumptions — see causal diagrams and the backdoor criterion and target trial emulation. Parallel trends is a counterfactual assumption, unverifiable in exactly the way exchangeability is.
• The full taxonomy of negative controls — see instrumental variables.
• Time-related traps — see time-varying covariates. Policy-evaluation data often carry a time-varying exposure and time-varying confounding at once.

Rerunning every number on this page

/opt/homebrew/bin/Rscript figures/scripts/B6-10-its-did.R

Report a problem on this page