E-values and sensitivity analysis for unmeasured confounding

What this page is for

There are two moments where an E-value lands in front of you.

The first is reading a paper. A database study reports an association, and the last paragraph of the discussion says “we calculated an E-value of 1.9” and then stops. Is that large or small? What exactly is it guaranteeing? Without an answer, the sentence is decoration that happens to look technical.

The second is being reviewed. Submit a study built on Taiwan’s National Health Insurance Research Database and one of the most frequent reviewer comments you will get is a request for a sensitivity analysis for unmeasured confounding. What is being asked for is almost always an E-value, because it is cheap: no extra data, no validation subsample, three lines of code.

The reason it exists is that observational research has a hole that never closes. Propensity score matching and weighting can only handle the covariates you actually measured; a DAG helps you see which back-door paths need blocking, but it cannot block a path you have no variable for. No better model fixes this. All you can do is quantify it honestly.

The E-value inverts the question: instead of asking whether unmeasured confounding exists, ask how strong an unmeasured confounder would have to be to explain the whole association away. The first question cannot be answered. The second is a number.

The definition: the bias factor and the E-value

Suppose there is an unmeasured confounder $U$. It has two association strengths:

• $RR_{EU}$, the prevalence ratio of $U$ comparing exposed to unexposed — the confounder’s association with the exposure
• $RR_{UD}$, the risk ratio of $U$ for the outcome within levels of the exposure — the confounder’s association with the outcome

VanderWeele and Ding proved something rather elegant: the maximum amount by which those two associations can inflate an observed risk ratio has a closed form that involves nothing else. It is called the bias factor.

For the observed risk ratio $RR_{obs}$ to be explained away entirely you need $B \geq RR_{obs}$. Impose the extra condition that the two associations are equally strong and solve, and out drops the E-value:

If $RR_{obs}$ is below 1 — a protective association — invert it first. That is the whole of the mathematics, which is why this page installs no package: the formula is shorter than the documentation of any library that implements it.

The contour geometry is worth more than the formula

Solve $B = RR_{obs}$ for $RR_{UD}$:

That is a hyperbola with two asymptotes: a vertical one at $RR_{EU} = RR_{obs}$ and a horizontal one at $RR_{UD} = RR_{obs}$. Both asymptotes say the same thing, and it is the single sentence worth carrying away from this page:

To explain away an observed association, the unmeasured confounder’s association with the exposure and its association with the outcome must both exceed the observed association itself. One strong side is not enough. However far you push the other, the curve only approaches its asymptote and never reaches it.

The E-value is one particular point on that curve: the point where the two associations are equal, which is where the curve crosses the diagonal. That point is not chosen because it carries some special causal meaning, but because it is the cheapest combination on the curve — move along the curve in either direction and the other association has to grow. So the E-value is a lower bound: any confounder capable of explaining the association away has at least one side at least this strong.

Two E-values, two different questions

The point estimate and the confidence limit each get their own E-value, and both belong in the report. They ask different things:

• The E-value for the point estimate asks: how strong a confounder would move the effect estimate back to the null?
• The E-value for the confidence limit nearer the null asks: how strong a confounder would make this result unable to exclude the null?

The second is the answer to “could confounding overturn this finding”. The reasoning is the same as on the effect size page: a point estimate carries no uncertainty of its own, and to overturn a conclusion you do not have to drag the point estimate to the null, only drag the interval far enough to touch it.

The computation: take the limit of the interval closer to the null and put it through the same formula. When the interval already covers the null, the answer is 1 — no confounding of any strength is needed, because the data are already compatible with no effect.

Reporting only the point estimate’s E-value is common in the literature, and it always makes a result look harder to overturn than it is. The two worked examples below show each of these situations in turn.

The first step is always converting the effect measure to a risk ratio

The formula takes a risk ratio. A hazard ratio is not one and neither is an odds ratio, so feeding either in unconverted gives the wrong answer — and the error has a fixed direction. It is too large, which makes the result look harder to explain away. This is the step most often skipped.

For a hazard ratio there are two routes, and which one is right depends on how rare the outcome is. When the outcome is rare,

is good enough. When it is not rare, VanderWeele’s square-root form is required:

The conventional dividing line is an event proportion of 15%. The blood_storage cohort used in example two has 47 events among 287 complete cases, an event proportion of 16.4%, which is above that line — so the square-root form is the correct route there.

An odds ratio takes a third formula, and it needs something the hazard ratio route does not:

Here $p_0$ is the baseline risk among the unexposed. As $p_0 \to 0$ the risk ratio approaches the odds ratio, which is the rare-outcome approximation; and when the odds ratio exceeds 1, any $p_0 > 0$ makes the risk ratio smaller than the odds ratio. That property becomes the deciding argument in example one.

Example one: an odds ratio from a case-control study, and two E-values far apart

Example one reuses the case-control chapter’s datasets::infert: 1 case : 2 controls, matched on age, parity and education, 248 records, of which 83 are cases and 165 controls. The exposure is a history of induced abortion and the outcome is secondary infertility. That page’s conditional logistic regression, further adjusted for spontaneous abortions, gives:

Why only the rare-outcome approximation is available here

Converting that odds ratio needs a baseline risk $p_0$, and a case-control study cannot give you one. That fact is itself one of the lessons on this page.

Cases make up 33.5% of the sample, which looks a great deal like a risk. It is not: the ratio of cases to controls was chosen by the design (1 to 2 here), and sampling at a different ratio would change the number. Using it as $p_0$ would be treating a design parameter as a population risk, which is inventing data. The stats file records this as two flags — baselineRiskEstimable is false and caseProportionIsNotP0 is true — so that nobody rerunning the script later picks that proportion up and uses it.

So the rare-outcome approximation is used, which is RR = OR. It is chosen not for convenience but because it is the most generous option: as noted above, when the odds ratio exceeds 1 any $p_0 > 0$ gives a smaller risk ratio, and a smaller risk ratio gives a smaller E-value. Taking the $p_0 = 0$ limit therefore yields the largest defensible E-value these data can support. If even that maximum fails to convince, no other choice of $p_0$ will help.

To see what the assumption is worth, put a few hypothetical baseline risks through the general formula:

The two E-values

Under the rare-outcome approximation the risk ratio equals the odds ratio, so:

• the point estimate 4.150 has an E-value of 7.77
• the lower confidence limit 1.779 has an E-value of 2.96

Both are well above 1, and they are far apart. That is precisely why both get reported. Moving the point estimate back to the null would require an unmeasured confounder associated with the exposure and with the outcome at 7.77 each, which is a strong thing to have gone unmeasured. Making the result merely unable to exclude the null needs only 2.96 — less than half as much. A reader given only the first number would form a far more optimistic impression of how settled this finding is.

This is the only presentation that really does an E-value justice, because it restores the single number to the whole curve it came from. Walk right along the solid curve and the required $RR_{UD}$ falls as $RR_{EU}$ grows, but it never drops below 4.150 — that horizontal asymptote is the observed risk ratio itself. Walking up does the same in mirror image. So the question a reader should actually be asking is not whether 7.77 is a big number, but whether any unmeasured variable in this clinical setting could plausibly sit anywhere on that curve.

Example two: every estimate from one cohort, and every interval E-value at its floor

Example two reuses the cohort behind the propensity score matching and IPTW pages (medicaldata::blood_storage, 287 complete cases and 47 biochemical recurrences) and takes all 7 estimates those two pages produced. The effect measure is a hazard ratio and the event proportion of 16.4% is not rare, so the square-root conversion applies.

The last column is 1.00 the whole way down.

Putting both examples on one curve turns “large or small” into a visible position. It also makes a property of the E-value obvious: the curve is steep near the null. Move the observed risk ratio a little away from 1 and the E-value is already appreciably above 1 — so an E-value that looks comfortably greater than 1 may correspond to a very weak observed association. An E-value of 3.41 corresponds to an observed risk ratio of only 2; the observed ratio is not 3 until the E-value reaches 5.45.

Skipping the conversion inflates the E-value, and by how much is measurable

Feed the hazard ratios into the formula unconverted and this is what comes out:

The direction is fixed: skipping the conversion always makes the E-value larger, which always makes the finding look harder to overturn. The gap is modest here because these hazard ratios sit close to 1; the further an estimate is from the null and the commoner the outcome, the larger the error grows.

The question to ask of any paper, as a reviewer or as a reader, is: which effect measure was this E-value computed from, and was it converted to a risk ratio? A paper reporting a hazard ratio and an E-value with no mention of conversion in the methods has usually skipped the step.

How to compute it

# --- the E-value itself -----------------------------------------------------
evalue_rr <- function(rr) {
  r <- if (rr < 1) 1 / rr else rr          # invert a protective association
  r + sqrt(r * (r - 1))
}

# The limit of the interval CLOSER TO THE NULL. If the interval covers 1 the
# answer is 1: no confounding is needed, the data already reach the null.
evalue_ci <- function(lo, hi) {
  if (lo <= 1 && hi >= 1) return(1)
  if (lo > 1) evalue_rr(lo) else evalue_rr(hi)
}

# --- convert the effect measure to a risk ratio first ------------------------
hr_to_rr_rare   <- function(hr) hr                                    # rare outcome
hr_to_rr_common <- function(hr) (1 - 0.5^sqrt(hr)) / (1 - 0.5^sqrt(1 / hr))
or_to_rr        <- function(or, p0) or / (1 - p0 + p0 * or)           # p0 = baseline risk

# --- the contour: which (RR_EU, RR_UD) pairs explain rr away -----------------
bias_factor <- function(x, y) x * y / (x + y - 1)
contour_y   <- function(x, rr) rr * (x - 1) / (x - rr)   # asymptotes at x = rr and y = rr

# --- applied to the IPTW ATE -------------------------------------------------
# These three numbers are B6-03's output — primary.hr / hrLcl / hrUcl in this
# page's stats file — typed in so the block runs on its own. What the page
# shows is substituted at build time, so rerunning B6-03 moves these too.
hr <- 1.070253; lo <- 0.589014; hi <- 1.944676

rr <- hr_to_rr_common(hr)                  # event proportion > 0.15, square-root form
c(point = evalue_rr(rr),
  ci    = evalue_ci(hr_to_rr_common(lo), hr_to_rr_common(hi)))

import math


def evalue_rr(rr: float) -> float:
    r = 1 / rr if rr < 1 else rr           # invert a protective association
    return r + math.sqrt(r * (r - 1))


def evalue_ci(lo: float, hi: float) -> float:
    if lo <= 1 <= hi:                      # the interval already covers the null
        return 1.0
    return evalue_rr(lo) if lo > 1 else evalue_rr(hi)


def hr_to_rr_common(hr: float) -> float:
    return (1 - 0.5 ** math.sqrt(hr)) / (1 - 0.5 ** math.sqrt(1 / hr))


def or_to_rr(odds_ratio: float, p0: float) -> float:
    return odds_ratio / (1 - p0 + p0 * odds_ratio)


# Same three numbers as the R block: primary.hr / hrLcl / hrUcl in this page's
# stats file.
hr, lo, hi = 1.070253, 0.589014, 1.944676
rr = hr_to_rr_common(hr)
print(evalue_rr(rr), evalue_ci(hr_to_rr_common(lo), hr_to_rr_common(hi)))

Judging whether a reported E-value is large or small

To evaluate that sentence in the discussion section, ask four questions:

1. Which effect measure was it computed from, and was it converted? If the paper does not say, assume it was not, and discount the number accordingly.
2. Were both E-values reported? Reporting only one, and having that one be the point estimate, states the conclusion more firmly than the data do.
3. Against what benchmark? An E-value has no intrinsic scale. The useful yardstick is the study’s own measured covariates: if the strongest known risk factor in the model is weaker than the E-value, then believing an unmeasured factor is stronger still needs a reason. A study whose E-value falls below several of its measured covariate associations has effectively conceded fragility.
4. How strong was the association to begin with? The E-value is a function of the risk ratio, not independent evidence. An E-value of 3.41 means an observed ratio of 2; an E-value of 5.45 means 3.

The fourth point deserves one more sentence. Because the E-value is a monotone function of the risk ratio, it contains no new information mathematically. Its entire value is communicative: it translates an abstract worry about residual confounding into a number that can be held up against clinical knowledge. Treating it as fresh evidence is the misuse.

Common misuse

Where this page sits

• The gap it fills — propensity score matching and IPTW can only handle measured covariates; the E-value prices what they cannot reach. Example two’s estimates come straight from those two pages.
• Which back-door paths to block — DAGs and the back-door criterion. A DAG is how you notice there is a path you have no variable for; the E-value is how you price it.
• The other route is to change the design — instrumental variables and self-controlled designs do not rely on measuring confounders at all; they design them out. The E-value stays inside the original design and labels the price honestly. The two are complementary.
• Target trial emulation — emulating a target trial. When no reasonable active comparator exists, confounding by indication cannot be escaped, and that page points here.
• The rest of quantitative bias analysis — measurement error and misclassification and selection bias. The E-value covers unmeasured confounding only; those two need separate quantification.
• Where example one comes from — the case-control chapter, with the model itself on conditional logistic regression.
• The clinical setting of example two — the cohort chapter and database studies. The latter is where an E-value is most often demanded.

Rerun every number on this page

This script depends on the stats files from B6-02, B6-03 and D2, so those three have to run first (alphabetical order already does this):

/opt/homebrew/bin/Rscript figures/scripts/B6-02-psm.R
/opt/homebrew/bin/Rscript figures/scripts/B6-03-iptw.R
/opt/homebrew/bin/Rscript figures/scripts/D2-case-control-infert.R
/opt/homebrew/bin/Rscript figures/scripts/B6-07-e-value.R

Report a problem on this page