ExpertReporting guideline: RECORDIndependently reviewed, not yet spot-checked by a human

Database studies and real-world data

A national claims database does not exist for research; it exists for reimbursement. This chapter covers what that gap costs you, the limits specific to Taiwan's NHIRD, and a seven-rung worked example — including the rung that lands on the right answer by the wrong method.

This data was never collected for you

The purpose of a health insurance claim is to get paid. An ICD code is entered to justify why this encounter should be reimbursed, not to record that this person definitely has this disease. That gap is the source of every methodological problem in database research.

Its consequences are concrete and predictable:

Claims logic	Consequence for research
The diagnosis code exists to justify payment	“Suspected” and “confirmed” often share a code; a rule-out diagnosis leaves a record behind
One encounter can carry several codes, and their order need not mean anything	The line between primary and secondary diagnosis is blurrier than you would assume
What was not claimed does not exist	Out-of-pocket services and untreated disease are entirely invisible
Reimbursement rules change	The same code can represent a different clinical situation in different years

What is specific to Taiwan’s NHIRD

Taiwan’s National Health Insurance Research Database (NHIRD) is the research extract of the country’s single-payer claims system, which covers essentially the whole population. Below are the things to build into a study design. Application and usage rules change, and the governing authority’s current announcements are what actually apply; what follows is only about the consequences for design.

What is in it: outpatient and inpatient diagnosis codes, prescriptions (drug, dose, days supplied), claimed tests and procedures, procedure codes, encounter dates and the level of the institution. Coverage approaches the entire population, follow-up can run very long, and patients are not lost when they move house or switch hospitals — that last point is its biggest advantage over any single hospital’s records.

What is not in it (and these are usually exactly the confounders that matter):

Laboratory values — a claim shows that HbA1c was performed, never what it came back as
The content of imaging and pathology reports
Lifestyle: smoking, alcohol, exercise, BMI
Disease severity: stage, functional class, symptoms
Cause of death (requires separate linkage to the death registry)
Events that never brought the patient in

Defining disease: the coding algorithm needs validating

Because a diagnosis code is not a diagnosis, a database study has to define for itself what counts as having the disease. The usual forms an algorithm takes:

N outpatient visits or 1 admission carrying the code (much the commonest form)
The code plus a matching prescription (a diabetes code plus a glucose-lowering drug, say)
The code plus a specific test or procedure
A time window constraint on top (at least two occurrences within a year, for instance)

The accuracy of these algorithms has to rest on something. Ideally you cite a published validation study — the algorithm checked against chart review, reporting PPV and sensitivity. Failing that, run a sensitivity analysis at minimum: repeat the study under a stricter and a looser definition and see whether the conclusion holds.

Seven rungs: one dataset, seven answers

This section uses simulated data to show what each of the commonest design errors in database research actually costs. The setup: 8000 people, a true HR of 0.80 (the treatment is protective), a comparator drug with no effect, and the textbook confounding by indication between exposure and prognosis.

The “analysis n” column is the size of the dataset that particular model actually ran on, not the size of the whole simulated cohort — change the design and you change who is eligible to be analysed.

Approach	Analysis n (treated / comparator)	Events	HR	95% CI	Distance from the true 0.80
1. Naive comparison (on the drug vs not, ignoring timing)	5092 (2003 / 3089)	1482	1.13	1.02–1.25	Points the wrong way, and significantly so
2. New-user design, but follow-up timed from the start of the study window	5737 (870 / 4867)	1015	0.78	0.66–0.93	⚠️ Almost exactly right — by a method that is wrong
3. New-user design, clock pinned to the first prescription, unadjusted	5737 (870 / 4867)	1015	1.34	1.13–1.59	Points the wrong way
4. New user, adjusted for the measured confounders	5737 (870 / 4867)	1015	1.01	0.85–1.20	Back near 1; the protective effect is still not detected
5. New user plus active comparator, unadjusted	1778 (870 / 908)	339	0.93	0.75–1.15	Right direction, no statistically significant difference
6. New user plus active comparator plus adjustment	1778 (870 / 908)	339	0.79	0.63–0.99	Close to the truth
7. New user plus active comparator plus IPTW	1778 (870 / 908)	339	0.78	0.62–0.99	Close to the truth

The table above, drawn as one row per analysis. The black dashed line is the simulated true hazard ratio of 0.80. What to read here is not which row lands closest to the line, but that the ladder does not descend monotonically towards it: row 2 sits almost on the line, and it is the row whose time zero is most plainly wrong.Plotting script figures/scripts/B6-06-target-trial.R

Row 2 is the most important row in this table, precisely because it got the right answer.

An HR of 0.78 against a true 0.80, a tidy confidence interval, a significant p-value. Judged on the result alone, this is a “successful” analysis.

But its time zero is wrong. Follow-up starts the moment the study window opens rather than on the prescription date — which credits the treated group with the stretch of time they had to survive in order to receive a prescription at all, a stretch during which, by definition, they could not have had the event. This is immortal time bias. Row 3 moves the clock back to the first prescription and that immortal time disappears.

The reason it lands on the right answer is that this bias happens to run opposite to the uncorrected confounding by indication, and the two cancel. Change the dataset, change the effect size, and they cancel to a different degree — and you would have no way of knowing.

A right answer is not evidence of a right method. A result that only appears under the wrong method is not a result.

Rows 3 through 6 show what each of the two corrections contributes on its own: switching to an active comparator (rows 3→5) handles the problem that people on a drug are fundamentally different from people on no drug at all, and adjusting for covariates (rows 5→6) handles the measurable differences left over. It takes both to get back near the truth.

Three design principles

New-user design

Enrol only people who start the drug, and follow them from the first prescription — with time zero placed correctly.

Do otherwise and you enrol prevalent users: people who have already been on the drug for a while and are still on it. That group carries two problems: they are the survivors — the ones who tolerated the drug — and their baseline characteristics have already been altered by the drug itself.

Active comparator

Use another drug treating the same indication as the comparison group, not “people not on any drug”.

The reason is visible in the ladder above: people not on treatment differ from treated people in disease severity, health-seeking behaviour and comorbidity burden, and most of those differences were never recorded. Two groups who are both on treatment are at least comparable in that a clinician judged each of them to need treating.

Time zero has to be defined explicitly

Three things must happen at the same moment: meeting the eligibility criteria, being assigned to an exposure, and starting the follow-up clock.

Let any one of them slip and you manufacture immortal time bias or selection bias. This is also the reason target trial emulation exists — it writes out, item by item, what the protocol would say if this were a randomised trial, and then checks whether the analysis of the observational data has drifted away from it.

What RECORD asks you to report

RECORD is an extension of STROBE aimed specifically at studies using routinely collected data. On top of the STROBE items, it additionally requires:

The source and version of the database, the population it covers and the time period
The codes and algorithms used to define every variable — listed in full, usually in a supplement
Whether validation studies support those algorithms
The data cleaning and linkage process, including the proportion of failed linkages
Whether the original medical records could be obtained for confirmation

Common misuses

Misuse	Why it is wrong
Using “people not on the drug” as the comparison group	The two groups differ systematically on unmeasured characteristics; use an active comparator
Enrolling prevalent users	Survivor bias, and the baseline has already been changed by the drug
Time zero out of step with eligibility and assignment	Manufactures immortal time bias
A disease definition with neither validation evidence nor a sensitivity analysis	The direction and size of the misclassification are completely unknown
Analysing trends across years without checking reimbursement policy history	Claims shifts caused by policy cannot be told apart from real trends (ITS / DiD treat the policy change as the intervention)
Treating an adjusted estimate as causal	Unmeasured variables — severity, lifestyle — cannot be adjusted away
Leaving unaddressed that treated patients return more often and so are diagnosed more readily	Detection bias, which manufactures associations
Reporting the final model without listing the algorithms	The reader cannot judge how the cohort was assembled, or reproduce it
Not interrogating the method because the result matched expectations	See row 2 above: biases that cancel also produce right answers

Reproducing this chapter’s ladder

/opt/homebrew/bin/Rscript figures/scripts/B6-06-target-trial.R

Read the figure

The answer comes from the same statistical output that produced this page's figures, not from a number typed in beside them.

Row 2 of the ladder lands almost exactly on the hazard ratio the simulation was built with, its interval is tight and its p value is significant. Does that make it a good analysis?

Show the answer and why

Correct answer: No. The truth is 0.80, and this row lands near it because immortal time bias happens to run opposite to the confounding and the two cancel

Row 2 has the wrong time zero: follow-up starts when the study window opens rather than at the first prescription, which counts the stretch spent surviving until a prescription could be filled, a stretch in which no event can occur by construction, as exposed follow-up. Row 3 pins the clock back to the prescription date and the same data give 1.34, in the opposite direction. What makes 0.78 look right is two biases of opposite sign cancelling, and how completely they cancel depends on the effect size and the strength of confounding in this particular dataset, so it would differ in another one and you would not know. A right answer is not a right method: a result that appears only under the wrong method is not evidence.

Going from row 3 to row 6 applies two corrections: switching to an active comparator and adjusting for measured covariates. Which statement is right?

Show the answer and why

Correct answer: Both are needed. The comparator handles that drug takers differ from non-takers, adjustment handles the measurable differences left, and only both returns 0.79

Row 4's 1.01 shows that adjustment alone does not get there: people who take nothing differ systematically from people who take something in disease severity, health-seeking behaviour and comorbidity burden, most of which was never recorded, and adjustment can do nothing about variables it never saw. Row 5's 0.93 shows the comparator alone is not enough either: two treated groups are at least comparable in a clinician having judged treatment necessary, but the measurable differences remain. Doing both yields 0.79, close to the value the simulation was built with. That is why a new-user design with an active comparator is written up as the default opening in pharmacoepidemiology rather than as two interchangeable tricks.

The analysis n column of the ladder differs on every row. Why does one simulated dataset give different analysis sizes?

Show the answer and why

Correct answer: Because the design decides who is included. With an active comparator only 1778 people remain, since only those who actually started one of the two drugs can enter

8000 is the whole simulated cohort, and no row runs on it. Row 1's 5092 is who the naive comparison includes; the new-user design includes a different set again; and by the active-comparator rows only 1778 remain, because the comparator changes from people taking nothing to people taking another drug, which redefines both sides. So the varying analysis n is neither sampling nor missing data: the design itself decides who counts. When the sample sizes in a database study differ between analyses, the first question is which eligibility rule changed, not whether someone was lost.

Row 1, the naive comparison of ever-treated against never-treated with no regard for timing, gives a hazard ratio pointing the wrong way and reaching significance. What does that show?

Show the answer and why

Correct answer: That confounding by indication can be strong enough to reverse the direction. 1.13 reflects clinicians giving the drug to people who look like they need it, not the drug's effect

The simulation was built with a protective treatment effect, and the naive comparison returns 1.13, in the opposite direction and significant. The reason is textbook confounding by indication: the people who received the drug were sicker to begin with. Statistical significance offers no protection here; it only says the sample was large enough to estimate a bias precisely. The p value of 0.02 is what that looks like, since a bias can be highly significant. As for the comparator, the simulation set its hazard ratio to 1.00, meaning no effect at all, so the third statement turns something defined as neutral into something harmful.

Row 6, covariate adjustment, and row 7, inverse probability weighting, give almost the same hazard ratio. Does that make the weighting redundant?

Show the answer and why

Correct answer: No. Both methods use the same measured covariates, so agreement only says their assumptions both hold here, and 0.787 matching is a credibility signal

Regression adjustment and inverse probability weighting handle the same measured covariates by different routes, one putting them into the outcome model and the other deriving weights from an exposure model and comparing weighted groups. Both routes give 0.787 and 0.785 on the same data, from which what follows is that neither specification is obviously wrong here, which is how a sensitivity analysis is used rather than evidence that one contributes nothing. The 0.931 is the unadjusted estimate, and its distance from the other two measures how much adjustment did, which says nothing about the relationship between the two adjustment methods. More importantly, both handle only what was measured: disease severity and lifestyle, absent from claims data, are as far out of reach for weighting as for adjustment.

Rows 2 and 3 use the same people and the same model and differ only in when follow-up starts for the exposed. How large is that difference in the data?

Show the answer and why

Correct answer: The immortal time wrongly credited to the exposed group totals 21125 person-months, a stretch in which no event can occur by construction

The two rows share their people and their model exactly: 870 new users, the same comparison group, the same specification, differing only in when exposed follow-up begins. Starting the clock when the study window opens credits the exposed denominator with the time spent surviving until a first prescription, here 21125 person-months in total. The denominator inflates while the numerator does not, and the exposed event rate is diluted into what looks like protection. The 3173 is the whole simulated cohort's deaths, and it does not change with where the clock is placed; what changes is the denominator. This is what immortal time bias looks like in data: no single cell appears wrong, and what is wrong is how time was counted.

Methods used in this chapter

Watch next

ENMichael Fralick· 6 minDatabase studies are mostly retrospective cohorts or nested case-control studies. Get the boundary between those two straight first.

Sources and licences

This page is original writing