Validating FHIR results with Cypress, QRDA I and the EHR dashboard

A FHIR extraction can run without errors and still be wrong. Paging drops records, codes don’t match, dates land outside windows, and nothing tells you. You need a control: an independent answer you trust, to compare your FHIR results against.

Use these controls, in order:

1. Cypress controls your calculation. Each year, prove your measure logic is correct against the official test deck.
2. The EHR’s QRDA I export controls your extraction. Once the calculation is proven, calculate the measures from the EHR’s own QRDA I files, then from your FHIR extraction, and compare.
3. The EHR’s quality dashboard is a second reference. Most EHRs show their own eCQM performance with patient lists for each population. Compare those lists with your FHIR results too.

Cypress test deck ──► measure calculation ✔ certified for the year
                               │
EHR QRDA I export ─────────────┼──► results A  (control / source of truth)
                               │
EHR FHIR extraction ───────────┴──► results B
                                        │
EHR quality dashboard ──► results C     │
                             compare A, B and C, patient by patient

Step 1 — Pass Cypress every year

Measure specifications, value sets and the Cypress test deck all change every year. Before trusting any results for a new performance year, run your calculation against that year’s Cypress deck.

• Run the Cypress test deck through your QRDA I parser and calculation locally, before submitting. It is the single most valuable step.
• Your calculating report and your QRDA III export are often two separate implementations, and Cypress only grades the export. Reconcile them. Bugs we found this way include:
  • a data-completeness counter incremented before the exclusion check;
  • a case-sensitive comparison of sex codes;
  • missing “best record per patient” filtering.
• Take population identifiers and the measure set id from the measure’s HQMF.

What common Cypress failures usually mean:

Failure	Usual cause
Extra patients	Duplicates under name variants — use relaxed matching
CONF:4509-29534	A lab result attached directly to the test entry instead of a nested result observation
Numerator 0 vs 1	A conformance error caused the entry to be discarded
Denominator exclusion 0 vs 1	An assessment with no result value was skipped by the parser

A branch in one measure’s code that looks unnecessary is usually a fix from an earlier Cypress round. Check before deleting it.

Passing Cypress tells you one thing: given correct QDM data, your calculation produces the right answer. It says nothing about whether your FHIR extraction finds the right data. That is step 2.

Step 2 — Use the EHR’s QRDA I as the source of truth

Most certified EHRs can export QRDA Category I files for a practice and measurement period. The EHR built that export from its own database for certification, so it is the closest thing you have to “what the chart really says” in a form your certified calculation can read.

1. Pick a control set: one practice, one measurement period, the measures you extract over FHIR.
2. Get the QRDA I export for that practice and period from the EHR.
3. Calculate the measures from QRDA I with your Cypress-certified logic. These are your control results (A).
4. Run your FHIR extraction for the same practice, period and measure versions, and calculate the same measures (B).
5. Compare A and B.

Another reference: the EHR’s quality dashboard

Most EHRs have a quality dashboard, usually for eCQMs, that shows each measure’s performance rate for a provider or practice. The useful part is the drill-down: for each population, the EHR lists the patients by name or id, for example “Jane Doe” in the denominator and the numerator. Typical lists are:

• denominator;
• numerator (performance met);
• performance not met;
• denominator exclusion;
• denominator exception.

Export those lists and compare them with your FHIR results (C against B), patient by patient, just as with QRDA I. The dashboard is quick to check and often available when a QRDA I export is not. It is also what the practice sees, so it is what they will compare your numbers with.

Treat the dashboard as a reference, not as certified truth:

• It is the EHR’s own calculation. It may run a different measure version from the one you report, or lag the new performance year.
• Settings change the answer. Check the reporting period (calendar year or rolling), whether it attributes by rendering provider or by practice, and which locations are included. Match your FHIR run to them.
• It may refresh overnight, so data entered today may not be counted yet.
• Identifiers differ. The dashboard shows names and MRNs; FHIR has its own resource ids. Match on the MRN identifier where the EHR exposes it, otherwise on name and date of birth.
• Patient lists are PHI. Keep exports and comparisons inside your secure environment. Never paste them into tickets, chat or email.

Step 3 — Compare patient by patient, not just totals

Matching totals can hide offsetting errors: ten patients wrongly added and ten wrongly dropped still look like a perfect match. Compare sets of patients for each population:

Population	Compare
Initial population / denominator	Which patients are in A, in B, or in both
Exclusions and exceptions	Which patients were excluded in each, and why
Numerator	Which patients met the measure in each

Match patients across the two sources by name and date of birth (QRDA I and FHIR use different identifiers), and check that both runs used the same performance year and measure version.

Step 4 — Explain every difference

Each patient who is in one result but not the other has a reason. The usual ones:

Difference	Typical cause	Where to look
Missing from the FHIR denominator	Paging stopped early; encounter status or type not qualifying; condition not active in the period	Extraction patterns
Missing from the FHIR numerator	Result only in a note or document; result in an unexpected resource; code only in a non-first coding	The EHR’s page; the measure’s page
Extra in the FHIR numerator	Text fallback matched something that isn’t a screening; duplicate result on a different scale	The measure’s page
Exclusion in QRDA I only	Exclusion data not exposed over FHIR (common for frailty and advanced illness)	The EHR’s page
Different dates	FHIR date precedence differs from the QRDA entry’s effectiveTime	Dates and timing
Different screening result	The sources record different things: a clinician’s judgment in one, a score in the other	Sources record different things

Sources record different things

Sometimes a difference is not an error: each source records something different about the same visit.

Depression screening: judgment vs score

• QRDA I (and usually the dashboard) records the result of the screen: depression screening positive or negative. That is the clinician’s judgment, or the institution’s standard. It usually does not include the PHQ-9 score.
• FHIR usually has the PHQ-9 (or PHQ-2) score as an Observation or QuestionnaireResponse, but may not have the positive or negative judgment.

So with FHIR data you apply the score thresholds yourself (PHQ-9 ≥ 10 is positive, for example). Your result will differ from QRDA I whenever the clinician’s judgment differs from the threshold, for example when a PHQ-9 of 9 was judged positive. Expect these differences, list them separately, and don’t try to “fix” the extraction for them. When FHIR does carry the judgment, use it, because it is what the measure counts.

Other examples

• Exclusion evidence such as frailty, advanced illness or hospice often reaches QRDA I and the dashboard but is not exposed over FHIR.
• Free-text results: an A1c or screening noted only in a clinical note can be counted by the EHR’s own calculation but isn’t in a structured FHIR resource.
• Encounters the EHR counts: the EHR may qualify visits using internal visit types that FHIR doesn’t expose, so its denominator can include patients your FHIR run cannot qualify.

Fix the extraction, re-run from your archived NDJSON, and compare again. Decide in advance how much difference you accept, and record any differences that remain with their causes. Some data really does not exist over FHIR.

Step 5 — Repeat when anything changes

Re-run the comparison:

• every performance year, after passing that year’s Cypress deck;
• when you onboard a new EHR or a new practice on an existing EHR;
• when the EHR vendor upgrades its FHIR API;
• when you change extraction or measure logic.

Once a practice’s FHIR results match its QRDA I results, you can extract more often and more cheaply over FHIR.

Reading QRDA I correctly

The control is only as good as your QRDA I parsing:

• Combine each entry’s primary code and translation codes before matching value sets.
• Results are often in a nested result observation, not on the test entry itself.
• Date results by the test’s own effectiveTime.
• Blank date-of-birth null flavors (UNK, ASKU) rather than parsing them.
• Drop future-dated entries.
• Detect which measures a file contains from the eMeasure identifiers, then population identifiers, then text references. Take the reporting period from the document’s first effectiveTime.
• Large archives can contain many practices. Route each patient by the attending provider’s TIN, not by the custodian.
• The same patient often appears in several files, sometimes under name variants. Group records by name and date of birth with relaxed matching (two of three identifying points).