Why we count chemical environments, not peaks

When we first scored MolTrace's GSD (Global Spectral Deconvolution) sidecar against the NMRShiftDB2 reference corpus, the raw number was alarming. On the 19-fixture curated corpus at detection level 2, the median absolute difference between the detector's total peak count and the expert reference count was 17 peaks — a measured value in the committed internal validation report, not an estimate. A gap that size reads, at first glance, like a detector that cannot count.

It wasn't. The detector was doing exactly what a detector should. The mismatch was a units problem, and mistaking a units problem for an algorithm problem is one of the easier ways to spend a week fixing the wrong thing.

The GSD backend itself is conventional and deliberately so: single-pass detection with scipy.signal.find_peaks, per-peak Lorentzian/pseudo-Voigt fitting via lmfit, level-aware overlap resolution across levels one through five, then an expert-system pass that labels each peak compound | solvent | impurity | artifact | 13C_satellite. Nothing about that pipeline is trying to be clever about counting. It resolves lines. That is the point of a deconvolution engine.

The diagnosis came from lining up what each side of the comparison actually counts. NMRShiftDB2-style expert reference tables count distinct chemical environments — curated, assigned shift positions, one entry per chemically distinct nucleus. A spectral detector's natural output is at multiplet-line granularity: every resolved line of every doublet, triplet, and AB system is a peak. One environment can present as several lines. So the reference and the detector were both correct and were counting in different units. This is documented directly in the validation harness, which notes that NMRShiftDB2 counts environments, not lines — it is a stated finding, not a post-hoc inference.

The measured contrast made the granularity effect concrete. On the NMRShiftDB2 corpus, raw compound peak-count landed within ±5% of the expert reference on only 37% of fixtures — but compound environment-count landed within manifest tolerance on 63%. Environment granularity matched the reference far better than peak granularity did, which is exactly what you'd expect if the two sides were counting different things.

The 17-peak gap was never evidence the algorithm was broken. It was evidence we were comparing a line count against an environment count — the right measurement on the wrong scale.

The fix was to make the comparison happen on the reference's scale. In v0.5.0 the sidecar gained a post-detection multiplet-clustering layer, cluster_into_environments, that groups adjacent same-category peaks within a nucleus-aware J-coupling window into a single chemical-environment entry. Crucially it preserves provenance: each Environment record carries centre_ppm, peak_count, total_intensity, total_area, category, multiplicity, and constituent_peak_indices — so every environment remembers which detector lines were merged into it. Nothing is thrown away; the detector output is re-expressed at a coarser granularity that a reviewer can unfold.

The J-coupling window defaults to 30 Hz for ¹H and 5 Hz for ¹³C, tuned against the NMRShiftDB2 20-fixture corpus to absorb strong-coupling AB systems and constrained-ring geminal H–H couplings up to 25–30 Hz. These are nucleus-aware defaults fitted to the fixture corpus, not universal physical constants. Classification helps keep the count honest from the other direction, too: ¹³C-satellite peaks are detected at ±½·J_CH (125 Hz for sp³, 160 Hz for sp²) so those non-analyte lines don't inflate the compound count.

Watching the metric move as granularity is reconciled is the clearest way to see the whole story. On the same corpus, the median absolute delta goes 17 (all peaks) → 3 (compound peaks only) → 2 (clustered compound environments). Each step removes a source of unit mismatch: filtering to compound-category peaks, then collapsing multiplets to environments.

The strict production promotion gate has two published thresholds: solvent peak auto-detected on ≥95% of fixtures that have a known residual-solvent reference, and median compound-environment-count delta ≤2 versus the expert reference. Two facts about how it was cleared matter for honesty.

First, the ≤2 is a target, and the corpus median lands exactly on it — 2.0, meeting the gate, not beating it. It should be read as "cleared," never "exceeded with margin." The final push came from widening the ¹H clustering window from 20 Hz to 30 Hz, which dropped the median environment-count delta from 3 to 2. The solvent arm cleared at 100% (17/17 fixtures with a known residual reference) on the 19-fixture corpus — a small curated set, not a universal accuracy claim.

Second, the "19" is honest curation. One fixture, 60000023_1h, was dropped as a documented data-quality outlier: its chemical-shift referencing is off by ~1.7 ppm, placing the CHCl₃ residual at 8.96 instead of 7.26 ppm, outside the CDCl₃ solvent window regardless of detector quality. The exclusion and rationale are recorded in the manifest. In the technical write-up, the window tuning that closed the median-delta half and the outlier removal that closed the solvent half are what let the strict-gate test drop its long-standing xfail marker and pass unconditionally in v0.6.0. A regression-floor companion test locks the baseline so any change that materially degrades the sidecar fails CI loudly — the gate is enforced, not aspirational.

The reconciliation was reproduced across three independent corpora: NMRShiftDB2 (19 fixtures, 100% solvent, median environment delta 2), an HMDB-style synthetic 20-fixture multiplet-line corpus (19/20 within tolerance on environments, 20/20 on multiplet lines), and a real-instrument HMDB 100-fixture corpus (95% parseable, 93% solvent auto-detect). The synthetic corpus is the load-bearing check: the same clustering algorithm clears the median-≤2 gate when scored against multiplet-line references, which is only possible if the original 17-peak gap was a corpus-granularity effect rather than an algorithm failure. It's also why we don't gate on absolute peak-count against HMDB at all — HMDB's distinct-peaks is curator-dependent, ranging from 1 to 190 across the 100-fixture subset, so a single absolute-delta gate there would be meaningless.

Pick the measurement scale before you judge the number. Our 17-peak median wasn't a bug to fix in detection code; it was a signal that the detector and the reference counted in different units, and the honest move was to add a clustering layer that compares environments to environments while keeping every constituent line addressable.

Two boundaries stay fixed. The GSD backend is opt-in and experimental: true — the default /spectrum/analyze flow and the legacy pipeline remain authoritative, so this work never destabilized the shipping path. And the gate is a detector-vs-reference reconciliation metric on curated fixtures: it measures solvent auto-detection and environment-count agreement, nothing more. It is not a claim of structure-identification accuracy, and MolTrace's controls are designed to support standards like 21 CFR Part 11 and GAMP 5 — full computerized-system validation remains the customer's responsibility.