Validation against references that count the way detectors count

A previous note worked through why our detector's peak count sat a median of 17 peaks away from the NMRShiftDB2 reference: the reference counts distinct chemical environments, the detector resolves multiplet lines, and one environment can present as several lines. The conclusion was that the gap was a units mismatch rather than an algorithm defect.

That conclusion is comfortable, and comfortable conclusions about your own software deserve suspicion. "The reference is measuring something else" is exactly what you would say if your detector were simply bad. The claim only becomes evidence if you can find a reference that counts the other way and show that the same algorithm, unchanged, agrees with that one.

That is what the HMDB-style validation harness is for.

gsd_hmdb_style_validation.py runs backwards relative to the normal pipeline. It starts from a published peak list at HMDB / Pretsch granularity — every resolved line of every multiplet, as a human tabulated them — and forward-models a noisy Lorentzian spectrum from it. That synthetic spectrum then goes through the full GSD pipeline as though it had come off an instrument, and the output is scored against the list it was built from.

The point of generating the spectrum rather than measuring one is that the ground truth is known exactly. There is no curator judgement between the reference and the signal, because the signal was constructed from the reference. If the detector disagrees, the disagreement is the detector's.

Two details keep the synthesis from being too kind. The noise is correlated rather than white — a Gaussian σ=2 filter, which mimics the band-limited baselines you actually get from Fourier-transformed NMR — because a detector tuned against white noise will flatter itself on real data. And sparse spectra carry synthesis-floor-aware per-fixture tolerances, recorded in each entry's notes field, so a fixture with three signals is not scored as though a one-peak error were the same fraction of the answer as it would be on a fixture with thirty.

The corpus itself is 20 fixtures, hand-curated from Fulmer and Pretsch reference data, committed at tests/fixtures/hmdb_style_minicorpus/hmdb_style_minicorpus_v1.json. Small, and deliberately so: every entry was checked by hand, and a corpus nobody has read is not a reference.

The harness reports two deltas per fixture rather than one. The environment-count delta compares against distinct chemical environments; the multiplet-line-count delta compares against resolved lines. Keeping them apart is the whole design — a single blended score would hide precisely the effect we were trying to isolate.

The committed report at detection level 2 (gsd_hmdb_style_validation_report_v1) reads:

• 20 fixtures processed, 20 completed, 0 errors.
• Environment count within tolerance on 19 of 20 fixtures — a median absolute delta of 1.
• Multiplet-line count within tolerance on 20 of 20 fixtures — a median absolute delta of 2.

The same clustering algorithm that looked 17 peaks adrift against an environment-counting reference lands within tolerance on every fixture of a line-counting one. That is the shape of result a units mismatch produces. It is not the shape a broken detector produces, because a broken detector has no reason to agree with either.

A validation number is only as meaningful as your understanding of what the reference was counting. Two references, two scales, one unchanged algorithm — that is the check that actually distinguishes a units problem from a defect.

There is a third corpus: 100 real-instrument HMDB acquisitions, with no synthesis anywhere in the path. It measures the things a forward-modelled corpus structurally cannot — whether we can read what an instrument actually wrote.

95 of 100 fixtures are parseable. The five that are not were each traced to the archive rather than the reader: four are Bruker layouts carrying stray acqu2 / acqu2s two-dimensional parameter remnants that the HMDB curator left inside 1D archives, and one is missing its fid binary entirely. Solvent auto-detection runs at 53 of 57 on the subset with a known solvent reference.

What this corpus is not gated on is per-fixture peak count — and that omission is deliberate, documented, and worth explaining, because an ungated metric usually means someone is hiding from it.

HMDB's distinct-peaks field is curator-dependent. Across the curated 100-fixture subset it ranges from 1 to 190 peaks per fixture. That is not a scale; it is several different people's conventions stacked into one column. A single absolute-delta threshold across that range would be satisfied or violated mostly according to which curator happened to enter a given record, and a gate that moves with the curator rather than the detector tells you nothing about the detector. The semantically meaningful signals from this corpus are parseability and solvent auto-detection, so those are the ones with thresholds on them.

Declining to gate a metric is a defensible engineering decision. Declining to publish that you declined is not, which is why the reasoning sits in the changelog next to the numbers.

Three corpora, three jobs. NMRShiftDB2 checks agreement with expert environment assignments on real spectra. The HMDB-style synthetic corpus checks the detector against a known ground truth at line granularity. The real-instrument HMDB corpus checks that we can read what instruments produce. No one of them would be sufficient, and the reason for running all three is that each is blind to what the others catch.

The boundaries are narrow and worth stating plainly. These are peak-count reconciliation and solvent-detection metrics on curated fixtures. They say nothing about structure-identification accuracy, which is a different claim requiring different evidence. The GSD backend remains opt-in and experimental: true, with the legacy pipeline authoritative on the default path. And MolTrace's controls are designed to support standards such as 21 CFR Part 11 and GAMP 5 — full computerized-system validation remains the customer's responsibility.

The generalisable part is smaller than the numbers and more useful than them: before you threshold a validation metric, find out what the reference was counting. If you cannot answer that, the threshold is decoration.