What 'experimental' actually means in our promotion gate

Most software uses "experimental" to mean we are not confident yet. That is a feeling, and feelings do not survive contact with a regulated workflow. For an analysis backend whose output ends up in someone's evidence trail, the label has to mean something a reader can check.

Ours means two specific things. The backend is opt-in — the default /spectrum/analyze flow and the legacy pipeline stay authoritative, so a customer who changes nothing is unaffected by anything happening here. And the conditions under which it stops being experimental were written down before they were met.

That second half is the part that is easy to skip, and skipping it is how a promotion decision quietly becomes a matter of who is in the room.

The strict production promotion gate for the GSD sidecar is 95% solvent auto-detection and a median compound-environment-count delta of 2 or better, measured on the curated NMRShiftDB2 corpus.

Two numbers, one named corpus, both chosen before the run that would judge them. Neither is a threshold anyone can slide after seeing the result, because both were committed to the repository first — and a threshold you can adjust after seeing your score is not a gate, it is a rationalisation with a number attached.

The corpus matters as much as the thresholds. "95% solvent detection" is meaningless without stating on what; the same detector will produce very different rates on curated reference spectra and on whatever arrives from a customer's instrument. Naming the corpus is what makes the number auditable rather than promotional.

The gate was enforced by test_prompt3_gsd_meets_promotion_gate, and for as long as the sidecar fell short, that test carried a @pytest.mark.xfail decorator — it ran on every commit, measured the real thing, and was expected to fail.

This is worth more than deleting the test until the feature is ready. An expected-to-fail test keeps the measurement running continuously, so the distance to the gate is visible on every commit rather than rediscovered at the end. It also fails loudly if it ever unexpectedly passes, which is the case that matters: an xfail that starts passing means either you cleared the bar or you broke the measurement, and both deserve a human looking at them.

The moment a feature stops being experimental should be a diff, not a decision someone remembers making.

So the promotion event is a removed decorator. The xfail came off, and the test now passes unconditionally. There is no separate approval artefact to trust, because the artefact is the commit.

The change that cleared the gate was one default: the ¹H clustering window in _DEFAULT_CLUSTER_J_HZ_BY_NUCLEUS went from 20 Hz to 30 Hz. That dropped the NMRShiftDB2 median compound-environment-count delta from 3 to 2, which is the strict target.

The justification is chemical rather than numerical, and it has to be. A 20 Hz window splits couplings that belong together: strong-coupling AB systems and constrained-ring geminal H–H couplings run up to about 25–30 Hz, so lines from a single environment were being counted as separate environments. Widening the window to 30 Hz stops that. The number improved because the algorithm became more correct about coupling, not because a parameter was swept until the metric moved — and if the only defence of a parameter is that it improves the score, it is a fitted constant, not a decision.

One fixture, 60000023_1h, was dropped from the corpus. Removing data from the corpus you are being judged against deserves the most scepticism of anything here, so the standard is that the reason must be checkable by someone who assumes you are cheating.

Its chemical-shift referencing is off by roughly 1.7 ppm — the CHCl₃ residual peak lands at 8.96 ppm instead of 7.26 ppm. No detector can find a solvent residual outside the curated window it is looking in, so the fixture measures the archive's referencing error rather than detector quality.

Three things make that defensible rather than convenient. The exclusion and its rationale are recorded in the manifest's removed_fixtures array, not in a commit message someone has to go digging for. The raw archive is still committed, so the spectrum can be re-included the moment an evidence layer handles out-of-band TMS/DSS referencing correction. And the resulting corpus is stated plainly: 19 fixtures, with 100% solvent auto-detection across the 17 that carry a known residual reference.

Clearing the gate did not make the backend the default. It made it a measured backend with a published result — the opt-in boundary stays where it was, and the legacy pipeline remains authoritative on the default path.

The gate is also narrower than it sounds. It is a detector-versus-reference reconciliation metric on curated fixtures, covering solvent auto-detection and environment-count agreement. It is not a claim of structure-identification accuracy. And as everywhere else here, MolTrace's controls are designed to support standards such as 21 CFR Part 11 and GAMP 5; full computerized-system validation remains the customer's responsibility.

None of this is elaborate. Write the thresholds down before you measure, name the corpus, keep the failing test running, and let the diff be the record. The value is not in any one of those steps — it is that together they leave nobody, including us, able to quietly decide that a number was good enough.