docs: confirm 'no real improvement' verdicts for metal-TS systems (n=10)

[ericchansen]

Summary

Reruns pd-allyl, rh-conjugate, and heck-relay with --n-evals 10 (q2mm#286, now on master) and rewrites each per-system doc page with the statistically defensible verdict rather than the earlier "within noise" caveat from #283.

Companion data PR: ericchansen/q2mm-data#9

Verdicts (now decisive)

System	Mean Δ%	CI₉₅	Verdict
pd-allyl	−0.029 %	±0.34 %	NOT SIGNIFICANT
rh-conjugate	−0.080 %	±1.18 %	NOT SIGNIFICANT
heck-relay*	−0.59 %	±3.26 %	NOT SIGNIFICANT

* heck-relay run with --ratio-tol none (ratio = 1.378, formally fails default gate); even with the gate bypassed, the JaxLoss surrogate broke down (2 non-finite line-search values) and the result is inside the noise band.

Doc changes per system

Each page is rewritten so the earlier "within noise floor, cannot claim" caveat becomes a "Confirmed: published Wahlers/Rosales FF sits at a JaxLoss local minimum" success callout. The CI₉₅ excludes any improvement larger than the per-system noise floor, so this is now a defensible "no real improvement available", not "we can't tell".

Specific updates:

• pd-allyl — table now shows mean ± CI₉₅; caveat replaced with success callout; gap analysis unchanged
• rh-conjugate — table same shape; the 4602-ratio non-determinism caveat is removed (ratio is now stable at 1.01 with n=10); Investigate rh-conjugate JaxLoss ratio non-determinism (0.46-0.96-4602 across runs) #278 stays closed
• heck-relay — --ratio-tol none recommendation strengthened: with statistical rigor in place, the gate bypass demonstrably doesn't unlock useful optimization

Why these verdicts matter

The earlier #283 results flagged these three systems as "within noise" but couldn't say whether there was a real signal hiding under the per-call GPU noise. The n=10 + Student-t 95% CI confirms there isn't:

• All three CIs exclude any improvement larger than the per-system noise floor
• All well below any publishable improvement claim
• The published Wahlers/Rosales FFs sit at JaxLoss local minima for our engine
• Further improvement requires the engine-parity work in MM3 energy is non-smooth: JAX grad returns wrong subgradients at converged metal-TS geometries #284, not optimizer tweaking

Validation

• ruff check + format clean (no Python changes; docs only)
• properdocs build clean (no new warnings)
• All numbers in tables traceable to JSON in companion data PR Support for Gaussian Hessian, eigenvectors, etc. #9

Out of scope

• The MM3 non-smooth gradient bug (MM3 energy is non-smooth: JAX grad returns wrong subgradients at converged metal-TS geometries #284) — confirmed by these runs to be the real blocker for further improvement on metal-TS systems
• Engine-parity rewrite of JaxEngine.minimize — multi-day, separate PR

[Copilot]

Pull request overview

Updates the published-FF system documentation for three metal-TS benchmarks (pd-allyl, rh-conjugate, heck-relay) to report statistically defensible “no real improvement” verdicts using --n-evals 10 and 95% confidence intervals, replacing the earlier “within noise” caveats.

Changes:

• Replace “within noise” wording with decisive “NOT SIGNIFICANT” verdicts based on n=10 + CI₉₅.
• Update per-system benchmark tables to include mean ObjectiveFunction values and CI₉₅, plus CI₉₅ on mean Δ%.
• Revise narrative callouts/recommendations (including --ratio-tol none discussion for heck-relay).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
docs/systems/rh-conjugate.md	Updates benchmark table to n=10 mean/CI₉₅ and rewrites interpretation as “no detectable improvement.”
docs/systems/pd-allyl.md	Updates benchmark table to n=10 mean/CI₉₅ and reframes result as confirmed local minimum.
docs/systems/heck-relay.md	Updates benchmark table to n=10 mean/CI₉₅ with --ratio-tol none and strengthens recommendation to keep the ratio gate.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.