feat(diagnostics): backfill power-scaling sensitivity across the stored fits (#381)

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Opus 5).

What and why

Power-scaling prior sensitivity has been wired into every family pipeline since #408 and #416 — run_psense is called from 22 fit functions covering every family. What was missing was artefacts for fits stored before those merged: their reports showed no psense table, so a reader had no way to tell "measured, no concern" from "never measured". That distinction is #381's central meta-finding, and it mattered most for exactly the deliverables the issue names as most prior-dependent.

The correction that made this cheap: power-scaling is importance reweighting over the draws already in hand, not resampling. A stored trace carrying log_prior and log_likelihood can be measured after the fact, so 84 fits were backfilled with no sampling at all — and the numbers belong to precisely the posterior each published report was written from, which is a stronger guarantee than a fresh fit would give.

Coverage: 86 → 170 of 193 reporting fits.

Changes

• diagnostics.psense_artifacts(trace, out, var_names) extracted from run_psense, so the fit path and the backfill share one implementation rather than growing a second that can drift. run_psense delegates and keeps its context.tables bookkeeping; the existing stale-removal and atomic-replace tests in tests/statistical_models/test_diagnostics.py cover the refactor unchanged.
• scripts/regenerate_psense.py, following the regenerate_key_findings.py idiom (all / model id / fit dir, --output-dir, plus --dry-run and --force).
  • The reported-parameter set comes from each fit's own diagnostics.csv rather than being re-derived off the spec. That file is the fit's record of which parameters it reported, so the backfill covers exactly what the report shows, and it stays correct for a model whose spec has been edited since the fit.
  • Only names the posterior actually carries are measured; a name left behind by a spec edit is reported, not silently dropped, and does not fail the whole model.
  • Fits whose trace predates the log_prior wiring are reported as needing a refit, and the script exits non-zero while any remain — a sweep must not look clean while leaving estimands unmeasured.
  • Hidden output-transaction staging directories (.<id>-<config>.staging-XXXX) are excluded from the target set: backfilling into one writes artefacts about to be discarded, or races a live fit. Found by watching the first real sweep pick one up.

No report template change is needed — docs/models/_partials/_priors.qmd already renders psense_summary.csv and psense.png with a "No psense summary." fallback, so the backfilled artefacts appear on the next render.

What the measurement found

Across 1,647 parameters in 170 fits, 46 % carry some flag. That number should not be read as "46 % of the suite is unsound" — at n ≈ 54 a weakly-informative prior will often show sensitivity, and most flagged terms are nuisance or scale parameters whose prior dependence changes no reported claim. Restricting to the 155 headline causal terms across 77 models:

Family	term	✓	prior–data conflict	strong prior / weak likelihood
itt	tau	28	15	0
lf	b_grp_time	20	11	13
hs	beta	19	17	2
did	tau_t2	4	4	3
mech	beta_mech	10	5	0
bx	delta	2	2	0

The ITT suite — the primary deliverable — has no headline tau in the serious class. Fifteen show potential prior–data conflict, which is unsurprising and informational at this sample size; none is prior-driven. That is a real result rather than an absence, and it was unmeasured for a third of the suite until now.

Five models have their one randomised causal quantity flagged potential strong prior / weak likelihood — the class where the prior does more work than the data: lrp-rli-lf-005 and lrp-rli-lf-006 at b_grp_time[1] (the t2 contrast; phase coords are 0-based), both floor-limited outcomes where a thin likelihood is explicable; and lrp-rli-did-001, -003, -013 at tau_t2. These are precisely the refits #382 proposes — this pass supplies the measurement that motivates them instead of assuming it. Separately, lrp-rli-hs-001's slab scale hs_c2 reads prior 1.12 against likelihood 0.028, exactly the horseshoe dependence #381 predicted.

Recorded in notes/202607261700-psense-coverage-backfill.md.

Deliberately not in scope

23 fits cannot be repaired this way. Their traces predate the log_prior / log_likelihood wiring: the med family (18 of 19), all three rli-mm, rlm-jc-001, and rlm-mm-001 (the one legitimate exemption #381 already records — a non-converged posterior, where a sensitivity diagnostic would mean nothing). The rest need a --config reporting refit, after which the fit-time wiring emits psense with no further change.

prior_pushforward is not extended. #381 also asks for it across every estimand-reporting family, but it is not the same shape of job: it pushes a treatment term through the items-scale average marginal effect and is ITT-shaped by construction, taking the arm indicator and a trial count. Families whose estimand is a curve (mech), a ranking (hs), a latent change (lcsm) or a correlation (lcf) have no single "estimand scale" to push a prior onto — adding a call site there would produce a plausible-looking artefact that means nothing. It needs a per-family decision about what the estimand-scale check is, so it is written up rather than guessed. For context, the outcome-scale prior-predictive check is already near-universal (169 of 193); the estimand-scale one sits at 51 of 193.

Verification

• 1100 tests pass (1093 + 7 new in tests/test_regenerate_psense.py).
• ruff check src/ scripts/ tests/, npm run format:check, npm run spellcheck all clean. One allow-list addition (reweighting), inserted in convention order as a one-line diff.
• The full sweep ran end to end: 84 written, 88 already present, 23 needing a refit.

Refs #381, #382

🤖 Generated with Claude Code