feat(diagnostics): close the psense refit gap and diagnose the last two holdouts (#381)

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Opus 5).

Follow-on to #442, which backfilled power-scaling sensitivity for every fit whose stored trace could support it (86 → 170 of 194). This closes the remainder: the fits whose traces predated the log_prior / log_likelihood wiring, where power-scaling genuinely could not be reconstructed without resampling.

Coverage: 170 → 192 of 194.

The refits

Twenty-one fits refitted at --config reporting — the med family (18) and all three rli-mm — through a resumable per-model driver rather than one all invocation. That mattered: the sweep was reaped partway through, and resuming skipped the completed models and cost only the fit that was in flight. The OutputTransaction staging design also held up exactly as intended — the killed fit left its published directory intact and its partial work in an abandoned hidden sibling.

These overwrote 19 published fits, so the first check was whether anything moved

Nothing did.

Mediation headlines reproduce the values already quoted in #404 and the findings notes: med-066 NIE_B −0.030 words, med-075 −0.033, med-074 proportion mediated 0.7 %.

The three rli-mm models reproduce their documented gate failures to the digit, against notes/202607210914-findings-measurement.md (2026-07-21, five days before the refit):

Model	recorded 2026-07-21	this refit
mm-001	FAIL, R-hat 1.019, ESS 354, 1 divergence	R-hat 1.0195, ESS 354, 1
mm-002	FAIL, R-hat 1.048, ESS 64, 0 divergences	R-hat 1.0478, ESS 64, 0
mm-101	FAIL, R-hat 1.021, ESS 260, 57 divergences	R-hat 1.0213, ESS 260, 57

Those failures are pre-existing and documented, with the structural legs already on HOLD under the measurement-model policy. Worth noting the exact reproduction is itself a check the suite had not previously run: identical numbers across a two-week gap and the dse-research-utils v0.8.0 bump in #437.

One consequence for reading the new artefacts: those three fits flag ~96 % of their parameters (168/176, 170/179, 169/176), which corroborates #383's diagnosis that HalfNormal(1) on loadings-and-residuals puts ~32 % of prior mass on loadings > 1. But they fail the gate, so the reasoning that exempts rlm-mm-001 applies here too — read it as corroboration of #383, not as an independent measurement.

rlm-jc-001: a code gap, then a diagnosed naming seam

#442's note listed this among the fits needing a refit. That was wrong, and the correction is in this PR. fit_rlm_joint_growth calls _run_sampling_and_loo(ctx, compute_loo=False) and never called compute_log_likelihood_and_prior or run_psense — #416 added that wiring to fit_mediation and fit_correlated_factor but not to this family. A refit alone would have burned sampling time and produced nothing.

With the wiring added, the refit shows the real obstacle, which is neither what the note assumed nor what I expected before running it:

exact match required for all data variable names, but
['eta_cell', 'sigma_subject', 'measure_corr_chol_cholesky', 'z_subject', 'kappa'] !=
['eta_cell', 'sigma_subject', 'kappa', 'measure_corr_chol', 'z_subject']

The model draws pm.LKJCorr("measure_corr_chol", ...) — chosen deliberately over LKJCholeskyCov to avoid its unidentified nuisance sd scales (factories.py:6315-6319) — and PyMC stores the transformed value variable in the posterior as measure_corr_chol_cholesky while the free RV keeps the base name. compute_log_prior and compute_log_likelihood both require an exact name match, so both refuse; the prior group is blocked as well as the likelihood group.

That is a naming seam, not an intractable likelihood, so it is plausibly fixable rather than an intrinsic exemption. Reconciling the LKJCorr value-variable name across the compute_log_* seam reaches well beyond psense coverage, so it is left as a follow-up. The value of adding the wiring is that the absence is now diagnosed with a specific error instead of being silent — which is the same distinction #381 exists to make.

rlm-mm-001 remains the one true exemption #381 already records: a non-converged posterior, where a sensitivity diagnostic would not mean anything.

Changes

• pipeline.py — fit_rlm_joint_growth gains compute_log_likelihood_and_prior(ctx, strict=False) + run_psense(ctx, var_names=diag_vars), with a comment recording why strict=False is right for a compute_loo=False family.
• notes/202607261700-psense-coverage-backfill.md — headline updated to 192/194; new closing section recording the refits, the no-change verification, and the rlm-jc-001 correction.

Verification

• 1115 tests pass.
• ruff check src/, npm run format:check, npm run spellcheck clean.
• 21/21 refits succeeded, 0 failures; 18 pass the convergence gate, the 3 failures pre-existing as above.
• regenerate_psense.py all --dry-run now reports needs refit: 2, present: 192.

What remains on #381

prior_pushforward across the estimand-reporting families (142 fits) is still blocked on a per-family decision about what the estimand-scale check is for a curve, a ranking or a correlation — #446 is establishing the pattern for the families where it is well-defined. The indicator-scale prior-predictive check for lcf / mm / rlm-mm has not been started.

Refs #381, #383

🤖 Generated with Claude Code