docs(notes): record the item-difficulty exchangeability review and likelihood sensitivities

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Fable 5).

Summary

Records the 2026-07-26 review of whether ordered item difficulty (later words are harder on the word-reading and vocabulary tests) undermines the Binomial/Beta-Binomial working likelihoods, and adds the diagnostic and sensitivity tooling that grounds it. The decision record is notes/202607261405-binomial-exchangeability-item-difficulty-review.md.

• Keep the Beta-Binomial as the default working likelihood; diagnose with existing artefacts, then patch locally. A Conway–Maxwell-binomial probe for word reading — free to express sub-binomial variance — chose overdispersion (ν = 0.50, 89 % [0.37, 0.67]) and left the items-scale treatment effect invariant (+2.38 → +2.49 items, P(>0) = 0.99 both), so the headline W result is likelihood-robust and there is no evidence the Beta-Binomial's variance floor binds.
• The negative word-reading age association is not a measurement artefact. Null simulations from Rasch-type difficulty ladders (kinked, graded, homogeneous; corr(age, ability) matched at +0.37; zero true age effect) bias the fitted age coefficient positive (+0.17 to +0.22), and real-data hinge terms at the score kink leave the slope unchanged — so the observed negative slope is substantive, and the ladder structure if anything masks part of it. Attribution corrected along the way: −0.12 (P = 0.99) is the stacked gain-factors figure; the single-window ITT posterior is −0.08 (P(<0) = 0.83).
• Blending's headline is link-sensitive (the one live substantive call, flagged for the education lead): under a guessing-floor link respecting the 3AFC chance floor, the ≈ 1-item strong-evidence ITT effect halves to ≈ 0.5 items at moderate evidence (P 0.98 → 0.90), with PSIS-LOO a tie between links.
• PPC coverage sweep across 190 stored reporting fits: 50 % bands overcover everywhere (0.65–0.92 pooled by outcome), shown to be substantially mechanical (discrete intervals, floor-concentrated scores, in-sample checks on hierarchical fits) rather than a likelihood defect.
• Rasch/IRT assessed and closed: original scoring sheets are not readily available, so no item-level response matrix can be assembled; a value-ordered agenda is recorded in case records ever surface.

New files

• notes/202607261405-binomial-exchangeability-item-difficulty-review.md — the full decision record (questions, assessment, follow-up results, dated decisions with owners)
• scripts/ppc_coverage_sweep.py — standing diagnostic pooling ppc_summary.csv coverage by outcome/family
• scripts/age_artefact_check.py — real-data hinge check + null-ladder simulation (seeded, reproduces the note's table exactly)
• scripts/likelihood_sensitivity_prototypes.py — the blending chance-floor and word-reading CMB prototype fits (mirror the ITT structure and priors; standard arms reproduce the stored reporting fits, W τ +0.354 to three decimals)
• config/spellcheck/allow-en.txt — eight new terms; the re-sort also normalised an unsorted tail block and merged duplicates (verified nothing lost)

Checks

ruff check src/ scripts/…, npm run format:check, and npm run spellcheck all pass.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR records the 2026-07-26 decision and supporting evidence around ordered item difficulty vs the Binomial/Beta-Binomial working likelihoods, and adds standalone scripts to reproduce the key diagnostics/sensitivity checks described in the note.

Changes:

• Add a decision record note documenting the exchangeability/likelihood review, follow-up analyses, and decisions.
• Add three one-off/diagnostic scripts: a PPC coverage pooling sweep, an age “artefact” hinge+simulation check, and likelihood sensitivity prototypes (chance-floor link; Conway–Maxwell-binomial probe).
• Update the spellcheck allow-list with additional terms and some re-ordering/deduplication.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/ppc_coverage_sweep.py	Aggregates existing ppc_summary.csv artefacts across stored fits to summarise PPC coverage by outcome and model family.
scripts/likelihood_sensitivity_prototypes.py	Prototype likelihood/link sensitivity fits mirroring ITT structure (standard vs chance-floor link; standard vs CMB).
scripts/age_artefact_check.py	Implements a hinge-term real-data check and null simulations to test whether item-ladder structure could induce a spurious negative age slope.
notes/202607261405-binomial-exchangeability-item-difficulty-review.md	Decision record capturing the exchangeability/likelihood review, evidence, and follow-up actions/owners.
config/spellcheck/allow-en.txt	Adds/normalises allow-list terms needed for the new prose/scripts.

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Fable 5).

Thanks — applied in b92bbdc, and following the convention through turned up something more consequential than the narrow point.

The comment itself. Correct on the convention: diagnostics.py:583, loo_refit.py:202, scripts/design_analysis.py:108 and the shared dse_research_utils gate all use int(np.asarray(...["diverging"].values).sum()). In fairness to the original line, the claimed TypeError did not actually occur — the script ran cleanly four times — so this was a portability/consistency fix rather than a live bug. Worth having anyway: this file hit two genuine ArviZ API drifts during development (az.rhat(...).to_array() and az.compare(ic=...)), so defensive coercion here is well earned.

What it surfaced. The house pattern pairs that coercion with az.summary(..., round_to="none"), and the repo's own comment explains why: round_to=None falls through to rcParams["stats.round_to"] (2 significant figures) — the dseinternational/research#65 failure mode where a rounded R-hat silently clears a gate. The prototypes were doing exactly that, and my pd.to_numeric(...) workaround was papering over the string-formatted output instead of fixing the cause.

Reading unrounded changes the reported diagnostics:

fit	was reported	actual (unrounded)
B, standard Beta-Binomial	R-hat 1.0000	1.0018
B, chance-floor	R-hat 1.0000	1.0011
W, standard Beta-Binomial	R-hat 1.0000	1.0014
W, Conway–Maxwell-binomial	R-hat 1.0000	1.0022

min ESS now takes the bulk/tail minimum as the gate does (4,137 for B; 1,494 for W), rather than bulk only.

No conclusion moves. All four fits stay well inside the gate (R-hat ≤ 1.01, ESS ≥ 400, 0 divergences), and every posterior summary is byte-identical on a reseeded rerun — τ, the items-scale marginal effects, ν = 0.503, and the PSIS-LOO tie are all unchanged. The affected diagnostic lines in §5 and §6 of the note are corrected in the same commit.

The unused pandas import went with the workaround.

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Fable 5).

Shared helper added in d2b57c0 — and building it surfaced a live defect in loo_refit.py, which I'd like a second opinion on before this merges.

The defect

MechanismSamplingWrapper._assert_refit_converged (added in #438/#439) read its gate signals with az.summary(idata, round_to=None). That does not disable rounding — round_to=None falls through to rcParams["stats.round_to"], which is "2g", two significant figures. Verified in this environment against a stored reporting trace:

call	R-hat returned	dtype
round_to="none"	1.000444886	float64
round_to=None	1.0	float64
omitted	'1.00'	str

Because two-significant-figure rounding maps the whole band onto 1.0, the R-hat <= 1.01 arm of that gate was effectively R-hat < 1.05:

true R-hat	rounded	passes gate after rounding	should pass
1.011	1.0	✅	❌
1.020	1.0	✅	❌
1.049	1.0	✅	❌
1.050	1.1	❌	❌

The consequence is not cosmetic: a leave-one-out refit with R-hat up to 1.049 would pass, have its exact density spliced into the elpd, and the comparison then marked comparison_valid. The same call took ESS from ess_bulk alone, where the gate is defined on the bulk/tail minimum. There's an irony in the module's own comment — it takes the thresholds from the shared package "rather than by a duplicated set of numbers that could drift", but the extraction was duplicated and drifted.

I have no evidence any published comparison was actually affected — every refit I can check is far inside the gate — but that is luck, not the gate working.

The fix

statistical_models/sampling_quality.py is now the single place a trace's R-hat, ESS, BFMI and divergences are read, unrounded and correctly coerced. diagnostics.py, loo_refit.py, scripts/design_analysis.py and the likelihood prototypes all route through it.

Deliberate scope limits, since this touches a production gate:

• The helper extracts only — it does not decide pass/fail. Call sites legitimately differ in which variables they gate over (curated list vs all) and in how they treat a missing BFMI (diagnostics fails, loo_refit passes); homogenising those silently would be a second bug. Both policies are preserved verbatim.
• subfit_convergence keeps its "uncheckable" (converged=None) semantics when sample_stats has no diverging, which previously arose from a KeyError.
• loo_refit keeps summarising all variables rather than a curated set.

So the only intended behaviour change is that the refit gate now applies the threshold it always documented.

Verification

• 13 new tests in test_sampling_quality.py, including a fixture tuned into the (1.01, 1.05) band that asserts the rounded value clears the gate the true value fails — so this cannot regress silently.
• Full suite green (1,089 tests), ruff and the mypy boundary clean.
• The prototypes reproduce identical posterior summaries through the helper.
• One existing test moved its monkeypatch seam from diagnostics._bfmi_per_chain to sampling_quality._bfmi_per_chain, since that is now the call path; its assertions are unchanged.

METHODS.md now directs all diagnostic reads through the helper rather than describing the trap in prose.

Worth a look from a reviewer who knows #438: the loo_refit change is in someone else's recently-merged code, and while I believe the fix is strictly correct, a stricter gate could in principle cause a previously-passing refit to fail and fall back to the per-model table. That is the intended behaviour, but you may want to confirm it against a real mechanism-family run before merging.