Harden ITT release gates, longitudinal integrity, and joint summaries

[frankbuckley]

Note

Drafted by a LLM-based AI tool (Codex/GPT-5).

Summary

A critical review of the ITT statistical-model implementation found that the core single-outcome model is statistically defensible and internally consistent, but the code is not ready for unconditional publication sign-off. Two publication-facing release-gate problems can expose findings before required robustness evidence is validated, and two edge cases can produce inconsistent or pseudo-replicated estimates.

The ordinary single-outcome model correctly treats randomised assignment as identifying the treatment effect, uses own baseline and linear age as precision terms, validates bounded counts, fits a Beta-Binomial working likelihood, reports counterfactual average marginal effects on probability and item scales, and applies comprehensive convergence diagnostics. The remaining work is principally about making the reporting guarantees match the methodological contract and hardening supported data paths.

Findings

P1: the P/N floor-sensitivity gate does not gate the findings-first summary

_kf_build_itt() handles the post-hoc P/N floor-rule models as ordinary ITTs. It emits a headline and an unqualified cause-and-effect sentence before the floor-sensitivity release gate in _results_floored.qmd is reached.

The locally stored P and N reporting artefacts have key_findings.json status ok even though tau has a potential prior-data-conflict diagnosis and the required floor_tau_prior_sensitivity.csv is absent. The later report section can say that the release gate failed, but readers have already seen an apparently released findings box.

Recommendation: make key-findings generation floor-aware. Validate the trace-backed 3 × 2 sensitivity grid before emitting scientific findings; otherwise return a dedicated robustness_unresolved or withheld status. The causal sentence must identify this as a post-hoc, baseline-floor, available-case subgroup contrast subject to eligibility and outcome-missingness assumptions.

P1: ordinary ITT key findings ignore diagnosed prior sensitivity

_emit_itt_extras() computes psense, but generate_key_findings() checks only the sampling-quality gate. A potential tau prior-data conflict therefore does not change report or key-findings status.

The stored reporting artefacts, generated from commit 898ea03691b06403aee1d45bef306336dd18a22a, flag potential tau prior-data conflict in ten non-floor ITT or robustness fits while still producing status-ok findings. This is methodologically material because the distal treatment-prior tier was selected after the original ITT posterior table existed. The trace-validated 44-cell standard sensitivity sweep is strong, but it is separate from individual report release, is not locally present in the default output root, and does not cover every affected model or robustness variant.

Recommendation: classify tau sensitivity before key-findings generation. When a conflict is present, bind the report to appropriate provenance-validated sensitivity evidence or explicitly withhold/qualify the findings. Define the policy for base models, adjusted robustness models, and outcomes outside the current standard grid rather than treating all conflicts as informational.

P2: joint item-scale marginals ignore outcome-specific observation masks

tau_summary_joint() correctly uses the flattened cell maps to average each outcome over rows where it is observed. joint_treatment_marginals() independently averages over every fitted row.

A synthetic posterior with outcome-specific missingness reproduced materially different item-scale estimates from the two summaries. The current registered joint datasets are complete on their selected outcome sets, so current estimates are unaffected, but the joint factory explicitly supports outcome-specific missing post-scores.

Recommendation: make every joint summary call _joint_ame_draws() or a single successor. Define whether cross-outcome contrasts target each outcome's observed population or a common complete-case population, record that population in the artefacts, and test all summary/key-finding paths under asymmetric missingness.

P2: the primary longitudinal loader does not enforce key integrity

load_and_prepare() does not reject duplicate (subject_id, time) rows or treatment-group changes within a subject before the pre/post merge. A duplicated t1 row was reproduced as two fitted observations for one child, silently giving that child additional likelihood weight.

The current source CSV is clean: 216 rows, 54 subjects, no duplicate subject-time keys, and stable assignment. This is nevertheless a latent correctness defect for future data. The floor and historical loaders already contain suitable fail-loud validation patterns.

Recommendation: validate subject-time uniqueness, allowed waves, and time-invariant randomised group immediately after reading the source data and before any transition or baseline merge.

Methodological qualifications to preserve

• Describe the single-outcome estimand as an available-case modified ITT effect, not an unqualified full-population ITT effect.
• Treat the Beta-Binomial as a working model for bounded total scores, not as evidence of literal item exchangeability.
• Keep P/N explicitly post-hoc and exploratory.
• Treat factorised joint-model per-outcome effects as usable marginal estimates, but do not present cross-outcome effect differences as definitive paired contrasts without a dependence-aware or randomisation-based sensitivity analysis.
• Keep the large outcome family and post-hoc practical thresholds visible when interpreting posterior direction probabilities.

Refactoring recommendations

1. Move fit_itt, the floor branch, and fit_joint from the 9,000-line pipeline.py into family-owned pipeline modules or classes while retaining the shared stage runner.
2. Introduce typed JointIttSettings and JointIttRunPlan contracts; joint models still consume loosely typed spec.extra values.
3. Replace the two drifting prepared-data subset helpers with PreparedData.subset(mask, reason=...), recording missingness exclusions, eligibility restrictions, and factory drops separately.
4. Centralise ITT/joint counterfactual summaries and define an explicit averaging-population type.
5. Centralise release policy so convergence, prior sensitivity, floor grids, and other required robustness evidence are evaluated before key findings are generated.
6. Separate validated model/run contracts from report-prose rendering to make the statistical specification easier for unfamiliar readers to inspect.

Acceptance criteria

• P/N key findings are withheld when the required floor sensitivity is missing, invalid, or not provenance-aligned.
• P/N key findings explicitly identify the post-hoc baseline-floor subgroup and missingness assumptions when released.
• A standard tau prior-data conflict changes report/key-findings status unless appropriate validated sensitivity evidence is attached.
• Joint probability- and item-scale summaries use the same declared observation mask and averaging population.
• Cross-outcome contrasts either use a common population or explicitly identify differing populations.
• Duplicate subject-time rows and within-subject group inconsistencies fail before pre/post merging.
• Regression tests cover each failure mode.
• The family refactor preserves the resolved run plan, model recipe, output schema, and all-free-variable convergence gate.
• A fresh reporting-scale refit or provenance-validated trace reuse confirms the corrected end-to-end reporting path.

Review evidence

• 336 focused ITT, preprocessing, factory, reporting, diagnostics, key-findings, and sensitivity-contract tests passed.
• Relevant source files passed Ruff.
• The 31 locally stored ITT/joint reporting fits all passed their sampling-quality gates: zero divergences, maximum R-hat 1.000821, minimum ESS 6,525, and minimum BFMI 0.945.
• Those posterior artefacts predate current HEAD and subsequent pipeline/reporting changes, so they support the unchanged model construction but are not an end-to-end verification of the current reporting path.

[ethanbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Opus 4.8).

Proposed psense release policy for ITT key findings (#392 P1)

Right now generate_key_findings only checks the convergence gate, so a tau prior-data conflict from psense still produces a status-ok, unqualified causal headline. This sets out when a psense flag should withhold, qualify, or permit the findings.

Starting point: the effect priors are zero-centred and conservative, so a prior-data conflict means the prior attenuates a real effect, not that it invents one. The case worth gating is where the prior out-works the data on tau, not where the data dominate.

Classify tau from psense_summary.csv

Threshold s* = 0.05 (arviz default):

Class	Condition
clean	max(prior_sens, lik_sens) < s*
data-led conflict	sensitive and lik_sens >= prior_sens
prior-dominant	sensitive and prior_sens > lik_sens

Action by class and model tier

	primary	adjusted-robustness	outcome outside the 44-cell sweep
clean	release	release	release
data-led conflict	release + note	release + note	release + note
prior-dominant	withhold, unless validated evidence attached (then qualify)	qualify, never withhold	withhold, unless a per-model sweep attached

• note: one sentence saying the effect is prior-attenuated on the conservative side, so the size is a lower bound and the direction is the reliable part.
• qualify: status stays ok, but the causal claim is labelled prior-informed / exploratory and the note says the estimate leans on the prior.
• withhold: a new status robustness_unresolved (like gate_failed), no findings sentences, with a reason naming the unresolved tau prior dependence.

What counts as validated evidence (turns withhold into qualify)

A tau prior-scale sweep that is present in the output dir, computed from the same trace and commit as the posterior, and shows the sign of tau is stable across the grid. Sign stability is the bar, not interval width, since the magnitude is expected to move under a conservative prior.

Scope

Applies to families that emit psense and report a causal tau: itt, and the same rule mirrors onto did tau_t2 and gf beta_trt. Off-floor P/N are out of scope here. The floor-sensitivity gate (the other #392 P1) already handles them, and this must not double-gate.

Knobs to confirm

1. s* = 0.05.
2. Prior-dominance test: prior_sens > lik_sens, or require a margin.
3. Primary off-grid outcomes: withhold (as drafted) or always qualify.

Knob 3 is the main choice. Withhold is stricter: a prior-dominant headline with no sweep shows nothing. Always-qualify trusts that the finding is real but attenuated and labels it heavily instead of hiding it. I lean withhold for primary, qualify for robustness.

@frankbuckley, can you confirm or adjust the three knobs? I will implement once it is signed off.

[ethanbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Opus 4.8).

#392 status + a policy question on the P1 release gates

Progress so far

• P2b (loader key integrity) — landed in fix(itt): enforce source key integrity in the longitudinal loader (#392) #396.
• P2a (joint item-scale marginals ignoring outcome-specific masks) — fix(itt): joint item-scale marginals respect outcome-specific masks (#392 P2a) #399 (open): joint_treatment_marginals now routes through the masked _joint_ame_draws, so the item- and probability-scale joint summaries share one observed-row population. Behaviour-preserving on the current complete datasets; regression test under asymmetric missingness added.

P1a + P1b are ready to wire, but need a methodological/release-policy decision first. The two findings share one core question — what should a diagnosed robustness gap do to the findings-first box? — and the machinery to detect the gap already exists, so this is mostly a policy call, not an engineering one.

What already exists (the mechanism)

• _emit_itt_extras() already runs run_psense(ctx, var_names=[term]), so the tau power-scaling diagnosis (incl. potential prior-data conflict) is emitted per fit.
• sensitivity.evaluate_floor_sensitivity() + the release gate in _results_floored.qmd already recompute the 3×2 floor grid from the content-addressed traces and decide ready/complete, gated on _floor_psense_status in {"conflict", "unavailable"}.
• The gap: generate_key_findings() checks only the convergence gate, and _kf_build_itt() emits the P/N floored primaries as ordinary ITTs — both before/independent of those robustness checks. So a findings box can show status ok that the later report section then contradicts.

So the wiring is small: consult the existing tau psense diagnosis and evaluate_floor_sensitivity inside the key-findings path, and set status accordingly. The open question is the policy that status encodes.

Decision 1 — what a diagnosed gap does to key-findings status

When tau has a potential prior-data conflict (P1b), or a P/N floored model's required floor grid is missing/incomplete/not provenance-aligned (P1a):

• (A) Evidence-bound withhold (my recommendation) — set a robustness_unresolved status that suppresses the unqualified causal headline, released to a normal finding only when provenance-validated sensitivity evidence covers the model. This makes key-findings consistent with the report-section gate that already exists (_results_floored.qmd), so the box and the prose can no longer disagree.
• (B) Qualify, never withhold — still emit the headline but attach a mandatory prior-sensitive / post-hoc-subgroup caveat. Lighter, but a prior-dominated direction call still ships as a headline.
• (C) Informational only — status quo (status stays ok; the diagnosis is recorded but doesn't change release). This is the behaviour the review flags as the defect.

Decision 2 — model tiers (P1b)

The trace-validated standard sensitivity sweep is strong but is separate from individual report release, is not present in the default output root, and does not cover every model or robustness variant. Should the policy apply uniformly, or graded, across:

• base ITT models,
• adjusted robustness models,
• outcomes outside the current standard grid?

(The review explicitly asks this to be defined rather than treating all conflicts as informational — e.g. does an adjusted-robustness conflict withhold, or only qualify?)

Decision 3 — what evidence lifts the withhold

Presumably the provenance-validated grid that the floor gate already recomputes (data / model / sample-count / config-hash / trace-hash / free-variable-set / sampling-provenance match), and/or the standard sweep where it demonstrably covers the model. Please confirm what counts.

Decision 4 — the P/N released-case causal sentence (P1a)

When a P/N floored model is released, the causal sentence should identify it as a post-hoc, observed-baseline-floor, available-case subgroup contrast subject to eligibility and outcome-missingness assumptions (per the "methodological qualifications" in the issue). Confirm the wording contract.

Next step

Happy to implement on your ruling — my default, absent other direction, is (A) evidence-bound withhold, applied uniformly across tiers, with the provenance-validated grid (and the standard sweep where it covers the model) as the lifting evidence, and the post-hoc-subgroup sentence for released P/N models. I'll hold the code until you confirm the policy, since the status semantics are a release/methodology call rather than a mechanical one. The refactoring recommendations (moving fit_itt/fit_joint into family modules, the PreparedData.subset helper, centralising release policy) overlap with #394 and are best kept as separate mechanical-relocation PRs.