Correct level-factors causal estimand, priors, and reporting

[frankbuckley]

Note

Drafted by a LLM-based AI tool (Codex/GPT-5).

Summary

A critical review of the level_factors family found that its core data preparation, wait-list crossover indexing, group sign convention, Beta-Binomial/Bernoulli likelihood branches, child random intercept, and PyMC construction are largely correct. No direct sign, indexing, or likelihood-construction defect was found. However, the current t2 items/risk-difference headline does not map cleanly to one named causal estimand, the level-intercept priors are poorly centred for this population, and several causal-reporting safeguards used by the ITT family are absent. Resolve these issues before treating the level-factor headlines as final publication-ready causal estimates.

Review performed against main at 691aa9ef098156d37d0e37e09aa116de5358755c.

Findings

1. The t2 marginal effect is a hybrid quantity

The model fits one time-invariant group × ability term across all four waves (factory). That coefficient is therefore informed by t1 and post-crossover t3/t4 associations as well as the randomised t2 contrast.

The reporting function removes the complete fitted group contribution, including gamma_grp_ability × ability, but adds back only b_grp_time[1] while leaving the ability main effect evaluated at each observed child's ability (reporting). The resulting items-scale quantity is neither:

• the population-average t2 effect, because fitted effect modification is excluded; nor
• literally a prediction for a mean-ability child, because the ability main effect remains evaluated over observed profiles.

The key-findings builder nevertheless describes it without qualification as how much “the intervention changed” the outcome (key findings).

As an impact check using the saved reporting traces, incorporating the fitted interaction changed the posterior median from +1.45 to +1.67 items for W and from −3.80 to −4.18 items for R. Those shifts are modest relative to the uncertainty but demonstrate that the choice is not algebraically neutral.

Recommendation: remove group_ability from this longitudinal sensitivity family and estimate effect modification in a dedicated t2 ITT model. If it remains, make it phase-specific, identify the t2 component from the randomised wave, and define whether the headline is a population-standardised average effect or a conditional effect at a stated profile.

2. The intercept parameterisation is redundant and its priors are poorly centred

The linear predictor includes a free global alpha plus a complete four-element alpha_time vector (factory). Only their sums are identified by the likelihood; the priors make the posterior proper but do not make the decomposition substantively identifiable. In the saved fits, posterior correlations between alpha and individual alpha_time elements range from approximately −0.62 to −0.96.

Both terms are centred at logit zero, implying scores around half the instrument maximum before seeing data. Saved prior-predictive means are poorly aligned with the study population:

Outcome	Prior-predictive mean	Observed mean
W	39.1	11.8
R	86.1	41.1
E	84.1	35.0

Recommendation: use either one phase-intercept vector with substantively justified locations, or a global intercept plus reference-coded/sum-to-zero time deviations. Calibrate locations from external knowledge where possible; if an empirical-Bayes anchor is adopted, label it explicitly and perform a sensitivity analysis.

3. Prior-sensitivity warnings are not interlocked with the headline

All eleven saved reporting fits pass the convergence gate with zero divergences, but power-scaling flags the headline t2 coefficient for:

• W and L: potential prior–data conflict;
• P and B: potentially strong prior / weak likelihood;
• N: potential prior–data conflict.

These warnings appear only in a collapsed prior section. The convergence badge and key-findings headline remain unqualified. The level family also lacks the estimand-scale prior_pushforward.csv emitted for ITT/gain models.

Recommendation: surface a warning beside the headline whenever t2 power-scaling is not clear, require a direct prior-grid sensitivity for flagged outcomes, and emit the same estimand-scale prior pushforward used by the other randomised-effect families.

4. Available-case causal limitations are missing from metadata and prose

The repository contains 54 of 57 randomised children, and the level fits use 53–54 children depending on outcome. Randomisation supports the t2 contrast in the fitted available-case population; extending the claim to excluded or missing children requires an ignorable-missingness assumption.

ModelSpec.__post_init__ applies the trial metadata only to itt and joint (context). Level-factor configs consequently persist null family, design, estimand_type, and causal_status fields while reports publish an unqualified cause-and-effect statement.

Recommendation: centralise available-case randomised-design metadata and report wording for every family that publishes a randomised term, including level factors, gain factors, and arm-by-wave DiD.

5. SP/RW timing protects t2 but changes the longitudinal association interpretation

Speech production and phonological memory are correctly read at t1 to avoid post-treatment adjustment, while hearing remains contemporaneous (pipeline). This protects the t2 causal contrast.

However, gamma_deapp_c and gamma_erbto are then baseline-predictor associations broadcast across all waves, not contemporaneous factors associated with each wave's score under the single-wave DAG. Reports should name them as baseline predictors. If contemporaneous association estimates remain an objective, use a separate descriptive longitudinal model rather than asking one model to satisfy both causal-t2 and contemporaneous-association interpretations.

6. The family boundary is difficult to audit and vulnerable to drift

The implementation is spread across large shared modules: factory construction, pipeline loading/diagnostics, reporting transformation, and eleven repetitive spec modules. ModelSpec.extra is an untyped dictionary, so a misspelled setting can silently fall back to a default. Coefficient names and roles are separately reconstructed by _lf_coef_names, _lf_diag_vars, the factory, and the reporting code.

Recommendation: create a family-owned level_factors.py, following the typed ITT architecture, with immutable LevelFactorsSettings and a resolved run plan. The builder/run plan should own coefficient roles, diagnostic variables, causal terms, covariate timing, and estimand metadata. Keep the individual discovery modules thin or generate them from one typed outcome-configuration table.

Acceptance criteria

• Record a dated methodological decision defining the t2 estimand and the treatment of group×ability effect modification.
• Make the reported probability/items-scale contrast exactly implement that estimand, with unit tests against an explicit reference calculation.
• Replace the redundant alpha + alpha_time[0:4] decomposition with an identified parameterisation.
• Document and validate the revised intercept prior on the prior-predictive and estimand scales.
• Emit a level-family prior pushforward and surface t2 power-scaling warnings beside key findings.
• Run direct prior-grid sensitivity analyses for W, L, P, B, and N, or document why a warning does not require one.
• Persist and render the available-case modified-ITT scope and missingness assumption for the t2 causal term.
• Describe SP/RW coefficients as baseline-predictor associations, or separate the contemporaneous descriptive model.
• Introduce typed level-factor settings/run-plan validation before data loading and output reset.
• Fail before fitting if t2 lacks either randomised arm, required ability values are non-finite, or the requested causal contrast is otherwise unidentified.
• Make the factory/run plan the single source of truth for coefficient names, roles, diagnostic variables, and causal terms.
• Refit all eleven level-factor models and compare the revised t2 effects with the ITT, gain-factor, and arm-by-wave DiD estimates.
• Run ruff check src/, npm run format:check, npm run spellcheck, and the relevant statistical-model tests.

Verification completed during review

• 226 tests across factory, reporting, preprocessing, priors, and prior-inventory suites passed.
• Ruff passed on all level-factor and directly shared implementation files.
• All eleven saved reporting fits pass R-hat, ESS, BFMI, and divergence thresholds.
• LF-004 has unreliable conditional observation-level PSIS-LOO (max Pareto-k ≈ 1.001); W and E each have one observation above 0.7. This does not by itself invalidate the effect posterior, but the affected LOO scores should not be used as reliable model-comparison evidence without the stated sensitivity.