Correct gain-factor estimands, floor-rule hierarchy and causal reporting

[frankbuckley]

Note

Drafted by a LLM-based AI tool (Codex/GPT-5).

Summary

A critical end-to-end review of the gain-factor statistical models found that the central ANCOVA/GLMM design is defensible and that the stored fits are computationally stable, but the family should not yet be treated as fully correct or publication-ready. Two implementation defects can change reported qualitative conclusions, and the current causal language does not fully reflect how treatment interactions are estimated.

This issue records the findings, proposed corrections and acceptance criteria. It covers the gain-factor factory, preprocessing, pipeline, reporting, predicted-score generation, model comparison and model specifications.

Confirmed implementation defects

1. Direction probabilities are calculated for beta_trt, not for the reported marginal effect

The reporting code calculates an interaction-aware period-1 average marginal effect (AME) on the probability scale, but prob_trt_pos, probability of direction and the derived evidence label use P(beta_trt > 0). These are different estimands whenever treatment interactions are active.

Relevant code:

• src/language_reading_predictors/statistical_models/reporting.py:2495-2520
• src/language_reading_predictors/statistical_models/reporting.py:457-513
• The gain-factor key-findings builder subsequently consumes the mismatched probability.

Independent recalculation from the stored reporting traces found material differences:

Model/outcome	Coefficient probability	AME probability
TR	0.633	0.928
TE	0.818	0.946
R, probability of harm	0.831	0.729

The reported AME median, interval and ROPE magnitude are largely correct, but several qualitative evidence labels and key-findings statements can change.

Recommendation: retain one period-1 AME value per posterior draw and use those draws consistently for the median, 50% and 89% intervals, probability of direction, ROPE classification and key-findings prose. Keep the raw coefficient probability only as an explicitly labelled coefficient-scale diagnostic if it remains useful.

2. The floor-rule model removes the own-baseline main effect but retains trt × own

For the Bernoulli off-floor likelihood, gamma_own is omitted while the treatment-by-own-baseline interaction is still constructed:

• src/language_reading_predictors/statistical_models/factories.py:3715-3739
• Affected registered specifications include LRP-RLI-GF-005 and LRP-RLI-GF-011.

This violates interaction hierarchy and forces own baseline to have exactly zero association among controls. Period-1 P data directly contradict that restriction:

Control baseline status	Off floor at post
At floor	2/17
Off floor	7/8

The P fit's gamma_int_trt_own posterior has median 0.804 and 89% interval 0.423 to 1.175, consistent with the interaction compensating for the omitted main effect rather than representing credible effect modification.

Recommendation: either include a suitably regularised baseline off-floor main effect, preferably using an interpretable pre-period off-floor indicator, or remove all interactions containing own. A period-1-only floor model would avoid treatment-affected later baselines and provide the cleanest causal specification.

Methodological issue requiring a design decision

3. Period-1 marginalisation does not isolate estimation to the randomised period

The main treatment contrast is identified by the randomised period-1 comparison. However, treatment–moderator interactions are estimated using all stacked periods. Later own-baseline measurements are post-treatment, and later periods have no untreated comparison. The mask at src/language_reading_predictors/statistical_models/pipeline.py:4521-4545 selects period-1 rows after fitting; it changes the population over which the posterior is averaged, not the observations that informed the interaction coefficients.

Consequently, the interaction-aware AME is partly model-dependent extrapolation and should not be described as exclusively randomised period-1 evidence or as a genuinely independent replication of the ITT result.

Recommended design: separate the two scientific questions:

1. A period-1 causal ANCOVA model, using only pre-randomisation moderators if effect modification is retained.
2. An all-period longitudinal association model for descriptive skill relationships and treated-period patterns.

A simpler alternative is to omit treatment interactions from the causal model and retain them only in the explicitly associational analysis.

Additional correctness and auditability findings

4. Predicted-score population labels do not consistently match the generated draws

In src/language_reading_predictors/statistical_models/predicted_scores.py:279-300, graded score simulations integrate a new child random intercept, but the probability-scale AME remains conditional on fitted child intercepts. The Bernoulli branch returns before new-child integration entirely. The resulting rows are nevertheless given one population label describing a new typical child with the child random intercept integrated out.

Because the inverse-logit transformation is nonlinear, a conditional sample-average effect and a population-average effect for a new child are not generally equal.

Recommendation: define and name the target explicitly, then apply it consistently. If both targets are useful, report separate observed_child_sample_ame and new_child_population_ame outputs rather than combining them under one label.

5. Cross-family comparisons use a coefficient rather than the gain-factor estimand

scripts/compare_statistical_models.py:334-349 reads raw beta_trt from factor_summary.csv. This is not the interaction-aware period-1 AME reported as the gain-factor headline and is not directly comparable with marginal estimands from other families.

Recommendation: make the comparison code consume the same canonical estimand artefact used by the model report, with scale and population recorded in explicit columns.

6. Gain-factor run plans and causal metadata are weakly typed

Gain-factor settings are stored in the generic ModelSpec.extra dictionary, and saved configurations leave fields such as design, estimand type, causal status and dataset reference unset. This makes contradictory settings easier to express and weakens the audit trail. Stored fits also use 53–54 children, so the causal target is an available-case period-1 population under an ignorable-missingness assumption, not automatically the complete randomised cohort.

Recommendation: introduce immutable GainFactorsSettings and GainFactorsRunPlan types following the ITT family pattern. Resolve and validate the run plan before output reset or data loading, and record the analysis population and missing-data assumption in config.json and model_recipe.md.

What the review found to be sound

The following aspects should be preserved:

• Post-score regression conditional on the pre-score is a defensible ANCOVA formulation.
• Group, phase and transition coding are internally consistent.
• Predictor sets match the declared revised-DAG parent sets.
• Count ranges and integer-valued outcomes are validated before fitting.
• The beta-binomial is a reasonable overdispersed working likelihood for bounded scores, and the off-floor Bernoulli estimand is appropriate for severe floor effects.
• Priors are regularising and substantively plausible.
• The non-centred child random intercept appropriately handles repeated-observation dependence, although it is not a control for latent general ability.
• Observational coefficients are generally labelled as associations rather than causal effects.

All 19 stored reporting fits passed the convergence gate: zero divergences, maximum R-hat below 1.005 and minimum effective sample size above 1,300. Four fits had cell-wise Pareto-k values above 0.7, so their LOO comparisons should be treated cautiously. These diagnostics establish reliable computation conditional on the model; they do not establish model-specification validity.

Refactoring recommendations

1. Add typed gain-factor settings and a resolved run-plan object with validation of allowed likelihoods, phase modes, covariates and interaction hierarchy.
2. Separate the period-1 causal model from the all-period association model in both code and reports.
3. Create one authoritative counterfactual engine returning clearly named posterior draws such as coefficient_effect, sample_average_effect and new_child_population_effect.
4. Move gain-factor construction and reporting from the large shared factories.py and pipeline.py modules into a family-owned module.
5. Replace near-identical outcome modules with a central typed outcome/DAG configuration table and thin registrations.
6. Derive coefficient inventories, diagnostic variables and report terms from the resolved run plan or built model rather than reconstructing them independently.

Acceptance criteria

• Probability of direction, ROPE classification and key findings are calculated from the same posterior AME draws used for the reported median and intervals.
• Tests reproduce the TR, TE and R direction probabilities from the stored traces or equivalent fixed fixtures.
• No registered specification contains an interaction whose required main effect is absent unless a documented parameterisation proves the hierarchy is represented another way.
• The P and N floor-rule baseline parameterisation is documented and covered by a simulation or recovery test.
• Causal treatment terms and treatment interactions are either estimated from period 1 only or the resulting estimand is explicitly described as model-dependent and partly informed by post-crossover data.
• Prediction CSVs distinguish observed-child conditional effects from new-child population effects and test both targets.
• Cross-model comparison consumes the canonical gain-factor estimand rather than raw beta_trt.
• Gain-factor configurations record the resolved design, estimand, causal status, analysis population and missing-data assumption.
• Registered-specification tests cover every primary and treated-only gain-factor model.
• Focused unit tests, ruff check src/, Markdown format checking and spelling checks pass.

Verification already performed

The review traced preprocessing, factory construction, priors, sampling, diagnostics, marginal effects, predicted scores, key findings and model comparison end to end. Fifty-one focused tests covering the factory, reporting, predictions and prior inventory passed, as did Ruff over the relevant source files. Those tests confirm existing arithmetic and structural expectations but do not currently cover the semantic defects above.

[frankbuckley]

Note

Drafted by an LLM-based AI tool (Claude Code/Fable 5).

Decision comment on the deferred findings (2–6), following the review of #395 (which fixes finding 1 correctly; see the review comment there, including the required --reuse-trace re-emit of the shipped gain-factor artifacts).

Finding 2 — floor-rule hierarchy (gf-005/011): drop trt×own and add a binary pre-period off-floor indicator main effect

Decision: drop ("trt","own") (and any other own-involving interaction) from the off-floor specs, and add a regularised binary off-floor-at-pre indicator as the baseline main effect. Not the "restore a graded gamma_own" option. Three reasons:

1. The control-arm data flatly contradict the implied zero own-baseline association (2/17 at-floor vs 7/8 off-floor moving off floor at post) — the restriction is untenable, so hierarchy must be restored.
2. The standardised pre-logit of a heavily-floored measure is a near-degenerate spike, so a binary off-floor-at-pre indicator is the honest functional form for the main effect — consistent with how the suite floor rule already thinks about these outcomes.
3. The A4 rationale for dropping gamma_own (later-period baselines are treatment-affected) applies with equal force to trt×own, so keeping the interaction was internally inconsistent with A4 itself.

Read the gamma_int_trt_own posterior (median 0.804, 89% CrI 0.423 to 1.175) as main-effect leakage, not effect modification — a treatment-by-baseline moderation claim on roughly 25 period-1 control children is not seriously estimable in any case. If an effect-modification question is retained at all, it should be trt × (binary pre off-floor) in an explicitly associational companion, not in the headline spec. Requires a refit of gf-005/011 plus their treated-only companions.

Finding 3 — interactions estimated on all stacked periods: make the causal headline interaction-free

Decision: remove the treatment interactions from the headline gain-factor specs and keep them in explicitly associational variants (variant_of), rather than building a separate period-1-only causal model. All 19 gf modules declare ("trt","ability") and ("trt","own"), so this is family-wide.

The bind is real and has no labelling-only exit: with fitted treatment interactions the AME must net them out (the earlier fix did exactly that) — but then the headline is partly identified by post-crossover data; not netting them out gives a wrong AME (the pre-fix bug). The clean exit consistent with the house discipline ("only the randomised term is causal") is an interaction-free headline. This is cheaper and more honest than a period-1-only fit, which would sacrifice the stacked structure that identifies the nuisance terms and still be underpowered for moderation.

Interplay to note: once adopted, the #395 direction fix becomes a no-op for the headline (without interactions the AME direction equals the coefficient direction draw-by-draw) but remains correct and necessary for the interaction variants.

Finding 4 — predicted-score population labels: two-name split, observed-sample conditional AME as the headline

Decision: adopt the explicit two-target naming (observed_child_sample_* vs new_child_population_*) and make the observed-sample conditional AME the headline. Verified in predicted_scores.py: the probability-scale quantities condition on the fitted child intercepts while the graded score simulation swaps in a fresh new-child intercept, and the Bernoulli branch returns before any new-child step — so a single shared label is genuinely wrong for somebody. The headline should be the observed-sample quantity because (a) it is what rope_summary and the key findings already report (the guard-tested AME), and (b) a "new child from the population" claim leans on the normality of the random-intercept distribution — an extrapolation the cohort statement does not need. Keep the new-child integration for the illustrative predicted-score displays, labelled as the population target, and give the Bernoulli branch either a real new-child path or an honest observed-sample-only label.

Finding 5 — cross-family comparison: move to the probability/items-scale AME over a common population; the logit-coefficient forest is not fixable by labelling

Decision: the triangulation should consume each family's canonical AME artefact on the probability/items scale, averaged over the common randomised transition (ITT prob_ame/items at t2, gf period-1 marginal, DiD t2 contrast), with explicit scale and population columns; keep a logit forest only as a clearly-caveated appendix, if at all.

The deeper problem the finding understates: comparing the gain-factor conditional coefficient (child random intercept + baseline adjustment) with the ITT marginal τ is confounded by logit non-collapsibility — the conditional coefficient is systematically larger in magnitude than the marginal one even with identical data, perfect randomisation and no interactions. So the current forest is apples-to-oranges regardless of which gf column it reads; switching it from beta_trt to the gf AME while leaving the other families on logit coefficients would not fix it either. Sequence this after the finding-3 decision, since that fixes what the gf canonical estimand is.

Finding 6 — typed GainFactorsSettings/RunPlan: endorse, sequence last

Decision: proceed, as the gain-factor instance of the already-planned Phase 3 family-owned migration, after the finding-2/3 decisions land. Doing it last lets the resolver encode the decisions as validated invariants rather than code-review catches: reject any interaction whose main effect is absent (finding 2 becomes structurally impossible), and record design, estimand, causal status, analysis population (the 53–54-child available-case cohort under the stated ignorable-missingness assumption) and dataset reference in config.json and model_recipe.md.

Sequencing and refit economics

Findings 2 and 3 both imply gain-factor refits — settle both first and run one reporting sweep. That sweep supersedes the #395 --reuse-trace re-emit for the refitted models; run the cheap re-emit promptly anyway if the refits will take time, because the shipped key-findings prose currently misleads in both directions (TR/TE understated, R-harm overstated). Then finding 5 (consuming the settled estimand), then finding 6 (encoding all of it).