Complete the statistical pipeline family split and artefact lifecycle refactor

[frankbuckley]

Note

Drafted by a LLM-based AI tool (Codex/GPT-5).

Summary

Complete the remaining statistical-pipeline refactor by turning pipeline.py into a thin compatibility facade, moving orchestration into family-owned modules, and centralising the primary-fit, sub-fit and artefact lifecycles. This is the structural follow-up to #361's Phase 3 foundation and complements the ITT correctness work in #392.

The work should be incremental and behaviour-preserving. It must not silently change an estimand, likelihood, prior, analysis population, fitted equation, sampling preset, diagnostic threshold or published artefact schema. Statistical or release-policy corrections should be reviewed separately from mechanical relocation.

Review evidence

The current src/language_reading_predictors/statistical_models/pipeline.py contains:

• 9,102 lines, 117 top-level functions and 25 public fit_* entry points.
• 179 runnable model modules that import it directly.
• 26 primary branches that independently spell out substantial parts of the prior-predictive, posterior-sampling, LOO, diagnostics, posterior-predictive, trace-persistence and sensitivity sequence.
• 102 direct CSV writes and 98 manual ctx.tables[...] registrations, with 78 distinct output filename literals.
• 158 references to untyped spec.extra settings and substantial family-specific reliance on untyped BuiltModel.extras.
• Several very long orchestration functions: the ITT floor-rule branch is 423 lines, gain factors 323, adjusted 319, concurrent 314, LCSM 291 and correlated factors 287.
• 27 broad except Exception handlers, mostly intended to protect expensive fits from optional plotting failures but without a common structured record of skipped artefacts.
• Specialised statistical algorithms, including the longitudinal correlated-factor exact child-level log-likelihood and constrained-scale log-prior recovery, embedded beside console, plotting and Quarto-publication helpers.

The existing stages.py boundary is a useful start, but it currently centralises only attachment, sampling plus optional LOO, posterior prediction, metadata and finalisation. Summary diagnostics, the all-free-variable convergence gate, trace persistence, prior-versus-posterior output and power-scaling sensitivity remain manually sequenced by each family. The module therefore does not yet fully own the invariant lifecycle described in its own documentation.

Recommended design

1. Make pipeline.py a compatibility facade

Move each family entry point to a family-owned orchestration module, for example statistical_models/pipelines/itt.py, joint.py, did.py, gain_factors.py, mediation.py, concurrent.py and longitudinal_corr_factor.py. Keep pipeline.py as explicit re-exports until all model modules and tests have migrated.

Do not move the 423-line floor branch into the existing itt.py specification/plan module; that would create another mixed-responsibility monolith. Keep settings, run-plan resolution and audits in itt.py, with orchestration in pipelines/itt.py and substantial floor-specific outputs in a focused companion module if required.

2. Complete the shared primary-fit lifecycle

Introduce a typed PrimaryFitPlan describing the genuinely variable execution choices:

• Curated diagnostic variables.
• Posterior-predictive nodes and primary node.
• Prior-predictive plot node.
• Causal or focal term.
• Variables requiring power-scaling sensitivity.
• Whether PSIS-LOO is computed and whether LOO-PIT applies.

Use it to centralise the invariant sequence of prior prediction, sampling, optional LOO, human-readable diagnostics, posterior prediction, the all-free-variable convergence gate, extended diagnostics, trace persistence, prior/posterior comparison and sensitivity. Keep family scientific summaries and exceptional audits explicit.

Avoid a base class with many overridable methods or a lifecycle controlled by numerous callbacks. The shared abstraction should expose the common mechanics without hiding genuine differences such as mediation's lack of ordinary LOO, concurrent multi-wave sub-fits, or the longitudinal correlated-factor likelihood workaround.

3. Centralise artefact persistence and release evidence

Add an ArtifactStore or equivalent service so a single operation writes a table, registers it on the run and validates any required columns. It should also record required, optional, present and skipped outputs in a structured artefact manifest.

Classify optional visualisation failures separately from required publication artefacts. Optional failures may warn and continue, but the failure type and message should be persisted. Required outputs must fail the unpublished transaction or produce an explicit withheld release decision.

Make report finalisation consume a structured release decision before generating key_findings.json. This should provide the architectural home for #392's convergence, prior-sensitivity and floor-sensitivity release requirements rather than distributing policy across pipeline functions, report readers and Quarto partials.

4. Extend typed settings and resolved plans

Apply the ITT IttModelSettings / IttRunPlan pattern to the remaining families. A resolved family plan should be the single source for loader arguments, effective adjustment, factory arguments, likelihood and observation node, LOO unit/policy, diagnostic variables, causal or associational focal term, and required release evidence.

Suggested order: joint ITT; DiD; gain/level factors; mediation; adjusted/concurrent; growth/historical/measurement families.

Retain _default_of() until typed family defaults provide a replacement single source of truth. Do not replace the current anti-drift reflection with duplicated numeric literals.

5. Consolidate secondary and sensitivity fits

Replace the independent floor-rule, mediation, adjusted and concurrent sampling paths with a SubfitRunner that returns a typed SubfitResult containing the trace, convergence verdict, fitted-data identity, sampling settings, optional posterior prediction/LOO, persisted trace filename and any structured failure.

The family pipeline should still show the scientific loop explicitly—for example, which waves or predictors are being fitted—while delegating only common sampling, convergence, persistence and provenance mechanics.

6. Separate computation from presentation

Refactor family summaries into pure or mostly pure functions that return typed results or DataFrames. Persist tables, draw figures and print console summaries in separate artefact/presentation functions. This will reduce the need for tests to monkeypatch private members of the monolithic pipeline and will make posterior-summary calculations testable without an output directory, Quarto template or Matplotlib session.

Move shared prior artefacts, PPC artefacts, treatment-effect/ROPE figures, trajectories and report-template publication into cohesive modules named by responsibility. Avoid splitting every small helper into its own file.

7. Isolate specialised statistical algorithms

Move the longitudinal correlated-factor exact child-level log-likelihood, constrained-scale log-prior recovery and LOO stitching into an LCF-specific likelihood or inference module. These routines are substantive numerical algorithms and should be testable independently of report publication and terminal output.

Promote any cross-module private interfaces used by pipelines, such as the ITT average-marginal-effect draw helper, to supported public functions before moving their callers.

8. Simplify run state after the boundaries are established

Replace substantial BuiltModel.extras dictionaries with typed family payloads. Consider state-specific fitted-run results or at least fail-loud context accessors for required model/trace/LOO state.

Repository-wide searches currently show that StatisticalFitContext.model_vars and BuiltModel.variables are copied during attachment but have no production consumer. Remove them in a dedicated clean-up after compatibility tests confirm that no external caller relies on them.

Suggested implementation sequence

1. Add characterisation tests for lifecycle order, fitted row identities, model variable names, artefact names and required columns, metadata keys and report publication behaviour.
2. Extract common artefact helpers without changing behaviour; retain compatibility re-exports.
3. Introduce the artefact store/manifest and explicit release-decision boundary.
4. Complete the shared primary-fit lifecycle and migrate one ordinary family as the reference implementation.
5. Move ITT and joint orchestration, preserving the resolved plan, model recipe, analysis audit, floor branch and output schema.
6. Migrate coherent family groups: DiD/factors/aligned/block exposure; mechanism/dose/mediation; adjusted/concurrent/horseshoe; growth/measurement/historical.
7. Add every migrated typed family module to the incremental MyPy gate.
8. Update model modules and tests to import family entry points directly, then retire compatibility wrappers and dead context fields only after repository-wide import searches and full tests pass.

Keep mechanical relocation and behavioural changes in separate commits or pull requests so equation or output changes cannot be hidden inside large file-movement diffs.

Acceptance criteria

• pipeline.py is a small documented compatibility facade with no family-specific statistical calculations.
• Every family has an identifiable orchestration module that an unfamiliar reader can follow from declaration through preparation, construction, inference, summaries and finalisation.
• The invariant primary-fit lifecycle is expressed once and covered by ordering tests.
• Every published table is written and registered through one artefact interface.
• Required and optional artefacts are declared, validated and recorded in a structured manifest.
• Report finalisation receives an explicit release decision before key findings are generated.
• Every secondary or sensitivity fit records convergence and sampling provenance through one shared runner.
• Typed settings and resolved plans replace spec.extra for each migrated family, with unknown and contradictory settings rejected before data loading or output reset.
• Family summary calculations can be tested independently of plotting, console output and Quarto publication.
• LCF likelihood/log-prior recovery is isolated and independently tested.
• Compatibility imports remain until all repository callers have migrated.
• Structural pull requests preserve prepared rows, model variables, priors, likelihoods, diagnostic thresholds, sampling settings, metadata keys and artefact schemas.
• Focused tests, the full test suite, Ruff, MyPy, Markdown formatting and spelling checks pass.

Related work

• Improve statistical-model readability, reproducibility and teaching clarity #361 established the Phase 3 migration boundary and ITT reference plan but closed before the remaining family-module migration was completed.
• Harden ITT release gates, longitudinal integrity, and joint summaries #392 tracks ITT release gates, longitudinal data integrity and joint-summary correctness. This issue provides the maintainable pipeline boundaries needed to implement those policies consistently, but it should not conceal their statistical corrections inside structural relocation work.