research: EP review phase 5 — reconcile the three deterministic-variable mechanisms

[Jammy2211]

Overview

Phase 5 of the EP framework review (umbrella: PyAutoMind/research/graphical_ep/ep_framework_review.md). PyAutoFit has three ways to express deterministic relations between model quantities, and they do not share machinery:

1. Graph-level deterministic variables — Factor(..., factor_out=v), propagated via FactorValue.deterministic_values, with dedicated quasi-Newton curvature transfer in the Laplace path. Exercised (as of Phase 3) by unit tests and the new autofit_workspace_test/scripts/graphical/ep_deterministic.py, but by no known real use case.
2. Compound prior arithmetic (autofit/mapper/prior/arithmetic/) — model-space composition (prior_a * x + prior_b), added by rhayes777 (PR to be located as part of this phase).
3. Free shared variables — share a prior across factors and encode the relation inside the likelihood (the IC50 cancer pattern: latents @ coef_matrix + coef_mean computed in log_likelihood_function).

Task (i)-(iv) of the original review asked: should anything be done about the low-level deterministic_variable API going unused by the real use cases? Output: a recommendation (unify / document both / deprecate one) with a migration-cost estimate, posted here for human decision.

Autonomy: --auto, effective supervised (research cap). Read-only investigation — no source edits; parks at the decision.

Plan

1. Locate the rhayes777 PR(s) that introduced compound prior arithmetic and any EP-side deterministic composition; reconstruct the intended design from the PR discussion.
2. Census each mechanism: implementation surface, EP interaction (does it create graph variables? messages? evidence contributions?), current users (source, tests, workspaces, HowToFit, IC50, cosmology), statistical semantics (is uncertainty propagated exactly, approximately, or not at all?).
3. Analyse the gaps: what each mechanism can express that the others cannot; where a user would be silently wrong choosing one over another.
4. Recommendation + migration cost estimate, posted as the phase verdict; decision (human) then feeds the fix/feature queue and Phase 2 docs get the outcome.

[Jammy2211]

Phase 5 verdict — census, analysis, recommendation (decision requested)

The rhayes777 PR located

#1153 "feature/declarative deterministic" (merged 2025-09-26): "illustration of creating declarative, deterministic graph using compound priors", plus the model.fwhm sugar (a class property becomes a CompoundPrior of the model's free priors). Notable: its test (test_autofit/graphical/test_declarative_deterministic.py) is entirely commented out — the declarative-deterministic pattern it introduced (an AnalysisFactor whose model is an af.Collection of compound priors drawn from other factors' models) has no protecting test on main. Compound prior arithmetic itself dates to #78 (2020) with majors at #517/#584/#974.

Census of the three mechanisms

	1. factor_out (graph API)	2. Compound priors	3. Free shared variables
Where	factor_graphs/{abstract,factor,jacobians}.py only	mapper/prior/arithmetic/	modelling pattern, no code surface
Reachable from declarative API (FactorGraphModel)	No — zero references in declarative/	Yes (#1153 pattern; model.fwhm sugar)	Yes (the default)
EP sees the deterministic quantity	Yes — own variable, own messages, cavity approx (FactorApproximation.deterministic_dist), Laplace curvature transfer (quasi_deterministic_update)	No — resolves at instance-creation inside the factor; zero references to Compound anywhere in autofit/graphical/	No — relation lives inside log_likelihood_function
Users found	4 unit-test files + docstrings + the new ep_deterministic.py integration script (Phase 3). No real use case.	#1153 illustration (test disabled)	IC50 cancer (latents @ coef_matrix + coef_mean in-likelihood); cosmology

The statistical analysis (the part that settles it)

Mechanisms 2/3 are statistically tighter, not a workaround: the deterministic relation is enforced exactly inside each factor's tilted-distribution fit, and message passing happens on the free components — no additional approximation is introduced. Mechanism 1 instead promotes z to a graph variable with its own message, which under the mean-field family means q(x)q(y)q(z) — the correlation between z and its parents is factorised away, an extra approximation. What mechanism 1 buys in exchange is per-factor dimensionality reduction and modularity: a downstream factor can constrain z without knowing (or re-evaluating) the relation, which matters for large structured graphs where the parent set is big. Neither dominates; they are different points on the exactness/modularity trade-off — which is why "the real use case doesn't use the low-level API" is not a defect.

Recommendation: (A) keep both, document the trade-off, protect the declarative pattern — no deprecation, no unification

1. Document the trade-off above in the Phase 2 autofit/graphical/README.md §6 (already drafted there in skeleton form on PR docs: formal Bayesian specification of autofit.graphical (EP review phase 2) #1334; a small follow-up sharpens it with the exactness-vs-modularity framing) and in user-facing docs: recommended default = compound priors / shared variables; factor_out = the low-level capability for structured graphs.
2. Resurrect PR feature/declarative deterministic #1153's commented-out test (or replace with a working declarative-deterministic integration script alongside Phase 3's ep_deterministic.py) so the compound-prior-through-EP pattern is protected. Small: the Phase 3 script provides the working template. (~half a day)
3. Keep factor_out + its Laplace curvature machinery, now guarded by ep_deterministic.py. No new declarative plumbing for it unless a use case demands z-level messages.

Rejected alternatives + costs: (B) expose factor_out through the declarative API — weeks of API design for no demanding use case, and it would steer users toward the looser approximation by default; (C) deprecate factor_out — small deletion (~35 references + the deterministic legs of laplace/newton.py) but permanently loses the only mechanism that gives deterministic quantities messages, foreclosing structured-graph scale-up for no maintenance win (the machinery passed the Phase 1 audit).

Decision requested

Approve recommendation A (→ files the small follow-up: README §6 sharpening + resurrected declarative-deterministic test), or direct B/C. IC50 needs no migration under A — its pattern is the recommended one.

[Jammy2211]

Recommendation A item 2 (protect the #1153 pattern with a working test) has been delivered on PR #1334 (commit 39eaf55) — it is decision-independent since it pins current behaviour. Investigation note for the A/B/C decision: the model.<property> compound sugar from #1153 was deliberately reverted by the maintainer (be6411755, 'undo annoying change', 2025-10-06), so the supported declarative-deterministic surface is the explicit compound pattern (model.sigma * 2.355), now documented in README §8 and pinned by the new seam tests. Remaining under decision A: only the README §6 sharpening (also now largely delivered via §8).

[Jammy2211]

Decision: Recommendation A adopted (human, 2026-07-10)

Maintainer decision: keep both composition paths, document the trade-off — no unification, no deprecation. Rationale as analysed: compound/shared-variable patterns are statistically tighter (the relation is exact inside factors); factor_out trades exactness for modularity (the deterministic variable gets messages and q(z) factorises from its parents). The model.<property> sugar from #1153 stays retired (deliberately reverted in be6411755); the supported declarative surface is the explicit compound pattern, pinned by the #1334 seam tests and README §8.

Delivered inventory: seam tests + lowering table (#1334, merged 6803441); IC50 needs no migration. Remaining: the README §6 sharpening — replace the 'open design item' flag with the adopted decision + trade-off statement. That doc edit ships with the in-flight EP statistics completion PR (F6 + F7b follow-ups from #1332). Closing this issue; the decision is recorded here.