research: root-cause the measure-thin solver branch flips in pixelized likelihoods

[Jammy2211]

Overview

Pixelized-source likelihoods contain deterministic single-float-input branch flips (width < 1e-15 in the parameter; ΔLL ≈ 1.577e-3 on the interferometer sparse 8×8 config, ~14 on imaging 28×28; same ΔLL for orthogonal parameter directions on a config; present under reg.Constant AND reg.Adapt; surface exactly linear elsewhere). Discovered during the kernel-CDF certification (#373, probes in that thread); they poison single-step FD (the jax_grad harness now runs an FD-step-sweep) and perturb any sampler landing on a bad float. Prime suspect: positive-only solver (PDIP/fnnls) iteration/tie-break knife-edges; related knob: nnls_solver_tol (Settings, default off, PR#371).

Plan

• Reproduce at the known bad input on current main (interferometer sparse variant-D config, gamma_2 = base − 1e-8; probe pattern on feat: kernel-density CDF transform for adaptive rectangular mesh #373).
• Instrument the solve: diff iteration count / active set / solution vector / residual between the bad float and adjacent clean floats; locate the branch point.
• Mechanism tests: does nnls_solver_tol move/remove the flips; does the unconstrained solve show any; does flip magnitude track tolerance.
• Deliverable: mechanism + recommendation here (tolerance/tie-break fix, documented LL accuracy floor, or won't-fix-documented). Small clearly-safe fix may ship behind the normal gates; anything larger files as its own task.

Read-only research on main (no worktree claim); probes in scratch; findings land here.

Original Prompt

PyAutoMind/research/autoarray/solver_branch_flips.md (verbatim in repo).

🤖 Generated with Claude Code

[Jammy2211]

Investigation findings (2026-07-10): mechanism class identified, solver exonerated, one concrete defect found and measured.

All probes on current main (post-#374/#375), interferometer sparse variant-D config, known flip at gamma_2 = base − 1e-8.

1. The flips are JIT-only. At the flip input: eager LL = −3164.0196392410 (exactly on the smooth linear trend), jit LL = −3164.0212166750 — the compiled graph, not the mathematics, takes the wrong branch. Everywhere else eager == jit. So these are point-localized eager-vs-JIT inconsistencies: an XLA fusion/reduction-order ulp difference crosses a discrete threshold and is amplified to finite ΔLL.

2. The solver is exonerated. The flip persists at the identical input with use_positive_only_solver=False (direct solve — no iterations, no active set; ΔLL 2.3e-4) and is byte-identical under nnls_solver_tol ∈ {1e-12, 1e-6}. Not PDIP/fnnls tie-breaking.

3. Interior bilinear crossings are continuous (measured: weight transfers cleanly across an interior integer-index crossing, centroid invariant to ±1e-13 perturbations), and at the flip input an isolated jit graph of the deflection→mapping chain produces indices/weights identical to eager (max diff 4e-15) — the knife edge is a property of the full fused likelihood graph, which is why localization-by-parts can't pin the exact op.

4. Concrete defect found: the linear rank-CDF forward transform is discontinuous at the data bounding-box edge. The empirical CDF maps the max point to N/(N+1) (= 0.99975 at N=4000) while the interp clamp maps anything beyond it to exactly 1.0 — a genuine U-jump of 1/(N+1) at the data max (measured: crossing the max point by 1e-13 redistributes 1.25e-3 of a query's scatter weight and shifts its drawn/index position discontinuously). Every dataset has queries at or adjacent to the bounding-box extremes, permanently on this knife edge. This is a defect of the linear meshes independent of #373; the kernel transform is continuous at its boundary by construction (its rescale hits exactly 1.0 AT the max), so the kernel-config amplifier is a different (still unlocalized) threshold in the fused graph.

Recommendations:

• Fix (small, concrete): make the linear empirical-CDF clamp consistent — e.g. rank values t = arange(1, N+1)/N (max point ↦ exactly 1) or interp right=t_max — removing the 1/(N+1) edge jump. Needs an FoM-impact check (it perturbs all rank positions by O(1/N)); should ship as its own gated change, not a rider.
• Kernel-side localization (follow-up): in-graph instrumentation (jax.debug.print/io_callback on boundary-adjacent floor/ceil inputs inside the actual fitness graph) at the flip input, or progressive subgraph compilation until the flip reproduces.
• Meanwhile (already shipped): FD certification uses the step-sweep; for samplers these are measure-zero spikes (isolated float inputs) — practical impact is a documented LL accuracy floor under jit of ~1e-3 (interferometer 8×8) to ~14 (imaging 28×28) at pseudo-random isolated points, worth keeping in mind for evidence estimates.

Probes are reproducible from the patterns in this thread + #373; scratch scripts not committed (research task, read-only on main).