test_xipm_theo fails under numpy>=2.5 (pyccl 3.3.4 float(array)); revisit where cosmology code lives

[cailmdaley]

Part A — immediate CI breakage: test_xipm_theo under numpy ≥ 2.5

cs_util/tests/test_cosmo.py::CosmoTestCase::test_xipm_theo fails under numpy 2.5.0 + pyccl 3.3.4 (Python 3.12):

TypeError: only 0-dimensional arrays can be converted to Python scalars
  at pyccl/boltzmann.py:119

The same test passes on Python 3.10, where the resolver picks numpy 2.2.6. The break is numpy 2.5 turning pyccl 3.3.4's float(<non-0-d array>) (boltzmann.py:119) from a deprecation warning into a hard TypeError. In our test it is reached through cs_util.cosmo.xipm_theo → ccl.WeakLensingTracer / ccl.angular_cl / ccl.correlation. This is an upstream pyccl-vs-numpy-2.5 incompatibility, not a cs_util bug.

Current stopgap (NOT the fix). test_xipm_theo is xfailed, conditioned on numpy ≥ 2.5:

@pytest.mark.xfail(
    np.lib.NumpyVersion(np.__version__) >= "2.5.0",
    reason="pyccl 3.3.4 calls float() on a non-0-d array in boltzmann.py, "
           "which numpy>=2.5 turns into a hard TypeError; not a cs_util bug. "
           "Stopgap, not the real fix. ...",
    strict=False,
)

This keeps CI green on both numpy lines but silently drops xi+/- coverage on numpy ≥ 2.5. A proper resolution should be one of:

• a pyccl release that supports numpy 2.5 (upstream fix to the float(array) call) — then delete the xfail;
• a deliberate, documented numpy < 2.5 pin — accepting the cost of falling behind numpy;
• or removing the cs_util cosmology layer entirely (see Part B) — if cosmology doesn't belong here, the test goes with it.

The point: this needs a real decision, not a permanent xfail.

Part B — where should the cosmology code live?

While investigating Part A, a broader smell surfaced: cosmology theory code lives in (at least) two places in the UNIONS stack, with no shared function between them — each rolls its own ccl-based xi+/- recipe.

What's in each place

cs_util/cosmo.py — small, reusable, tested primitives, wrapping pyccl:

• get_cosmo_default() — a fixed default ccl.Cosmology (+ spline params for the correlation integral);
• sigma_crit / sigma_crit_eff / sigma_crit_m1_eff — (effective) critical surface mass density for g-g lensing;
• xipm_theo(theta, cos, z, dndz) — theory shear 2PCF via WeakLensingTracer → angular_cl → ccl.correlation(type="GG±").
  All exercised by cs_util/tests/test_cosmo.py.

sp_validation/cosmo_inference/ — re-implements the same ccl xi+/- recipe inline, never importing cs_util.cosmo, in its inference/MCMC pipeline:

• cosmo_inference/scripts/chain_postprocessing.py:237-249 — builds ccl.Cosmology(...) then ccl.correlation(..., type="GG+") / "GG-";
• cosmo_inference/notebooks/cfis_mcmc.ipynb (≈544-667) — ccl.Cosmology + WeakLensingTracer + angular_cl + ccl.correlation GG±, the actual MCMC fit;
• cosmo_inference/notebooks/cfis_analysis.ipynb:443-457 — same pattern.

So the duplication is conceptual (the same pyccl xi+/- computation written twice), not a shared call. sp_validation's version is the inference-grade one — transfer_function='boltzmann_camb', halofit_version='mead2020_feedback' — and is embedded in chain-postprocessing / MCMC rather than factored into importable functions. cs_util's is the simpler library version.

Usage evidence (who actually imports cs_util.cosmo)

• shear_psf_leakage (CosmoStat) DOES use it. shear_psf_leakage/run_scale.py:19 from cs_util import cosmo as cs_cos, used at run_scale.py:50-51 in get_theo_xi(): cs_cos.get_cosmo_default() + cs_cos.xipm_theo(theta, cosmo, z, nz). run_scale is central to that package — imported by leakage_scale.py:5 and several notebooks (leakage_scale_spin.py:44 instantiates run_scale.LeakageScale(), leakage_xi_sys.py:104, leakage_scale_test_weights.py). This is a live consumer.
• sp_validation does NOT import cs_util.cosmo — it pulls other cs_util submodules (logging, cat, args, plots, calc) but never cosmo; it re-rolls the xi+/- recipe itself (above).
• shapepipe does NOT import cs_util.cosmo.
• Inside cs_util, cosmo is referenced only by its own test.
• sp_validation's inline cosmology is used by its own cosmo_inference pipeline (chain_postprocessing.py, cfis_mcmc.ipynb).

Design question + a proposal (not a decree)

Consolidate to one home. The original framing was "what if cosmology lived only in sp_validation, dropping cs_util.cosmo?" — but the evidence points the other way:

• cs_util.cosmo's only consumer is shear_psf_leakage, a lightweight CosmoStat library; sp_validation never touches it. Moving the primitives into sp_validation would invert the dependency arrow — shear_psf_leakage would have to depend on a heavy validation/inference repo for a two-line theory call. That's the wrong direction.
• The cleaner consolidation is the reverse: keep the small, reusable, tested cosmology primitives in the library (cs_util.cosmo), and have sp_validation/cosmo_inference consume them instead of re-implementing the ccl xi+/- recipe inline. sp_validation's inference-grade extras (camb transfer function, mead2020 halofit feedback) become parameters of an extended cs_util.cosmo.xipm_theo rather than a parallel copy.

So my read: cs_util.cosmo does have a real external consumer, so "cosmology only in sp_validation / drop cs_util.cosmo" is not viable as-is — it would break shear_psf_leakage. The de-duplication worth doing is sp_validation converging onto cs_util.cosmo, not the reverse. Happy to be argued out of this — flagging it as a design question rather than settling it.

— Claude on behalf of Cail

[martinkilbinger]

@claude try to replace the duplicated cosmology functions (e.g. xi_pm) in sp_validation/cosmo_inference by cs_util.cosmo .