spatial-signature shardmap reuse (#89)

[espg]

Closes #89.

Problem

A ShardMap is a spatial artifact (shard_keys + granule→shard assignment); it depends only on the spatial grid and the AOI/product/time query, not on what you aggregate. But the reuse guard over-constrained it:

• runner._check_signature (runner.py:287) did exact dict-equality between the run config's grid.signature() and the shardmap's stored grid_signature.
• signature() (both HealpixGrid and RectilinearGrid) includes output_fields = output_field_signature(config).

So two configs sharing the spatial grid but declaring different aggregation fields produced different signatures and the runner raised ValueError: ShardMap was built for a different grid than this run config — even though the shardmap content is byte-identical for both. Concretely, atl03_tdigest_healpix.yaml and atl03_gain_bias_healpix.yaml share parent_order 11 / chunk_inner 13 / child_order 19, so one shardmap maps both, yet you had to build (and CMR-fetch) two.

Fix

1. spatial_signature() on both grids — the structural fingerprint without output_fields (HEALPix: type, indexing_scheme, parent_order, child_order, layout; rect: type, crs, affine, shape, chunk_shape). signature() now composes spatial_signature() + {"output_fields": …}, so the two stay in sync and signature()'s output is byte-identical to before (verified — existing signature()/nests_with tests unchanged).
2. _check_signature compares the spatial signature, projecting the stored signature onto the spatial keys: stored_spatial = {k: expected.get(k) for k in grid.spatial_signature()}; raise only if stored_spatial != actual. The projection drops output_fields, so both newly-built spatial-only maps and old full-signature maps validate, and a map is reusable across agg fields. The expected is None early-return is kept.
3. ShardMap stores the spatial signature going forward — ShardMap.build now stores grid.spatial_signature() (field/JSON key stays grid_signature for back-compat; to_json/from_json round-trip unchanged). Docstrings updated.
4. nests_with is unchanged — it keys on output_field_signature(self.config) directly (the Support non-scalar aggregation outputs (vectors + ragged per-cell payloads) #29 co-aggregation guarantee), independent of the signature split.

grid_from_signature is not on main (it lives in the unmerged #44), so nothing consumes output_fields from the stored signature — no extra compat needed there.

Scope note

Orchestrator/catalog-side only — no Lambda redeploy. The worker never calls _check_signature; this only affects shardmap build + the run-time reuse guard.

Files

• src/zagg/grids/healpix.py, src/zagg/grids/rectilinear.py — add spatial_signature(); signature() composes it.
• src/zagg/runner.py — _check_signature projects + compares spatial only.
• src/zagg/catalog/shardmap.py — build stores spatial signature; docstrings.
• tests/test_runner.py, tests/test_shardmap.py, tests/test_runner_concurrency.py — new coverage + mock-grid stubs for the new method.

How tested

uv run pytest -q → 771 passed, 1 skipped. ruff check --select=E,F,W,I --ignore=E501 src tests clean. mypy unchanged (no new errors/categories vs main baseline).

New tests:

• Headline cross-aggregator reuse — a shardmap built from atl03_tdigest_healpix.yaml validates a run configured with atl03_gain_bias_healpix.yaml (same parent11/chunk_inner13/child19 spatial grid, different agg fields), and the reverse, with no raise.
• Real spatial mismatch still raises — different parent_order/child_order (HEALPix); different resolution/shape and different CRS (rect).
• Back-compat — an OLD-style stored grid_signature carrying output_fields (full signature) validates against a matching spatial grid and is reusable across differing agg fields (the projection handles it).
• nests_with semantics unchanged — covered by existing signature()/nests_with tests (still distinguishes output-field sets), plus spatial_signature() invariance-to-agg-fields tests.
• Round-trip — ShardMap.to_json/from_json preserves the spatial-only signature; _check_signature driven directly with built signatures.
• spatial_signature() excludes output_fields on both grids; signature() == {**spatial_signature(), "output_fields": …}.

Questions for review

• The reuse-guard ValueError now prints the projected stored_spatial (the spatial keys actually compared) rather than the raw stored dict, per the issue's stored_spatial != actual framing. If you'd prefer it also echo the raw expected, easy to add.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EN7X53ZAyaCca9rjwv9qZw

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

Fresh-context adversarial review of the spatial-signature shardmap-reuse change. No correctness bugs found — this is sound and well-tested; recommend merge after the one LOW nit (optional). Scrutiny summary:

• signature() byte-identity (HIGH-priority check) — PASS. Verified empirically on a HEALPix grid: {**spatial_signature(), "output_fields": …} yields the exact same keys, order, and JSON serialization as main's single-literal signature() (type, indexing_scheme, parent_order, child_order, layout, output_fields). Same holds structurally for rect (type, crs, affine, shape, chunk_shape, output_fields). nests_with and any persisted full-signature comparison are unaffected.
• nests_with unchanged — PASS. Both grids key it on output_field_signature(self.config) directly (healpix.py:351, rectilinear.py:281), never routed through signature()/spatial_signature().
• _check_signature projection edge cases — PASS. (a) old full-sig stored map validates a matching grid AND reuses across agg fields (projection drops output_fields); (b) new spatial-only map validates; (c) stored map missing a spatial key → expected.get(k) is None ≠ real value → raises (no silent None==None pass, since a real grid's spatial sig is non-empty); (d) genuine spatial mismatch (parent/child order for HEALPix, crs/affine/shape/chunk for rect) raises. The expected is None early-return is preserved; the empty-dict {} case only no-raises against a stub grid whose spatial_signature() is also {}, which is the existing test convention.
• ShardMap storage + round-trip — PASS. build now stores grid.spatial_signature(); to_json/from_json round-trip it; new maps no longer carry output_fields. Confirmed the only other full-signature() comparison in the tree is grids/__init__.py:102 (the nests_with error path), which never touches shardmap.grid_signature. grid_from_signature is indeed not on main.
• Rect parity — PASS. spatial_signature() excludes output_fields; cross-agg reuse works; resolution/CRS mismatch raises.

Local run: pytest tests/test_runner.py tests/test_shardmap.py tests/test_runner_concurrency.py tests/test_grids.py tests/test_rectilinear.py → 169 passed, 1 skipped.

---

Generated by Claude Code

[espg]

🤖 from Claude (review) — wrapping the single inline finding.

---

Generated by Claude Code

[espg]

🤖 from Claude (review) — one inline LOW nit on the reuse-guard error message (optional). No correctness issues; full review summary posted separately.

---

Generated by Claude Code

[espg]

🤖 from Claude (review) — LOW (debuggability nit, optional): the error now prints stored_spatial (the projected dict — i.e. the spatial keys the run grid declares, filled from expected with None for any key the stored map lacks). For the diagnostically interesting case — an old/incompatible stored map missing a spatial key — this renders that key as None and drops any extra keys the stored map did carry (e.g. a stored output_fields), so the message can obscure what the stored map actually was. Consider echoing the raw expected alongside the projection, e.g. f" shard map : {stored_spatial} (raw stored: {expected})\n". Purely cosmetic — the raise/no-raise logic is correct. (You already flagged this under "Questions for review"; noting it here for completeness.)

---

Generated by Claude Code