Milestones for version 1 release

[espg]

There's lots:

1. Fully working temporal aggregation ( Temporal aggregation infrastructure (Steps 1-3) #15 / Temporal aggregation #12 , and follow-up work)
2. Robust backend implementation ( i.e., Backend support (i.e., local processing support) #20 )
3. Additional scaling capabilities (i.e., step functions, no issue for this yet)
4. Fold in mortie version 1.x ( Integration with pending version 1 of mortie #57 )
5. Current documentation and benchmarks
6. Robust deployment CI/CD (i.e., move lambda testing into the CI/CD -- NOT mocked, but actually run)

Probably missing others as well.

#59 is post version 1, but adding it here so it still gets drawn into the discussion below about synergy between modules.

[espg]

Claude, the reason for this thread is to identify synergies between threads. Generally, when you build a work list to go over issues, there's one sub-agent per issue. This is great for pushing things forward, but it can lead to siloed design decisions. For this discussion, your role is to take a broader view-- i.e., to read up on all the issue threads and PR's listed in the issue summery here, and identify common threads or useful potential abstractions that could work across the library and it's modules.

Do you have any suggestions or thoughts, looking at the major bits of work that remain above, and how we might better design them to work together?

[espg]

🤖 from Claude

Cross-cutting synthesis — common threads and useful abstractions across the major work items:

Reading #12, #20, #52, #53, #54, #56, #57, #59, and the recent PR threads (#41, #44, #47), here are the cross-cutting themes:

---

1. Dispatcher abstraction as shared infrastructure (#20, #12)

The most impactful single investment. Both temporal (#12) and future cluster backends (#20) need the same generic fan-out → retry → measured-cost loop that currently lives as two bespoke functions in runner.py. A dispatch.py with a clean Executor protocol extracts this once. Every pipeline type (spatial, temporal, event) then inherits Lambda and local execution for free, and items 3–4 of #20 (vaex, ray/dask/slurm) plug in behind the same interface without touching pipeline logic.

This is the prerequisite that unblocks both #12 Phase 1c and #20 items 3–4.

---

2. Plugin registry as shared vocabulary (#12, #59)

Phase 1a of #12 introduces registry.py with named registries for spatial functions, reducers, mask providers, etc. This is also the capability-enumeration surface the MCP server (#59) needs to describe available aggregation options — registry.list_spatial_functions() directly feeds the MCP describe_products tool.

Shared design decision worth making early: all capability references stay as names (strings), never callables, so they serialize in YAML/JSON for both Lambda payloads and MCP tool parameters. This invariant must hold across the temporal engine, the plugin mechanism, and the MCP server.

---

3. Mortie v1 as a coherent integration milestone (#57, #53, #54)

Three separate issues all resolve together when mortie v1 ships: the uint64→morton_index migration (#57), the plan_read grid-aware linestring decision (#53, deferred until API stabilizes), and the spherely SpatialIndex brute-force removal (#54, if SpatialIndex ships with v1). Treating these as a single "mortie v1 integration" PR is cleaner than three separate PRs chasing the same upstream event.

Key design question to settle with the mortie team before v1: does morton_index implement __int__/__array__? The answer determines whether zagg needs a thin boundary wrapper or a more invasive propagation.

---

4. Deprecation cleanup (#52) as v1 API hygiene

process_morton_cell, the "dense" layout default in HealpixGrid, xdggs_zarr_template(n_parent_cells=...) — these are independent cleanup items with no design decisions, just execution. They should land before the v1 tag since the deprecation warnings are loud in user code and the 1.0 surface should be clean.

---

5. Mid-run concurrency re-check (#56) as an operations concern

Orthogonal to the architecture questions above. The IAM grant (cloudwatch:GetMetricStatistics in template.yaml) is the minimal useful action; the re-probe logic matters only at scale. Neither is a v1 blocker but the grant is cheap.

---

Suggested sequencing toward v1:

Order	Item	Why
1	Deprecation cleanup (#52)	Clean API surface before 1.0
2	Executor abstraction (#20 + #12 Phase 1c)	Foundational for temporal and multi-backend
3	Temporal engine (#12 Phases 0, 1, 2)	Builds on the dispatcher
4	Mortie v1 integration (#57 + #53 + #54)	When upstream ships
5	MCP server (#59)	After v1 API stable
Anytime	IAM grant in template.yaml (#56)	Cheap, independent

Biggest design risk: the Strategy × Executor refactor in #12 Phase 1c touches the spatial production path. It needs a golden ATL06 run regression test before landing and will be the most scrutinized PR of the cycle. Plan for it to be reviewed against a real run, not just unit tests.

---

Generated by Claude Code