fix(s1-rtc): consolidate every orbit group, not just the last-ingested one#203
Merged
lhoupert merged 1 commit intoJun 23, 2026
Merged
Conversation
…d one S1 RTC cubes ended up with only one orbit consolidated on disk (staging: 30TWM asc✓/desc✗, 32TNN/30UVU asc✗/desc✓). Root cause: the pipeline ingests acquisitions one orbit at a time in separate pods, each of which strips all consolidated metadata (so `time` can resize), ingests its orbit, then calls `consolidate_s1_store(store, orbit_direction)` — which only re-consolidated the single orbit passed. So whichever orbit was ingested last is the only one left with on-disk consolidated metadata. Consolidate every orbit group present (iterate `root.groups()`), then the root. The minimal append-fetch already pulls all `zarr.json`, so both orbits' metadata is local — same reason the root consolidation already works. Signature unchanged (`orbit_direction` kept for logging/callers). Impact is cosmetic: the root consolidation is complete for both orbits and readers opening at the root get a synthesized child view (titiler renders the unconsolidated orbit fine). The fix matters for clients opening a single orbit group standalone (the STAC `<cube>.zarr/<orbit>` hrefs) with `consolidated=True`, which otherwise fall back to a listing. Test asserts each orbit group is consolidated *standalone* — checking via `root[orbit]` is a false-green because a consolidated root synthesizes the child's view. Existing cubes self-heal on their next re-ingest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
S1 RTC cubes end up with only one orbit consolidated on disk — staging shows 30TWM asc✓/desc✗, 32TNN & 30UVU asc✗/desc✓.
Root cause (traced + reproduced): each acquisition is ingested by a separate pod that strips all consolidated metadata (so
timecanresize), ingests its single orbit, then callsconsolidate_s1_store(store, orbit_direction)— which only re-consolidated the one orbit passed in. So whichever orbit is ingested last is the only one left with on-disk consolidated metadata.What
consolidate_s1_storenow consolidates every orbit group present (root.groups()), then the root. Signature unchanged —orbit_directionis kept for logging / caller compatibility (no changes to the CLI or the data-pipeline callers).Safety: the pipeline's minimal append-fetch already pulls all
zarr.json(ingest_v1_s1_rtc.py:336-337); only bulk chunks are skipped. So both orbits' metadata is local when this runs — the same reason the root consolidation already produces a correct both-orbit view.Impact — cosmetic, not a correctness/deploy blocker
The root consolidated metadata is already complete for both orbits, and a consolidated root synthesizes a child orbit's view — so titiler (opens at root, selects via
/orbit:vv) renders the unconsolidated orbit fine (verified: 30UVU ascending = HTTP 200 despite asc✗). The fix matters for clients that open a single orbit group standalone (e.g. following the STAC<cube>.zarr/<orbit>asset hrefs) withconsolidated=True, which otherwise fall back to a listing. Existing cubes self-heal on their next re-ingest.Tests (TDD, RED→GREEN)
tests/test_s1_rtc_ingest.py::TestConsolidation::test_consolidate_all_orbits_present— ingests both orbits, callsconsolidate_s1_store(store, "descending")(one orbit passed), asserts both orbit groups are consolidated standalone.RED on
ac3167f, GREEN after the fix. Full data-model suite green; ruff/mypy add zero new findings vs baseline.Stacks on #202; targets #180.
🤖 Generated with Claude Code