Skip to content

S1 RTC STAC builder fails on non-consolidated cube store (same-orbit time-slice append) #188

Description

@lhoupert

Symptom

build_s1_rtc_stac_item() (src/eopf_geozarr/stac/s1_rtc.py) opens the cube with zarr.open_consolidated(zarr_store, zarr_format=3), which raises:

ValueError: Consolidated metadata requested with 'use_consolidated=True' but not found in ''.

In the live S1 GRD RTC pipeline this fails STAC registration (register_v1_s1_rtc.py / register_per_acquisition.py), so the cube item never registers / the previous item is left deleted.

Trigger (reproduced)

The per-tile cube (s1-rtc-{tile}.zarr) accumulates one time slice per acquisition. The failure occurs specifically when a slice is appended to an existing same-orbit group (e.g. the 2nd+ ascending acquisition for a tile):

Scenario Result
Fresh single-acquisition store ✅ builds
Append a new orbit group (descending into an ascending store) ✅ builds
Append a slice to an existing same-orbit group (2nd ascending) ValueError: Consolidated metadata ... not found

Observed live on staging: 30TWM (descending appended into an ascending store) built fine; 31TEH (ascending appended into an existing ascending store) failed; a clean-slate 31TEH (fresh store) built fine.

Why it happens

The data-pipeline ingest does call consolidate_s1_store() after every acquisition (ingest_v1_s1_rtc.py:189,195s1_ingest.py:688-700, which consolidates the orbit group then the root). Yet after a same-orbit append on the S3 store, the root is read back without consolidated metadata, so open_consolidated raises. titiler reads the very same stores fine (its /info returns 200) because it does not require consolidated metadata.

Fix (done)

The builder must not require consolidated metadata — fall back to a direct zarr.open_group(zarr_store, mode="r", zarr_format=3) when it's absent (re-raising any other ValueError). Shipped in PR #180 (commit bbd18e3) with a regression test (tests/test_s1_stac.py::test_builds_from_non_consolidated_store: a 2-slice same-orbit store built with consolidate=False — fails before the fix, passes after).

Open sub-question (separate, not blocking)

Why does consolidate_s1_store() not leave the root consolidated after an S3 same-orbit append, despite being called? Likely S3 zarr-store consolidation behavior (note geozarr.py:1235 checks store.supports_consolidated_metadata). Worth tracking separately — the builder-tolerant fix unblocks production regardless, but losing consolidated metadata on appends forfeits its read optimization for all consumers that rely on it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions