What
The ResStock manifest system does not record pipeline runs in the _sb release directory's manifest.yaml when the pipeline is invoked on the raw release. This means just status --state MD reports "No runs match filter" for the _sb variant even though _sb data for MD exists on both EBS and S3.
As a result, the EBS ↔ S3 sync check for _sb compares only the overall manifest files (which may have a single non-MD run), not the actual state-level data. Stale local parquets in the _sb release go undetected.
Why
This caused a real debugging detour: the local EBS copy of res_2024_amy2018_2_sb/metadata_utility/state=MD/utility_assignment.parquet had HIFLD canonical names in sb.electric_utility (e.g. BALTIMORE GAS & ELECTRIC CO), while the S3 copy had the correct std_name values (e.g. bge). The manifest reported "✓ in sync" because it was comparing manifest run_ids, not file contents — and the _sb manifest had no MD run recorded at all.
Downstream consumers (e.g. build-rate-case-test-year, scan_load_curves_for_utility) read from the local EBS path and silently get wrong data.
How
- When the pipeline runs steps that produce
_sb data (clone, modify_metadata, assign_utility, etc.), record the run in the _sb release's manifest.yaml as well — not just the raw release's manifest. The _sb manifest entry should include the state filter so just status --state MD shows the run.
- Consider adding a per-file or per-partition check (e.g. comparing S3 object etags or last-modified timestamps against local files) so the sync check catches individual file staleness, not just manifest-level
run_id mismatches.
Deliverables
- PR that ensures
_sb manifest records per-state pipeline runs
just status --state MD correctly shows the latest run for _sb and detects EBS/S3 divergence at the state level
What
The ResStock manifest system does not record pipeline runs in the
_sbrelease directory'smanifest.yamlwhen the pipeline is invoked on the raw release. This meansjust status --state MDreports "No runs match filter" for the_sbvariant even though_sbdata for MD exists on both EBS and S3.As a result, the EBS ↔ S3 sync check for
_sbcompares only the overall manifest files (which may have a single non-MD run), not the actual state-level data. Stale local parquets in the_sbrelease go undetected.Why
This caused a real debugging detour: the local EBS copy of
res_2024_amy2018_2_sb/metadata_utility/state=MD/utility_assignment.parquethad HIFLD canonical names insb.electric_utility(e.g.BALTIMORE GAS & ELECTRIC CO), while the S3 copy had the correctstd_namevalues (e.g.bge). The manifest reported "✓ in sync" because it was comparing manifestrun_ids, not file contents — and the_sbmanifest had no MD run recorded at all.Downstream consumers (e.g.
build-rate-case-test-year,scan_load_curves_for_utility) read from the local EBS path and silently get wrong data.How
_sbdata (clone, modify_metadata, assign_utility, etc.), record the run in the_sbrelease'smanifest.yamlas well — not just the raw release's manifest. The_sbmanifest entry should include the state filter sojust status --state MDshows the run.run_idmismatches.Deliverables
_sbmanifest records per-state pipeline runsjust status --state MDcorrectly shows the latest run for_sband detects EBS/S3 divergence at the state level