Skip to content

Fix ResStock manifest to track _sb pipeline runs per state #484

Description

@alxsmith

What

The ResStock manifest system does not record pipeline runs in the _sb release directory's manifest.yaml when the pipeline is invoked on the raw release. This means just status --state MD reports "No runs match filter" for the _sb variant even though _sb data for MD exists on both EBS and S3.

As a result, the EBS ↔ S3 sync check for _sb compares only the overall manifest files (which may have a single non-MD run), not the actual state-level data. Stale local parquets in the _sb release go undetected.

Why

This caused a real debugging detour: the local EBS copy of res_2024_amy2018_2_sb/metadata_utility/state=MD/utility_assignment.parquet had HIFLD canonical names in sb.electric_utility (e.g. BALTIMORE GAS & ELECTRIC CO), while the S3 copy had the correct std_name values (e.g. bge). The manifest reported "✓ in sync" because it was comparing manifest run_ids, not file contents — and the _sb manifest had no MD run recorded at all.

Downstream consumers (e.g. build-rate-case-test-year, scan_load_curves_for_utility) read from the local EBS path and silently get wrong data.

How

  1. When the pipeline runs steps that produce _sb data (clone, modify_metadata, assign_utility, etc.), record the run in the _sb release's manifest.yaml as well — not just the raw release's manifest. The _sb manifest entry should include the state filter so just status --state MD shows the run.
  2. Consider adding a per-file or per-partition check (e.g. comparing S3 object etags or last-modified timestamps against local files) so the sync check catches individual file staleness, not just manifest-level run_id mismatches.

Deliverables

  • PR that ensures _sb manifest records per-state pipeline runs
  • just status --state MD correctly shows the latest run for _sb and detects EBS/S3 divergence at the state level

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions