Refactor rate-run orchestration with Prefect

[alxsmith]

Overview

Redesign the Prefect pipeline from scratch around a generic quartet-based orchestration model. The new pipeline replaces the old 1868-line run_pipeline.py with three focused modules (~1800 lines total) that scale cleanly across scenarios, utilities, and tariff structures.

Key changes:

• pipeline_config.py — Centralized config: PipelineConfig, ScenarioConfig, RunDefaults dataclasses, YAML loader, deterministic naming functions for tariff stems, map stems, run names, and all derived paths.
• run_pipeline.py — Prefect flows (run_batch → preflight → run_quartet → derive_tariffs), cairo_run task with subprocess isolation, threading.Semaphore concurrency gate, run index resume.
• pipeline_derive.py — Structure handler registry (seasonal, flat, + future TOU) using a dispatch pattern. Adding a new tariff structure = one function + one dict entry.
• run_scenario.py — Widened --run-num to accept string canonical names; writes a run index file on success for cross-invocation resume.
• Pipeline YAML — Declarative per-utility config (pipeline_bge.yaml, pipeline_rie.yaml) that defines scenarios, quartet kinds, subclass configs, and all inputs in one place.

Reviewer focus

• Naming convention correctness: trace a scenario through tariff_stem(), _scenario_map_stem(), _resolve_tariff_paths() and verify the derived paths match what CAIRO and downstream analysis expect.
• Concurrency model: threading.Semaphore + ThreadPoolTaskRunner(max_workers=2) for delivery/supply parallelism within each stage.
• Subprocess invocation: CAIRO runs as a child process for memory isolation; run index file is the resume/discovery mechanism.
• derive_tariffs flow: explicit failure on missing allocation or RR values (no silent fallbacks).

Non-obvious implementation details

• Run index over Prefect result cache: filesystem .runs/{name}.path files replace Prefect built-in caching. This gives cross-invocation resume without depending on Prefect server state.
• Map stems exclude residual allocation: tariff maps partition buildings by subgroup, which is allocation-independent. Only tariff JSON stems encode both delivery and supply allocations.
• tariff_base field: Required for single_rate quartets to decouple the tariff filename from the scenario name. Optional (unused) for multi-rate.
• output_base in config: Batch output root is configurable rather than hardcoded, enabling local vs S3/FUSE paths.
• Supply MC zeroing: For delivery-only runs, supply MC paths have /data.parquet replaced with /zero.parquet to zero out supply costs without branching logic in CAIRO.

[alxsmith]

this is bad. flag for generalizing

[alxsmith]

same here. needs upstream cleaning