Skip to content

Fix NEE flat-file var read; registry overlay hygiene; min_year default#129

Merged
zhongwangwei merged 1 commit into
mainfrom
fix/nee-flat-var-and-registry-hygiene
Jun 12, 2026
Merged

Fix NEE flat-file var read; registry overlay hygiene; min_year default#129
zhongwangwei merged 1 commit into
mainfrom
fix/nee-flat-var-and-registry-hygiene

Conversation

@zhongwangwei

Copy link
Copy Markdown
Owner

Summary

Three fixes from a debugging session on the persistent Net_Ecosystem_Exchange evaluation error (plus related cleanups).

1. NEE flat-file variable resolution (the actual bug)

Preprocessing correctly derives NEE from f_respc/f_assim and relabels the saved variable to the evaluation item (Net_Ecosystem_Exchange) for output-label correctness. But the flat file is named with the config varname (f_respc), and downstream readers hard-indexed ds[sim_varname] → xarray No variable named 'f_respc'. Variables on the dataset include ['Net_Ecosystem_Exchange', 'time', 'lat', 'lon'].

  • Add util.names.select_data_array(ds, *names): resolve by name (case-insensitive), else fall back to the sole data variable (flat files have exactly one).
  • Use it in runner/masking.py (the preprocess crash), core/evaluation.py (grid scoring), and all 7 core/_comparison_*.py modules.
  • Write-side relabel to the item is intentional and unchanged.

2. Registry overlay hygiene — openbench registry diff|prune

A legacy full-snapshot ~/.openbench overlay silently shadows bundled catalog fixes (this is why the CLARA albedo unit fix didn't take effect until the overlay was reset). New tooling:

  • Classify overlay entries: redundant / stale full-copy / delta / custom.
  • registry prune re-sparses behavior-preservingly (drops redundant, minimizes stale full-copies; merged registry identical before/after).
  • Throttled, silent-when-clean startup notice; suppress via OPENBENCH_NO_REGISTRY_CHECK.

3. min_year_threshold default → 1

schema + loader default 3 → 1 (matches GUI/smoke); init now writes 1 for every project span.

Testing

  • New: tests/test_flat_var_resolution.py (reproduces the exact error + end-to-end masking read), tests/test_registry_overlay_audit.py.
  • Updated test_cli_stubs.py, test_config/test_schema.py for the new defaults/command.
  • Full suite: 1532 passed, 5 skipped. ruff clean.

Remote sync note

After pulling + reinstalling on the HPC, run openbench registry prune once to un-shadow the bundled catalog (the remote ~/.openbench overlay is separate).

Flat-file variable resolution
  Preprocessed flat files store the relabelled evaluation-item variable
  (e.g. Net_Ecosystem_Exchange derived from an f_respc fallback/convert),
  but downstream readers hard-indexed ds[sim_varname] (f_respc) and raised
  xarray "No variable named 'f_respc'". Add util.names.select_data_array
  (resolve by name case-insensitively, else fall back to the sole data var)
  and use it in runner/masking, grid evaluation, and all _comparison_*
  modules. The write-side relabel to the item is intentional (output labels)
  and unchanged.

Registry overlay hygiene  (openbench registry diff/prune)
  Classify the user ~/.openbench overlay as redundant / stale full-copy /
  delta / custom and re-sparse it behavior-preservingly, so a legacy full
  snapshot no longer silently shadows bundled catalog fixes. Adds a
  throttled, silent-when-clean startup notice (suppressible via
  OPENBENCH_NO_REGISTRY_CHECK).

min_year_threshold default -> 1
  schema + loader default 3 -> 1 (matches GUI/smoke); init writes 1 for
  every project span.
@zhongwangwei zhongwangwei force-pushed the fix/nee-flat-var-and-registry-hygiene branch from 4379b98 to f4d0511 Compare June 12, 2026 09:24
@zhongwangwei zhongwangwei merged commit b8f8df6 into main Jun 12, 2026
10 of 11 checks passed
@zhongwangwei zhongwangwei deleted the fix/nee-flat-var-and-registry-hygiene branch June 12, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant