feat: add energy_score() multivariate probabilistic metric - #5
Open
GiGiKoneti wants to merge 6 commits into
Open
feat: add energy_score() multivariate probabilistic metric#5GiGiKoneti wants to merge 6 commits into
GiGiKoneti wants to merge 6 commits into
Conversation
…semble evaluation Adds probabilistic ensemble evaluation metrics to mllam-verification, as proposed in issue mllam#3 and approved by @mfroelund. New functions in statistics.py: - crps(): wraps scores.probability.crps_for_ensemble via compute_pipeline_statistic, following the same pattern as rmse() and mae(). Uses the fair (unbiased) estimator. Accepts any ensemble_member_dim name. - spread_skill_ratio(): computes ensemble spread / RMSE of ensemble mean. SSR = 1.0 indicates perfect calibration. SSR < 1.0 indicates underdispersion. New function in plot.py: - plot_rank_histogram(): wraps scores.plotdata.rank_histogram to produce a Talagrand diagram. Includes a reference line for perfect calibration. New test fixture in conftest.py: - da_ensemble_prediction_2d_utc: 10-member ensemble DataArray built from the existing deterministic prediction fixture. All functions follow the existing architecture exactly: compute_pipeline_statistic backbone, Google-style docstrings, cell_methods CF-convention attribute, 90-char line length. No new dependencies introduced. All functions use the existing scores>=1.2.0 dependency already pinned in pyproject.toml. Closes mllam#3
…ove imports - Remove unused groupby parameter from crps() and spread_skill_ratio() - Swap argument order to (ds_reference, ds_prediction) matching plot calling convention and mae() signature - Add preserve_dims support to spread_skill_ratio() for hovmoller plots - Move matplotlib/plot_rank_histogram imports to top of test_plot.py - Add crps and spread_skill_ratio to timeseries/hovmoller test parametrizations - Add da_ensemble_prediction_2d_elapsed fixture for elapsed-time tests - Update test_statistics.py to match new argument order All 38 tests pass. Pre-commit hooks (isort, black, flake8, mypy) clean.
…nable groupby support
Implements the Energy Score multivariate proper scoring
rule from scratch (not available in scores>=1.2.0).
Formula: ES = E||X-Y||₂ - 0.5·E||X-X'||₂
Follows exact same signature and convention as crps()
and spread_skill_ratio(). Four unit tests added to
TestEnergyScore class in test_statistics.py:
- returns correct type
- ensemble_member dim collapsed in output
- perfect ensemble produces ES=0 (mathematical ground truth)
- non-negative for any input (ES >= 0 by definition)
Cite: Gneiting & Raftery (2007), JASA 102(477)
Rasp et al. (2024), WeatherBench 2, JAMES
Relevant to neural-lam Issue #62 and GSoC Project 3
probabilistic evaluation pipeline.
GiGiKoneti
force-pushed
the
feat/energy-score
branch
from
April 23, 2026 09:35
7d1a934 to
2a08e56
Compare
Author
Collaborator
|
I just looked through the PR, and your implementation looks great. I'll wait for #4 to get merged before I submit my review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
adds
energy_score()tomllam_verification/operations/statistics.pyright after
spread_skill_ratio(). the energy score is a multivariateproper scoring rule that generalises CRPS to vector-valued forecasts —
useful for checking spatial coherence across ensemble members, not just
marginal correctness at individual grid points.
implemented from scratch since
scores>=1.2.0doesn't include it:ES = E||X - y||₂ - 0.5 · E||X - X'||₂
where X, X' are independent ensemble draws and y is the observation.
used the xarray rename trick (
ensemble_member_prime) for clean O(M²)pairwise computation without python loops. mirrored the
update_cell_methods()call and overall structure fromcrps()exactlyso it fits the existing pattern.
no new dependencies — formula implemented using xarray ops already
present in the codebase.
Issue Link
related to #3 (probabilistic evaluation coverage) — energy score is
the natural next metric after crps and spread_skill_ratio from PR #4
Type of change
Checklist before requesting a review
that clearly describe its purpose, expected inputs and returned values
hard-to-understand passages of my code
written in imperative form
Checklist for reviewers
Author checklist after completed review
in a section reflecting type of change:
- added:
energy_score()multivariate proper scoring ruleChecklist for assignee