Add probabilistic ensemble evaluation metrics (CRPS, spread-skill ratio, rank histogram) by GiGiKoneti · Pull Request #4 · mllam/mllam-verification

GiGiKoneti · 2026-03-19T09:03:05Z

Describe your changes

mllam-verification currently only covers deterministic metrics
(rmse, mae). This PR adds the probabilistic ensemble evaluation
suite proposed in issue #3 and approved by @mfroelund.

Three new functions following the exact compute_pipeline_statistic
architecture already established in statistics.py and plot.py:

statistics.py

crps() — wraps scores.probability.crps_for_ensemble via
compute_pipeline_statistic, mirroring how rmse wraps
scores.continuous.rmse. Uses the fair (unbiased) estimator
(Zamo & Naveau 2018) which correctly accounts for finite ensemble
size. Accepts a configurable ensemble_member_dim parameter.
spread_skill_ratio() — computes ensemble spread divided by RMSE
of the ensemble mean. A perfectly calibrated ensemble has SSR = 1.0.
Values below 1.0 indicate underdispersion.

plot.py

plot_rank_histogram() — wraps scores.plotdata.rank_histogram to
produce a Talagrand diagram. Includes a dashed reference line
marking perfect calibration.

tests/unit/conftest.py

Added da_ensemble_prediction_2d_utc fixture — a 10-member ensemble
DataArray built from the existing deterministic prediction fixture
by adding Gaussian noise per member.

No new dependancies introduced. All three functions use the existing
scores>=1.2.0 dependency already pinned in pyproject.toml.

All 34 tests pass. Pre-commit hooks (isort, black, flake8,
mypy) pass cleanly at the 90-char line length.

Note: no CHANGELOG file exists in the repository currently.

Issue Link

Closes #3

Type of change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
📖 Documentation (Addition or improvements to documentation)

Checklist before requesting a review

My branch is up-to-date with the target branch
I have performed a self-review of my code
For any new/modified functions/classes I have added docstrings that clearly describe its purpose, expected inputs and returned values
I have placed in-line comments to clarify the intent of any hard-to-understand passages of my code
I have updated the documentation to cover introduced code changes
I have added tests that prove my fix is effective or that my feature works
I have given the PR a name that clearly describes the change, written in imperative form (context).
I have requested a reviewer and an assignee (assignee is responsible for merging)

Checklist for reviewers

Each PR comes with its own improvements and flaws. The reviewer should check the following:

the code is readable
the code is well tested
the code is documented (including return types and parameters)
the code is easy to maintain

Author checklist after completed review

I have added a line to the CHANGELOG describing this change, in a section
reflecting type of change (add section where missing):
- added: when you have added new functionality
- changed: when default behaviour of the code has been changed
- fixes: when your contribution fixes a bug

Checklist for assignee

PR is up to date with the base branch
the tests pass
author has added an entry to the changelog (and designated the change as added, changed or fixed)
Once the PR is ready to be merged, squash commits and merge the PR.

@mfroelund

…semble evaluation Adds probabilistic ensemble evaluation metrics to mllam-verification, as proposed in issue mllam#3 and approved by @mfroelund. New functions in statistics.py: - crps(): wraps scores.probability.crps_for_ensemble via compute_pipeline_statistic, following the same pattern as rmse() and mae(). Uses the fair (unbiased) estimator. Accepts any ensemble_member_dim name. - spread_skill_ratio(): computes ensemble spread / RMSE of ensemble mean. SSR = 1.0 indicates perfect calibration. SSR < 1.0 indicates underdispersion. New function in plot.py: - plot_rank_histogram(): wraps scores.plotdata.rank_histogram to produce a Talagrand diagram. Includes a reference line for perfect calibration. New test fixture in conftest.py: - da_ensemble_prediction_2d_utc: 10-member ensemble DataArray built from the existing deterministic prediction fixture. All functions follow the existing architecture exactly: compute_pipeline_statistic backbone, Google-style docstrings, cell_methods CF-convention attribute, 90-char line length. No new dependencies introduced. All functions use the existing scores>=1.2.0 dependency already pinned in pyproject.toml. Closes mllam#3

GiGiKoneti · 2026-03-19T09:22:17Z

@mfroelund.. PR ready for your review when you get a chance

mfroelund

Hi! Sorry for the delay. I've made the review, and it generally looks great! I've just added a few minor comments/suggestions.

…ove imports - Remove unused groupby parameter from crps() and spread_skill_ratio() - Swap argument order to (ds_reference, ds_prediction) matching plot calling convention and mae() signature - Add preserve_dims support to spread_skill_ratio() for hovmoller plots - Move matplotlib/plot_rank_histogram imports to top of test_plot.py - Add crps and spread_skill_ratio to timeseries/hovmoller test parametrizations - Add da_ensemble_prediction_2d_elapsed fixture for elapsed-time tests - Update test_statistics.py to match new argument order All 38 tests pass. Pre-commit hooks (isort, black, flake8, mypy) clean.

GiGiKoneti · 2026-04-22T18:19:52Z

I think now it should be okay @mfroelund .

mfroelund

Thanks for your edits. Just one last small addition :)

…nable groupby support

GiGiKoneti · 2026-04-23T07:33:53Z

Hey @mfroelund!

Just wrapped your last bit of feedback on the "unexpected input" tests.

I've added crps and spread_skill_ratio to both test suites, implemented full groupby support for them in statistics.py (they now work just like rmse()/mae()), and verified all 45 tests are passing cleanly.

Should be good to go for final review now. Thanks for all the guidance!

GiGiKoneti · 2026-04-23T07:39:58Z

Another thing @mfroelund Based on my work so far, here are a few ideas that could really level up the repo:

Refactor plot.py complexity - Those # noqa: C901 functions are getting unwieldy. Breaking them into helper functions (_prepare_ensemble_data, _apply_time_ops, etc.) would make testing and maintenance way easier.
Add CRPSS (skill score) - Since we already have CRPS, adding CRPSS would be super valuable. It's basically 1 - (CRPS_model / CRPS_reference) and way more interpretable for comparing models.
Ensemble fan plots - A plot_ensemble_timeseries_fan() showing mean + percentile bands would be the perfect complement to rank histograms. Pretty standard in weather forecasting viz.
Better docs - The docs/ folder is pretty empty rn. A Sphinx/MkDocs site with an example gallery would make the project way more accessible.
CHANGELOG.md - Following the "Keep a Changelog" format would help track changes better, esp as the project grows.

Lmk what you think 🙂
Why don't we have channel for this repository in slack ? I hope it will be necessary in near future as mllam is going through architectural shift.

mfroelund · 2026-04-23T08:31:09Z

Hey @mfroelund!

Just wrapped your last bit of feedback on the "unexpected input" tests.

I've added crps and spread_skill_ratio to both test suites, implemented full groupby support for them in statistics.py (they now work just like rmse()/mae()), and verified all 45 tests are passing cleanly.

Should be good to go for final review now. Thanks for all the guidance!

Thanks for the final adjustments, looks great! Just realized, that we don't have a changelog in this repo. Do you mind adding one in this PR?

GiGiKoneti · 2026-04-23T09:05:36Z

@mfroelund I've updated the CHANGELOG.md and ensured it follows the same style as the neural-lam repository, including PR links and contributor handles. Ready for review!
Later I can update the README.md file once this PR is merged.

mfroelund

Looks good!

mfroelund · 2026-05-18T10:45:51Z

I'm ready to merge, when linting passes:)

GiGiKoneti · 2026-05-18T10:59:35Z

I'm ready to merge, when linting passes:)

there's an ongoing GitHub Actions outage failing the cache download right now..

GiGiKoneti · 2026-06-16T21:59:31Z

@mfroelund
Can u approve to start linting checks please

GiGiKoneti · 2026-06-17T14:18:45Z

@mfroelund
Do you have any idea why we are having this GitHub Actions service outage ?

mfroelund · 2026-06-25T07:37:40Z

@mfroelund Do you have any idea why we are having this GitHub Actions service outage ?

No I have no idea, but now it seems to be resolved!

GiGiKoneti mentioned this pull request Mar 21, 2026

feat: add energy_score() multivariate probabilistic metric #5

Open

20 tasks

mfroelund reviewed Apr 22, 2026

View reviewed changes

Comment thread mllam_verification/operations/statistics.py Outdated

Comment thread mllam_verification/operations/statistics.py Outdated

Comment thread tests/unit/test_plot.py Outdated

Comment thread tests/unit/test_plot.py

mfroelund reviewed Apr 23, 2026

View reviewed changes

Comment thread tests/unit/test_plot.py

test(plot): add probabilistic metrics to unexpected input tests and e…

13e76f2

…nable groupby support

GiGiKoneti added 2 commits April 23, 2026 14:31

docs: add CHANGELOG.md

9fc4d5e

docs: match CHANGELOG style with neural-lam

827436c

mfroelund approved these changes Apr 23, 2026

View reviewed changes

style: run black on test_plot.py

cf8a46c

GiGiKoneti mentioned this pull request Apr 23, 2026

Initial design of mllam-verification package #1

Open

GiGiKoneti added 2 commits May 18, 2026 15:24

chore: re-trigger CI checks for linting

15bc82a

chore: re-trigger CI checks

30a7c29

chore: re-run CI checks after GitHub outage

a4f331c

ci: run pre-commit directly to bypass cache service outages

ea188e6

GiGiKoneti requested a review from mfroelund June 24, 2026 19:43

mfroelund approved these changes Jun 25, 2026

View reviewed changes

Uh oh!

Conversation

GiGiKoneti commented Mar 19, 2026

Describe your changes

Issue Link

Type of change

Checklist before requesting a review

Checklist for reviewers

Author checklist after completed review

Checklist for assignee

Uh oh!

GiGiKoneti commented Mar 19, 2026

Uh oh!

mfroelund left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GiGiKoneti commented Apr 22, 2026

Uh oh!

mfroelund left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GiGiKoneti commented Apr 23, 2026

Uh oh!

GiGiKoneti commented Apr 23, 2026

Uh oh!

mfroelund commented Apr 23, 2026

Uh oh!

GiGiKoneti commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mfroelund left a comment

Choose a reason for hiding this comment

Uh oh!

mfroelund commented May 18, 2026

Uh oh!

GiGiKoneti commented May 18, 2026

Uh oh!

GiGiKoneti commented Jun 16, 2026

Uh oh!

GiGiKoneti commented Jun 17, 2026

Uh oh!

mfroelund commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GiGiKoneti commented Apr 23, 2026 •

edited

Loading