From 1c4bcf6e358bafbf2a8fb0e2f80bf631e30643c9 Mon Sep 17 00:00:00 2001 From: Mario Santa Cruz Date: Thu, 18 Jun 2026 10:01:48 +0000 Subject: [PATCH 1/2] docs: update contributing guidelines --- docs/contributing/guidelines.rst | 44 ++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/contributing/guidelines.rst b/docs/contributing/guidelines.rst index cd53155..c8d24d3 100644 --- a/docs/contributing/guidelines.rst +++ b/docs/contributing/guidelines.rst @@ -44,6 +44,11 @@ ATS, contributors can flag them using the following labels; - ``ATS Approved`` – Indicates that the PR has been reviewed and approved during the ATS meeting and is ready to be merged. +- ``Scientific Validation Required`` – Applies to new features (not + bug fixes) that do not require approval by ATS. The contributor must + provide evidence of the feature's scientific validity before the PR + can be merged. See :ref:`scientific-validation` below for details. + It is the responsibility of both the reviewer and the contributor to ensure that a PR is correctly labeled. If you're unsure which label to use, default to ``ATS Approval Needed`` or tag ``@anemoisecurity`` for @@ -123,6 +128,45 @@ Examples of PRs that can be labeled ``ATS Approval Not Needed``: Assuming those do not imply any breaking changes or dependency changes as explained above +.. _scientific-validation: + +Scientific Validation +===================== + +When a PR is labelled ``Scientific Validation Required``, the contributor +must demonstrate that the proposed feature provides tangible value. This +label applies only to **new features** (not bug fixes) that do not require +approval by ATS. The goal is to provide evidence that the feature is useful +and will actually be adopted in operational or research models. + +The form of evidence is intentionally flexible, but it should clearly +show the benefit of the feature. Examples of acceptable evidence include: + +- **Forecast skill improvement** – Show that the feature improves + forecast scores (e.g., RMSE, ACC, CRPS) on a representative + evaluation period compared to a baseline run without the feature. + +- **Computational cost reduction** – Demonstrate that the feature + reduces training/inference time, faster convergence, memory usage, or + other resources. + +- **Qualitative evaluation** – For features affecting specific + phenomena (e.g., forecast realism, small scale features, extreme events), + show case studies or visual comparisons that illustrate the improvement. + +- **Adoption evidence** – Show that the feature is already being used + or tested in a model configuration at one or more institutions. + +Contributors should include this evidence directly in the PR description +or link to supporting material (e.g., experiment reports, notebooks, or +external documents). Reviewers are responsible for verifying that the +provided evidence is sufficient before approving the PR. + +.. note:: + + If no scientific validation evidence is provided, the feature must be + escalated to ATS for discussion before the PR can be merged. + .. _branching-guidelines: ********************** From 0b563cf1e0cc454f3a809104fc8fc9b87289b01b Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:02:55 +0000 Subject: [PATCH 2/2] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/contributing/guidelines.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/guidelines.rst b/docs/contributing/guidelines.rst index c8d24d3..46ae549 100644 --- a/docs/contributing/guidelines.rst +++ b/docs/contributing/guidelines.rst @@ -147,7 +147,7 @@ show the benefit of the feature. Examples of acceptable evidence include: evaluation period compared to a baseline run without the feature. - **Computational cost reduction** – Demonstrate that the feature - reduces training/inference time, faster convergence, memory usage, or + reduces training/inference time, faster convergence, memory usage, or other resources. - **Qualitative evaluation** – For features affecting specific