Add zero-fraction inference metric#1323
Open
mcgibbon wants to merge 4 commits into
Open
Conversation
Add a default-off DryFractionMetricConfig / DryFractionAggregator that reports the area-weighted fraction of cells <= a threshold (default 0, i.e. exactly-dry) per variable, for prediction, target, and their difference. For precipitation this is the dry-cell fraction, which the existing RMSE/bias/histogram metrics do not surface directly. Wired into InferenceEvaluatorAggregatorConfig as an opt-in metric.
mcgibbon
commented
Jun 26, 2026
|
|
||
|
|
||
| @dataclasses.dataclass | ||
| class DryFractionMetricConfig: |
Contributor
Author
There was a problem hiding this comment.
Call this ZeroFractionMetricConfig, or not DryFraction. Update all documentation you've added accordingly.
| Parameters: | ||
| variables: when set, only compute the metric for these variables | ||
| (typically ``["PRATEsfc"]``). | ||
| threshold: a cell counts as "dry" when its value is ``<= threshold``. |
Contributor
Author
There was a problem hiding this comment.
Add per_variable_threshold, dict[str, float] = (field defaulting to dict) to be able to specify the threshold per-variable.
| strict: raise if the metric can't be built. | ||
| """ | ||
|
|
||
| variables: list[str] | None = None |
Contributor
Author
There was a problem hiding this comment.
Default the list of variables to an empty list, the user should specify the variables they want. Raise an exception if enabled=True but no variables are given, in the post-init.
|
|
||
| variables: list[str] | None = None | ||
| threshold: float = 0.0 | ||
| name: str = "dry_fraction" |
Contributor
Author
There was a problem hiding this comment.
Suggested change
| name: str = "dry_fraction" | |
| name: str = "zero_threshold_fraction" |
…re variables when enabled - Rename DryFraction* -> ZeroFraction* (file, config, aggregator, metric name default 'zero_threshold_fraction', config field 'zero_fraction'), updating docstrings. - Add per_variable_threshold (dict[str, float]) overriding the global threshold per variable. - Default variables to an empty list and raise in __post_init__ if the metric is enabled with no variables specified. - Drop the redundant metric-name component from the log keys (the framework already prefixes with the metric name), so keys are '<metric>/gen/<var>' rather than the previously doubled form.
Contributor
Author
|
Thanks — addressed in the latest commit:
All aggregator + inference-main tests pass; pre-commit (ruff/mypy) clean. |
mcgibbon
changed the title
mcgibbon
commented
Jun 26, 2026
| """ | ||
|
|
||
| variables: list[str] = dataclasses.field(default_factory=list) | ||
| threshold: float = 0.0 |
Contributor
Author
There was a problem hiding this comment.
Question for reviewer: should we call this eps?
mcgibbon
commented
Jun 26, 2026
Comment on lines
+131
to
+132
| for name, value in target_means.items(): | ||
| logs[f"target/{name}"] = value |
Contributor
Author
There was a problem hiding this comment.
Let's remove the target value, just keep gen/ and gen_minus_target/.
Report only gen/ and gen_minus_target/ per variable; the target fraction is still computed internally for the difference but no longer logged on its own. Update docstrings and tests accordingly.
- Export ZeroFractionMetricConfig from fme.ace (it is in the MetricConfig union, so the symbol-import and metric-documentation tests require it). - Add its autoclass entry to docs/evaluator_config.rst. - Regenerate docs/default-aggregator-config.yaml for the new zero_fraction default block.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a default-off inference metric reporting, per variable, the area-weighted fraction of cells whose value is
<= threshold(default 0) for the prediction, plus its difference from the target. For precipitation (PRATEsfc) with the default threshold this is the fraction of exactly-zero (dry) cells — a scalar the existing RMSE/bias/histogram metrics do not surface directly, useful for diagnosing a model's dry-cell / drizzle behaviour against the target distribution.This is the metric used in the keep_gradient_through_clamps precipitation validation (companion to #1322); opening as a draft to decide whether the zero-fraction diagnostic is worth keeping in
mainas a general-purpose metric.Changes:
fme.ace.aggregator.inference.zero_fraction.ZeroFractionMetricConfig/ZeroFractionAggregator: new default-off metric. Configurablevariables(required whenenabled— raises in__post_init__otherwise),threshold,per_variable_threshold(per-variable override of the global threshold), andname(defaultzero_threshold_fraction). Log keys are<name>/gen/<var>and<name>/gen_minus_target/<var>(the standalone target fraction is computed internally for the difference but not logged).fme.ace.aggregator.inference.main: registerzero_fractiononInferenceEvaluatorAggregatorConfig(opt-in).Tests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated