Feat: add configurable forecast post processing

[BelhsanHmida]

Description

• Add configurable forecast output post-processing through forecaster config fields:
  • lower: clip forecast values below this bound
  • upper: clip forecast values above this bound
  • snap: replace forecast values inside configured intervals with a target value
• Interpret unitless post-processing values in the output sensor unit.
• Keep existing ensure-positive behavior unchanged for backwards compatibility.
• Apply post-processing after prediction and before storing forecasts.
• Add tests for clipping, snapping, unit conversion, unitless values, schema loading, and schema defaults.
• Update generated OpenAPI specs and forecasting documentation.
• Add changelog entry.

How to test

Automated tests:

uv run pytest flexmeasures/data/tests/test_forecasting_pipeline.py -k "forecast_post_processing" -q
uv run pytest flexmeasures/data/schemas/tests/test_forecasting.py -k "forecast_post_processing or snap_interval" -q

Manual test:

1. Create a forecaster config file, for example /tmp/forecast-post-processing.yml:

lower: "0 kW"
snap:
  "0 kW": ["0 kW", "4 kW"]
upper: "20 kW"

2. Run a forecast for a kW sensor with enough historical data:

flexmeasures add forecasts \
  --sensor <SENSOR_ID> \
  --from-date "2025-01-09T00:00:00+00:00" \
  --to-date "2025-01-09T06:00:00+00:00" \
  --max-forecast-horizon "PT1H" \
  --forecast-frequency "PT1H" \
  --config /tmp/forecast-post-processing.yml

3. Verify the stored forecast values for that run:
   • values below 0 kW are stored as 0 kW
   • values above 20 kW are stored as 20 kW
   • values between 0 kW and 4 kW are stored as 0 kW

Alternative unit-conversion check:

lower: 0
upper: "20000 W"

Expected result: forecasts above 20 kW are clipped to 20 kW.

Unitless snap check:

If the output sensor unit is kW, this config:

snap:
  "0": [0, 4]

is interpreted as:

snap:
  "0 kW": ["0 kW", "4 kW"]

I manually tested this with a temporary kW sensor trained on constant 2.5 kW values. The stored forecast values were:

[0.0, 0.0, 0.0]

Further Improvements

• Validation of unit compatibility and lower <= upper currently happens when post-processing runs. We may move this earlier into schema/parameter validation.
• Overlapping snap intervals are order-dependent; this PR documents and tests the main intended non-overlapping use case.

Related Items

Closes #2270

Sign-off

• I agree to contribute this code under Apache 2 License.
• To the best of my knowledge, I have the right to submit this work under this license.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33506128 | 📁 Comparing 7287a7c against latest (a737210)

  🔍 Preview build  

15 files changed · ± 14 modified · - 1 deleted

± Modified

• changelog.html
• genindex.html
• py-modindex.html
• _autosummary/flexmeasures.api.v3_0.jobs.html
• _autosummary/flexmeasures.api.v3_0.sensors.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.predict.html
• _autosummary/flexmeasures.data.models.forecasting.utils.html
• _autosummary/flexmeasures.data.models.planning.storage.html
• _autosummary/flexmeasures.data.services.html
• _autosummary/flexmeasures.data.services.scheduling.html
• and 4 more...

- Deleted

• _autosummary/flexmeasures.data.services.scheduling_result.html

[Flix6x]

I didn't think of this before, but there is an edge case we did not think of:

If the forecast was exactly 4 kW is it also snapped to 0 kW or not? My first instinct was to have it not snap, i.e. the interval excludes the boundaries. However, that opens up a secondary edge case:

{
  "0 kW": [ 
    "0 kW", 
    "4 kW" 
  ],
  "10 kW": [ 
    "4 kW", 
    "10 kW" 
  ]
}

Does 4 kW now snap to 0 kW or 10 kW?

Actually, if the interval does include the boundaries, the same question applies.

Let's discuss this.

[BelhsanHmida]

Yes indeed, this is an interesting edge case to guard against , I thought about it and have three suggestions:

1. For adjacent intervals, interpret the shared boundary as belonging to the interval that starts there.

For example:

{
  "0 kW": ["0 kW", "4 kW"],
  "10 kW": ["4 kW", "10 kW"],
  "11 kW": ["13 kW", "15 kW"],
  "12 kW": ["15 kW", "16 kW"]
}

would be interpreted as:

[0, 4)
[4, 10]
[13, 15)
[15, 16]

So exactly 4 kW snaps to 10 kW, exactly 10 kW still snaps to 10 kW, exactly 15 kW snaps to 12 kW, and exactly 16 kW still snaps to 12 kW.

2. Reject shared boundaries as invalid config.

Most simple solution, so this would raise an error:

{
  "0 kW": ["0 kW", "4 kW"],
  "10 kW": ["4 kW", "10 kW"]
}

because exactly 4 kW has two possible snap targets.

3. Make the boundary behavior explicit with extra fields.

For example:

{
  "0 kW": {
    "lower": "0 kW",
    "upper": "4 kW",
    "include_lower": true,
    "include_upper": false
  }
}

This is the most explicit, but it makes the config too heavy maybe.

I like option 1 best if we document it clearly, because it keeps the config simple and gives adjacent intervals an intuitive behavior.