From 9dd779ad2b4a2c80473ec4cdfa364555bd5f58c5 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Thu, 4 Dec 2025 11:27:27 +0100 Subject: [PATCH 001/136] docs: new section describing commitments Signed-off-by: F.N. Claessen --- documentation/concepts/commitments.rst | 169 +++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 documentation/concepts/commitments.rst diff --git a/documentation/concepts/commitments.rst b/documentation/concepts/commitments.rst new file mode 100644 index 0000000000..5521f6ac1d --- /dev/null +++ b/documentation/concepts/commitments.rst @@ -0,0 +1,169 @@ +Commitments +=========== + +Overview +-------- + +A **Commitment** is the central economic abstraction used by FlexMeasures to +express *soft constraints, preferences and market positions* in the scheduler. + +A commitment describes: + +- a **baseline quantity** over time (the target or assumed position), and +- marginal prices for **upwards** and **downwards deviations** from that baseline. + +The scheduler converts all provided commitments into terms in the optimization +objective function so that the solver *minimizes the total deviation cost* +across the schedule horizon. Absolute physical limits (for example generator or +line capacities) are *not* modelled as commitments — those are enforced as +Pyomo constraints. + +Key properties +-------------- + +Each Commitment has the following important attributes (high level): + +- ``name`` — a logical string identifier (e.g. ``"energy"``, ``"production peak"``). +- ``device`` — optional: restricts the commitment to a single device; otherwise + it is an EMS/site-level commitment. +- ``index`` — the DatetimeIndex (time grid) on which the series are defined. +- ``quantity`` — the baseline Series (per slot or per group). +- ``upwards_deviation_price`` — Series defining marginal cost/reward for upward deviations. +- ``downwards_deviation_price`` — Series defining marginal cost/reward for downward deviations. +- ``_type`` — grouping indicator: ``'each'`` or ``'any'`` (see Grouping below). + +Sign convention (flows vs stocks) +-------------------------------- + +- **Flow commitments** (e.g. power/energy flows): + + - A *positive* baseline quantity denotes **consumption**. + + - Actual > baseline → *upwards* deviation (more consumption). + - Actual < baseline → *downwards* deviation (less consumption). + - A *negative* baseline quantity denotes **production** (feed-in). + + - Actual less negative (i.e. closer to zero) → *upwards* deviation (less production). + - Actual more negative → *downwards* deviation (more production). + +- **Stock commitments** (e.g. state of charge for storage): + + - ``quantity`` is the target stock level; deviations above/below that target + are priced via the upwards/downwards price series. + +Soft vs hard semantics +---------------------- + +Commitments in FlexMeasures are **soft** by design: they represent economic +penalties or rewards that the optimizer considers when building schedules. +Hard operational constraints (such as physical power limits or strict device +interlocks) are expressed separately as Pyomo constraints in the scheduling +model. If a “hard” behaviour is required from a commitment, assign very large +penalty prices, but prefer modelling non-negotiable limits as Pyomo constraints. + +Converting flex-context fields into commitments +----------------------------------------------- + +Users may supply preferences and price fields in the ``flex-context``. The +scheduler translates the relevant fields into one or more `Commitment` objects +before calling the optimizer. + +Typical translations include: + +- tariffs (``consumption-price``, ``production-price``) → an ``"energy"`` FlowCommitment with zero baseline so net consumption/production is priced; +- peak/excess limits (``site-peak-production``, ``site-peak-production-price``, etc.) → dedicated peak FlowCommitment(s); +- storage-related fields (``soc-minima``, ``soc-minima-breach-price``, etc.) → StockCommitment(s). + +A short example +--------------- + +Below is a compact example showing how the scheduler conceptually creates an +``"energy"`` flow commitment from a (per-slot) tariff: + +.. code-block:: python + + from pandas import Series, date_range + from flexmeasures.data.models.planning import FlowCommitment + + index = date_range(start="2025-01-01 00:00", periods=24, freq="H") + # zero baseline → the asset may consume or produce; deviations are priced. + baseline = Series(0.0, index=index) + + # consumption and production tariffs (per kWh) + consumption_price = Series(0.20, index=index) # 0.20 EUR/kWh for consumption + production_price = Series(-0.05, index=index) # -0.05 EUR/kWh reward for production + + energy_commitment = FlowCommitment( + name="energy", + index=index, + quantity=baseline, + upwards_deviation_price=consumption_price, + downwards_deviation_price=production_price, + _type="each" + ) + +The scheduler sets up such commitments (site-level and device-level) and, together with any prior commitments, hands them to the linear optimizer. + +Examples (commitments commonly derived from flex-context) +-------------------------------------------------------- + +The examples below map the most common `flex-context` semantics to the +commitments the scheduler constructs. + +1. **Energy (tariff)** + + - *Fields used*: ``consumption-price``, ``production-price``. + - *Commitment*: Flow commitment named ``"energy"`` with zero baseline and + the two price series as upwards/downwards deviation prices. + +2. **Peak consumption** + + - *Fields used*: ``site-peak-consumption`` (baseline) and ``site-peak-consumption-price`` (upwards-deviation price); the downwards price is set to ``0``. + - *Commitment*: Flow commitment named ``"consumption peak"``; positive baseline + values denote the prior consumption peak associated with sunk costs, and the upwards price penalises going beyond that baseline. + +3. **Peak production / peak feed-in** + + - *Fields used*: ``site-peak-production`` (baseline) and ``site-peak-production-price`` (downwards-deviation price); the upwards price is set to ``0``. + - *Commitment*: Flow commitment named ``"production peak"``; negative baseline + values denote the prior production peak associated with sunk costs, and the downwards price penalises going beyond that baseline. + +4. **Consumption capacity** + + - *Fields used*: ``site-consumption-capacity`` (baseline), and ``site-consumption-breach-price`` (upwards-deviation price); the downwards price is set to ``0``. + - *Commitment*: Flow commitment named ``"consumption breach"``; positive baseline + values denote the allowed consumption limit and the upwards price penalises going + beyond that limit. + +5. **Production capacity** + + - *Fields used*: ``site-production-capacity`` (baseline) and ``site-production-breach-price`` (downwards-deviation price); the upwards price is set to ``0``. + - *Commitment*: Flow commitment named ``"production breach"``; negative baseline + values denote the allowed production limit and the downwards price penalises going + beyond that limit. + +6. **SOC minima / maxima (storage preferences)** + + - *Fields used*: ``soc-minima``, ``soc-minima-breach-price``, ``soc-maxima`` and ``soc-maxima-breach-price``. + - *Commitment*: StockCommitment(s) that price deviations below minima or + above maxima. Hard storage capacities are set through ``soc-min`` and ``soc-max`` instead and are modelled as Pyomo constraints. + +7. **Power bands per device** + + - *Fields used*: ``consumption-capacity`` and ``production-capacity`` (baselines), ``consumption-breach-price`` (upwards-deviation price, with 0 downwards) and ``production-breach-price`` (downwards-deviation price, with 0 upwards). + - *Commitment*: FlowCommitment with either baseline and corresponding prices. + +Grouping across time and devices +-------------------------------- + +- ``_type == 'each'``: penalise deviations per time slot (default for time series). +- ``_type == 'any'``: treat the whole commitment horizon as one group (useful + for peak-style penalties where only the maximum over the window should be + counted). + +.. note:: + + Near-term feature: support for **grouping over devices** is planned and + documented here. When enabled, grouping over devices lets you express + soft constraints that aggregate deviations across a set of devices, + for example, an intermediate capacity constraint from a feeder shared by a group of devices (via **flow commitments**), or multiple power-to-heat devices that feed a shared thermal buffer (via **stock commitments**). From 13e93086fd3c0b928ce6d3fcc6e3c764d58a9a0e Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Thu, 4 Dec 2025 13:44:08 +0100 Subject: [PATCH 002/136] docs: rewrite the overview in commitments section Signed-off-by: F.N. Claessen --- documentation/concepts/commitments.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/documentation/concepts/commitments.rst b/documentation/concepts/commitments.rst index 5521f6ac1d..24d3143ad1 100644 --- a/documentation/concepts/commitments.rst +++ b/documentation/concepts/commitments.rst @@ -4,17 +4,19 @@ Commitments Overview -------- -A **Commitment** is the central economic abstraction used by FlexMeasures to -express *soft constraints, preferences and market positions* in the scheduler. +A **Commitment** is the economic abstraction FlexMeasures uses to express +market positions and soft constraints (preferences) inside the scheduler. +Commitments are converted to linear objective terms; all non-negotiable +operational limits are modelled separately as Pyomo constraints. A commitment describes: -- a **baseline quantity** over time (the target or assumed position), and +- a **baseline quantity** over time (the contracted or preferred position), and - marginal prices for **upwards** and **downwards deviations** from that baseline. The scheduler converts all provided commitments into terms in the optimization objective function so that the solver *minimizes the total deviation cost* -across the schedule horizon. Absolute physical limits (for example generator or +across the schedule horizon. Absolute physical limitations (for example generator or line capacities) are *not* modelled as commitments — those are enforced as Pyomo constraints. From 117919806ca403cb2873c5eb2db509a6c0c8d1d2 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Thu, 4 Dec 2025 13:52:05 +0100 Subject: [PATCH 003/136] docs: keep ems variables, but explain them as representing the site context Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 7d85e57415..3b87ac7963 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -5,8 +5,8 @@ Storage device scheduler: Linear model Introduction -------------- -This generic storage device scheduler is able to handle an EMS with multiple devices, with various types of constraints on the EMS level and on the device level, -and with multiple market commitments on the EMS level. +This generic storage device scheduler is able to handle a site with multiple devices, with various types of constraints on the site level and on the device level, +and with multiple market commitments on the site level. A typical example is a house with many devices. The commitments are assumed to be with regard to the flow of energy to the device (positive for consumption, negative for production). In practice, this generic scheduler is used in the **StorageScheduler** to schedule a storage device. @@ -45,9 +45,9 @@ Symbol Variable in the Code :math:`\epsilon(d,j)` efficiencies Stock energy losses. :math:`P_{max}(d,j)` device_derivative_max Maximum flow of device :math:`d` during time period :math:`j`. :math:`P_{min}(d,j)` device_derivative_min Minimum flow of device :math:`d` during time period :math:`j`. -:math:`P^{ems}_{min}(j)` ems_derivative_min Minimum flow of the EMS during time period :math:`j`. -:math:`P^{ems}_{max}(j)` ems_derivative_max Maximum flow of the EMS during time period :math:`j`. -:math:`Commitment(c,j)` commitment_quantity Commitment c (at EMS level) over time step :math:`j`. +:math:`P^{ems}_{min}(j)` ems_derivative_min Minimum flow of the site's grid connection point during time period :math:`j`. +:math:`P^{ems}_{max}(j)` ems_derivative_max Maximum flow of the site's grid connection point during time period :math:`j`. +:math:`Commitment(c,j)` commitment_quantity Commitment c (at site level) over time step :math:`j`. :math:`M` M Large constant number, upper bound of :math:`Power_{up}(d,j)` and :math:`|Power_{down}(d,j)|`. :math:`D(d,j)` stock_delta Explicit energy gain or loss of device :math:`d` during time period :math:`j`. ================================ ================================================ ============================================================================================================== @@ -58,8 +58,8 @@ Variables ================================ ================================================ ============================================================================================================== Symbol Variable in the Code Description ================================ ================================================ ============================================================================================================== -:math:`\Delta_{up}(c,j)` commitment_upwards_deviation Upwards deviation from the power commitment :math:`c` of the EMS during time period :math:`j`. -:math:`\Delta_{down}(c,j)` commitment_downwards_deviation Downwards deviation from the power commitment :math:`c` of the EMS during time period :math:`j`. +:math:`\Delta_{up}(c,j)` commitment_upwards_deviation Upwards deviation from the power commitment :math:`c` of the site during time period :math:`j`. +:math:`\Delta_{down}(c,j)` commitment_downwards_deviation Downwards deviation from the power commitment :math:`c` of the site during time period :math:`j`. :math:`\Delta Stock(d,j)` n/a Change of stock of device :math:`d` at the end of time period :math:`j`. :math:`P_{up}(d,j)` device_power_up Upwards power of device :math:`d` during time period :math:`j`. :math:`P_{down}(d,j)` device_power_down Downwards power of device :math:`d` during time period :math:`j`. From 1b103601a5d080f892cec0590d0aae61a8486e4b Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Thu, 4 Dec 2025 13:55:24 +0100 Subject: [PATCH 004/136] docs: cross-reference the commitments section and the linear problem formulation Signed-off-by: F.N. Claessen --- documentation/concepts/commitments.rst | 8 ++++++++ documentation/concepts/device_scheduler.rst | 1 + 2 files changed, 9 insertions(+) diff --git a/documentation/concepts/commitments.rst b/documentation/concepts/commitments.rst index 24d3143ad1..d21094e522 100644 --- a/documentation/concepts/commitments.rst +++ b/documentation/concepts/commitments.rst @@ -1,3 +1,5 @@ +.. _commitments: + Commitments =========== @@ -169,3 +171,9 @@ Grouping across time and devices documented here. When enabled, grouping over devices lets you express soft constraints that aggregate deviations across a set of devices, for example, an intermediate capacity constraint from a feeder shared by a group of devices (via **flow commitments**), or multiple power-to-heat devices that feed a shared thermal buffer (via **stock commitments**). + + +Advanced: mathematical formulation +---------------------------------- + +For a compact formulation of how commitments enter the optimization problem, see :ref:``. diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 3b87ac7963..cb36d0dc46 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -11,6 +11,7 @@ and with multiple market commitments on the site level. A typical example is a house with many devices. The commitments are assumed to be with regard to the flow of energy to the device (positive for consumption, negative for production). In practice, this generic scheduler is used in the **StorageScheduler** to schedule a storage device. The solver minimizes the costs of deviating from the commitments. +For a more detailed explanation of commitments in FlexMeasures, see :ref:``. From 6f6e52b0290a4af369b4cc1f898b8ca30dd07fd8 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 8 Dec 2025 15:54:12 +0100 Subject: [PATCH 005/136] fix: cross-references Signed-off-by: F.N. Claessen --- documentation/concepts/commitments.rst | 2 +- documentation/concepts/device_scheduler.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/concepts/commitments.rst b/documentation/concepts/commitments.rst index d21094e522..11ab4cc955 100644 --- a/documentation/concepts/commitments.rst +++ b/documentation/concepts/commitments.rst @@ -176,4 +176,4 @@ Grouping across time and devices Advanced: mathematical formulation ---------------------------------- -For a compact formulation of how commitments enter the optimization problem, see :ref:``. +For a compact formulation of how commitments enter the optimization problem, see :ref:`storage_device_scheduler`. diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index cb36d0dc46..289f40cfc8 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -11,7 +11,7 @@ and with multiple market commitments on the site level. A typical example is a house with many devices. The commitments are assumed to be with regard to the flow of energy to the device (positive for consumption, negative for production). In practice, this generic scheduler is used in the **StorageScheduler** to schedule a storage device. The solver minimizes the costs of deviating from the commitments. -For a more detailed explanation of commitments in FlexMeasures, see :ref:``. +For a more detailed explanation of commitments in FlexMeasures, see :ref:`commitments`. From 359937985193a7cf4e018df6e29b079e081eeaee Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 8 Dec 2025 15:54:26 +0100 Subject: [PATCH 006/136] docs: add commitments section to index Signed-off-by: F.N. Claessen --- documentation/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/index.rst b/documentation/index.rst index 337beb2585..748b160395 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -189,6 +189,7 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum concepts/flexibility concepts/data-model concepts/security_auth + concepts/commitments concepts/device_scheduler From 84fbd5a4f731417f9d4eb1f26937c050abb9562e Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 23 Jan 2026 11:42:18 +0100 Subject: [PATCH 007/136] feat: add a util function for printing out commitments in a tabulated format --- flexmeasures/data/models/planning/utils.py | 30 ++++++++++++++++++++++ flexmeasures/ui/static/openapi-specs.json | 2 +- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index 6ba41dd20e..30fe7b457c 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -8,6 +8,7 @@ from pandas.tseries.frequencies import to_offset import numpy as np import timely_beliefs as tb +from tabulate import tabulate from flexmeasures.data.models.planning.exceptions import UnknownPricesException from flexmeasures.data.models.time_series import Sensor, TimedBelief @@ -559,3 +560,32 @@ def initialize_device_commitment( stock_commitment["device"] = device stock_commitment["class"] = StockCommitment return stock_commitment + + +def print_commitments(flow_commitments): + """ + Pretty-print a list of FlowCommitment objects as tabulated pandas DataFrames. + + For each FlowCommitment, a DataFrame indexed by time is created containing + the commitment name, device values, group index, quantity, and any available + upward or downward deviation prices. Each commitment is printed separately + in a readable table format, making this function suitable for debugging, + logging, and interactive inspection. + """ + for fc in flow_commitments: + + df = pd.DataFrame(index=fc.device.index) + + df["commitment"] = fc.name + df["device"] = fc.device + df["group"] = fc.group + df["quantity"] = fc.quantity + + if hasattr(fc, "upwards_deviation_price"): + df["up_price"] = fc.upwards_deviation_price + + if hasattr(fc, "downwards_deviation_price"): + df["down_price"] = fc.downwards_deviation_price + + if not df.empty: + print(tabulate(df, headers=df.columns, tablefmt="fancy_grid")) diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 89629bb1cf..8c2dee2f83 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -7,7 +7,7 @@ }, "termsOfService": null, "title": "FlexMeasures", - "version": "0.31.0" + "version": "0.30.0" }, "externalDocs": { "description": "FlexMeasures runs on the open source FlexMeasures technology. Read the docs here.", From 90177bfcfdacd28134763969c07cc3cb90ca78c4 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 23 Jan 2026 12:54:01 +0100 Subject: [PATCH 008/136] refactor: move pretty printing method to class Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 28 +++++++++++++++++ flexmeasures/data/models/planning/utils.py | 30 ------------------- 2 files changed, 28 insertions(+), 30 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 636ff9e2b5..9a18f990f0 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -2,6 +2,7 @@ from dataclasses import dataclass, field from datetime import datetime, timedelta +from tabulate import tabulate from typing import Any, Dict, List, Type, Union import pandas as pd @@ -340,6 +341,33 @@ def __post_init__(self): ) self.group = self.group.rename("group") + def pretty_print(self): + """ + Pretty-print a list of FlowCommitment objects as tabulated pandas DataFrames. + + For each FlowCommitment, a DataFrame indexed by time is created containing + the commitment name, device values, group index, quantity, and any available + upward or downward deviation prices. Each commitment is printed separately + in a readable table format, making this function suitable for debugging, + logging, and interactive inspection. + """ + df = self.to_frame() + df = pd.DataFrame(index=df.device.index) + + df["commitment"] = self.name + df["device"] = self.device + df["group"] = self.group + df["quantity"] = self.quantity + + if hasattr(self, "upwards_deviation_price"): + df["up_price"] = self.upwards_deviation_price + + if hasattr(self, "downwards_deviation_price"): + df["down_price"] = self.downwards_deviation_price + + if not df.empty: + print(tabulate(df, headers=df.columns, tablefmt="fancy_grid")) + def to_frame(self) -> pd.DataFrame: """Contains all info apart from the name.""" return pd.concat( diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index 30fe7b457c..6ba41dd20e 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -8,7 +8,6 @@ from pandas.tseries.frequencies import to_offset import numpy as np import timely_beliefs as tb -from tabulate import tabulate from flexmeasures.data.models.planning.exceptions import UnknownPricesException from flexmeasures.data.models.time_series import Sensor, TimedBelief @@ -560,32 +559,3 @@ def initialize_device_commitment( stock_commitment["device"] = device stock_commitment["class"] = StockCommitment return stock_commitment - - -def print_commitments(flow_commitments): - """ - Pretty-print a list of FlowCommitment objects as tabulated pandas DataFrames. - - For each FlowCommitment, a DataFrame indexed by time is created containing - the commitment name, device values, group index, quantity, and any available - upward or downward deviation prices. Each commitment is printed separately - in a readable table format, making this function suitable for debugging, - logging, and interactive inspection. - """ - for fc in flow_commitments: - - df = pd.DataFrame(index=fc.device.index) - - df["commitment"] = fc.name - df["device"] = fc.device - df["group"] = fc.group - df["quantity"] = fc.quantity - - if hasattr(fc, "upwards_deviation_price"): - df["up_price"] = fc.upwards_deviation_price - - if hasattr(fc, "downwards_deviation_price"): - df["down_price"] = fc.downwards_deviation_price - - if not df.empty: - print(tabulate(df, headers=df.columns, tablefmt="fancy_grid")) From 9f34676c64ec49f145533f01cdf8d785965f12ab Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 23 Jan 2026 12:54:46 +0100 Subject: [PATCH 009/136] feat: Commitment supports device groups Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 40 ++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 9a18f990f0..df7e6dbd3f 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -281,6 +281,7 @@ class Commitment: name: str device: pd.Series = None + device_group: pd.Series = None index: pd.DatetimeIndex = field(repr=False, default=None) _type: str = field(repr=False, default="each") group: pd.Series = field(init=False) @@ -340,6 +341,36 @@ def __post_init__(self): "downwards deviation price" ) self.group = self.group.rename("group") + self._init_device_group() + + def _init_device_group(self): + # EMS-level commitment + if self.device is None: + self.device_group = pd.Series({"EMS": 0}, name="device_group") + return + + # Extract device universe + if isinstance(self.device, pd.Series): + devices = self.device.unique() + else: + devices = [self.device] + + devices = list(devices) + + # Default: one group per device (backwards compatible) + if self.device_group is None: + self.device_group = pd.Series( + range(len(devices)), index=devices, name="device_group" + ) + else: + # Validate custom grouping + missing = set(devices) - set(self.device_group.index) + if missing: + raise ValueError( + f"device_group missing assignments for devices: {missing}" + ) + self.device_group = self.device_group.loc[devices] + self.device_group.name = "device_group" def pretty_print(self): """ @@ -370,7 +401,7 @@ def pretty_print(self): def to_frame(self) -> pd.DataFrame: """Contains all info apart from the name.""" - return pd.concat( + df = pd.concat( [ self.device, self.quantity, @@ -381,6 +412,13 @@ def to_frame(self) -> pd.DataFrame: ], axis=1, ) + # map device → device_group + if self.device is not None: + df["device_group"] = self.device.map(self.device_group) + else: + df["device_group"] = 0 + + return df class FlowCommitment(Commitment): From 43107c827933cd071619f2332e1ea432a679f382 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 23 Jan 2026 12:55:47 +0100 Subject: [PATCH 010/136] feat: start testing device grouping Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 flexmeasures/data/models/planning/tests/test_commitments.py diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py new file mode 100644 index 0000000000..e4bd2ba2f7 --- /dev/null +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -0,0 +1,59 @@ +import pandas as pd + +from flexmeasures.data.models.planning import StockCommitment +from flexmeasures.data.models.planning.utils import initialize_index + + +def test_multi_feed(): + start = pd.Timestamp("2026-01-01T00:00+01") + end = pd.Timestamp("2026-01-02T00:00+01") + resolution = pd.Timedelta("PT1H") + index = (initialize_index(start=start, end=end, resolution=resolution),) + + device_group = pd.Series( + { + "gas boiler": "shared thermal buffer", + "heat pump power": "shared thermal buffer", + "battery power": "battery SoC", + } + ) + + max_thermal_soc = "100 kWh" + breach_price = "1000 EUR/kWh" + + commitment_a = StockCommitment( + name="buffer max", + index=index, + quantity=max_thermal_soc, + upwards_deviation_price=breach_price, + downwards_deviation_price=0, + device=pd.Series( + "gas boiler", index=index + ), # per-slot device resolution happens elsewhere + device_group=device_group, + ) + commitment_b = StockCommitment( + name="buffer max", + index=index, + quantity=max_thermal_soc, + upwards_deviation_price=breach_price, + downwards_deviation_price=0, + device=pd.Series( + "heat pump power", index=index + ), # per-slot device resolution happens elsewhere + device_group=device_group, + ) + commitment_c = StockCommitment( + name="buffer max", + index=index, + quantity=max_thermal_soc, + upwards_deviation_price=breach_price, + downwards_deviation_price=0, + device=pd.Series( + "battery power", index=index + ), # per-slot device resolution happens elsewhere + device_group=device_group, + ) + assert commitment_a.device_group[0] == "shared thermal buffer" + assert commitment_b.device_group[0] == "shared thermal buffer" + assert commitment_c.device_group[0] == "battery SoC" From b90d7f0da8d9036c535e0f397af740c0e3b1adcb Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 23 Jan 2026 14:35:35 +0100 Subject: [PATCH 011/136] dev: test multi-feed Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index e4bd2ba2f7..9472b845eb 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -57,3 +57,104 @@ def test_multi_feed(): assert commitment_a.device_group[0] == "shared thermal buffer" assert commitment_b.device_group[0] == "shared thermal buffer" assert commitment_c.device_group[0] == "battery SoC" + + +import pandas as pd +import numpy as np + +from flexmeasures.data.models.planning import StockCommitment +from flexmeasures.data.models.planning.utils import initialize_index +from flexmeasures.data.models.planning.linear_optimization import device_scheduler + + +def test_multi_feed_device_scheduler_shared_buffer(): + # ---- time setup + start = pd.Timestamp("2026-01-01T00:00+01") + end = pd.Timestamp("2026-01-02T00:00+01") + resolution = pd.Timedelta("PT1H") + index = initialize_index(start=start, end=end, resolution=resolution) + + # ---- three devices + devices = ["gas boiler", "heat pump power", "battery power"] + + # ---- device grouping + device_group = pd.Series( + { + "gas boiler": "shared thermal buffer", + "heat pump power": "shared thermal buffer", + "battery power": "battery SoC", + } + ) + + # ---- trivial device constraints (stocks unconstrained individually) + device_constraints = [] + for _ in devices: + df = pd.DataFrame( + { + "min": -np.inf, + "max": np.inf, + "equals": np.nan, + "derivative min": -np.inf, + "derivative max": np.inf, + "derivative equals": np.nan, + }, + index=index, + ) + device_constraints.append(df) + + # ---- no EMS-level constraints + ems_constraints = pd.DataFrame( + { + "derivative min": -np.inf, + "derivative max": np.inf, + }, + index=index, + ) + + # ---- shared buffer max = 100 (soft) + max_soc = 100.0 + breach_price = 1_000.0 + + commitments = [] + for dev in devices: + commitments.append( + StockCommitment( + name="buffer max", + index=index, + quantity=max_soc, + upwards_deviation_price=breach_price, + downwards_deviation_price=0, + device=pd.Series(dev, index=index), + device_group=device_group, + ) + ) + + # ---- run scheduler + planned_power, planned_costs, results, model = device_scheduler( + device_constraints=device_constraints, + ems_constraints=ems_constraints, + commitments=commitments, + initial_stock=0, + ) + + # ---- sanity: model solved + assert results.solver.termination_condition in ("optimal", "locallyOptimal") + + # ---- key assertion: exactly TWO commitment groups + # - one for "shared thermal buffer" + # - one for "battery SoC" + # + # i.e. NOT three (which would indicate per-device baselines) + commitment_groups = set(commitments[0].device_group.values) + breakpoint() + assert commitment_groups == { + "shared thermal buffer", + "battery SoC", + } + + # ---- key behavioural check: + # total commitment cost should be <= 1 breach per group per timestep + # + # If baselines were duplicated, cost would be ~2x for the shared buffer. + expected_max_cost = len(index) * breach_price * 2 + assert planned_costs <= expected_max_cost From 41799816ea8d9f2c44bcaa42828764a96781d6da Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 26 Jan 2026 15:47:14 +0100 Subject: [PATCH 012/136] update the ids of devices to be integers --- .../models/planning/tests/test_commitments.py | 77 ++----------------- 1 file changed, 6 insertions(+), 71 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 9472b845eb..4d856c733c 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,64 +1,3 @@ -import pandas as pd - -from flexmeasures.data.models.planning import StockCommitment -from flexmeasures.data.models.planning.utils import initialize_index - - -def test_multi_feed(): - start = pd.Timestamp("2026-01-01T00:00+01") - end = pd.Timestamp("2026-01-02T00:00+01") - resolution = pd.Timedelta("PT1H") - index = (initialize_index(start=start, end=end, resolution=resolution),) - - device_group = pd.Series( - { - "gas boiler": "shared thermal buffer", - "heat pump power": "shared thermal buffer", - "battery power": "battery SoC", - } - ) - - max_thermal_soc = "100 kWh" - breach_price = "1000 EUR/kWh" - - commitment_a = StockCommitment( - name="buffer max", - index=index, - quantity=max_thermal_soc, - upwards_deviation_price=breach_price, - downwards_deviation_price=0, - device=pd.Series( - "gas boiler", index=index - ), # per-slot device resolution happens elsewhere - device_group=device_group, - ) - commitment_b = StockCommitment( - name="buffer max", - index=index, - quantity=max_thermal_soc, - upwards_deviation_price=breach_price, - downwards_deviation_price=0, - device=pd.Series( - "heat pump power", index=index - ), # per-slot device resolution happens elsewhere - device_group=device_group, - ) - commitment_c = StockCommitment( - name="buffer max", - index=index, - quantity=max_thermal_soc, - upwards_deviation_price=breach_price, - downwards_deviation_price=0, - device=pd.Series( - "battery power", index=index - ), # per-slot device resolution happens elsewhere - device_group=device_group, - ) - assert commitment_a.device_group[0] == "shared thermal buffer" - assert commitment_b.device_group[0] == "shared thermal buffer" - assert commitment_c.device_group[0] == "battery SoC" - - import pandas as pd import numpy as np @@ -80,9 +19,9 @@ def test_multi_feed_device_scheduler_shared_buffer(): # ---- device grouping device_group = pd.Series( { - "gas boiler": "shared thermal buffer", - "heat pump power": "shared thermal buffer", - "battery power": "battery SoC", + 0: "shared thermal buffer", # gas boiler + 1: "shared thermal buffer", # "heat pump power" + 2: "battery SoC", #"battery power" } ) @@ -116,7 +55,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): breach_price = 1_000.0 commitments = [] - for dev in devices: + for d,dev in enumerate(devices): commitments.append( StockCommitment( name="buffer max", @@ -124,7 +63,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): quantity=max_soc, upwards_deviation_price=breach_price, downwards_deviation_price=0, - device=pd.Series(dev, index=index), + device=pd.Series(d, index=index), device_group=device_group, ) ) @@ -146,11 +85,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): # # i.e. NOT three (which would indicate per-device baselines) commitment_groups = set(commitments[0].device_group.values) - breakpoint() - assert commitment_groups == { - "shared thermal buffer", - "battery SoC", - } + assert commitment_groups == {"shared thermal buffer"} # ---- key behavioural check: # total commitment cost should be <= 1 breach per group per timestep From 8b108ed881a7576760d87c8a24a6bb48b8855eb4 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 26 Jan 2026 15:52:21 +0100 Subject: [PATCH 013/136] feat: function that group commitment quantities Signed-off-by: Ahmad-Wahid --- .../models/planning/linear_optimization.py | 68 +++++++++++++++++-- flexmeasures/ui/static/openapi-specs.json | 2 +- 2 files changed, 65 insertions(+), 5 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 2f1ba06d1d..38caeb127e 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -207,6 +207,22 @@ def convert_commitments_to_subcommitments( commitments, commitment_mapping = convert_commitments_to_subcommitments(commitments) + device_group_lookup = {} + + for c, df in enumerate(commitments): + if "device_group" not in df.columns or "device" not in df.columns: + continue + + device_group_lookup[c] = {} + + # keep only relevant rows + rows = df[["device", "device_group"]].dropna().drop_duplicates() + + for _, row in rows.iterrows(): + g = row["device_group"] + d = row["device"] # NOTE: must match model.d indexing + device_group_lookup[c].setdefault(g, set()).add(d) + # Oversimplified check for a convex cost curve df = pd.concat(commitments)[ ["upwards deviation price", "downwards deviation price"] @@ -243,13 +259,21 @@ def convert_commitments_to_subcommitments( ) model.c = RangeSet(0, len(commitments) - 1, doc="Set of commitments") - # Add 2D indices for commitment datetimes (cj) + def commitment_device_groups_init(m): + return ((c, g) for c, groups in device_group_lookup.items() for g in groups) + + model.cg = Set(dimen=2, initialize=commitment_device_groups_init) def commitments_init(m): return ((c, j) for c in m.c for j in commitments[c]["j"]) model.cj = Set(dimen=2, initialize=commitments_init) + def commitment_time_device_groups_init(m): + return ((c, j, g) for (c, j) in m.cj for (_, g) in m.cg if _ == c) + + model.cjg = Set(dimen=3, initialize=commitment_time_device_groups_init) + # Add parameters def price_down_select(m, c): if "downwards deviation price" not in commitments[c].columns: @@ -362,6 +386,40 @@ def device_derivative_up_efficiency(m, d, j): def device_stock_delta(m, d, j): return device_constraints[d]["stock delta"].iloc[j] + def grouped_commitment_equalities(m, c, j, g): + """ + Enforce a commitment deviation constraint on the aggregate of devices in a group. + + For commitment ``c`` at time index ``j``, this constraint couples the commitment + baseline (plus deviation variables) to the summed flow or stock of all devices + belonging to device group ``g``. StockCommitments aggregate device stocks, while + FlowCommitments aggregate device flows. Constraints are skipped if the commitment + is inactive at ``(c, j)`` or if the group contains no devices. + """ + if m.commitment_quantity[c, j] == -infinity: + return Constraint.Skip + + devices_in_group = device_group_lookup.get(c, {}).get(g, set()) + if not devices_in_group: + return Constraint.Skip + + center = ( + m.commitment_quantity[c, j] + + m.commitment_downwards_deviation[c] + + m.commitment_upwards_deviation[c] + ) + + if commitments[c]["class"].apply(lambda cl: cl == StockCommitment).all(): + center -= sum(_get_stock_change(m, d, j) for d in devices_in_group) + else: + center -= sum(m.ems_power[d, j] for d in devices_in_group) + + return ( + 0 if "upwards deviation price" in commitments[c].columns else None, + center, + 0 if "downwards deviation price" in commitments[c].columns else None, + ) + model.up_price = Param(model.c, initialize=price_up_select) model.down_price = Param(model.c, initialize=price_down_select) model.commitment_quantity = Param( @@ -565,6 +623,10 @@ def device_derivative_equalities(m, d, j): 0, ) + model.grouped_commitment_equalities = Constraint( + model.cjg, rule=grouped_commitment_equalities + ) + model.device_energy_bounds = Constraint(model.d, model.j, rule=device_bounds) model.device_power_bounds = Constraint( model.d, model.j, rule=device_derivative_bounds @@ -592,9 +654,7 @@ def device_derivative_equalities(m, d, j): model.ems_power_commitment_equalities = Constraint( model.cj, rule=ems_flow_commitment_equalities ) - model.device_energy_commitment_equalities = Constraint( - model.cj, model.d, rule=device_stock_commitment_equalities - ) + model.device_power_equalities = Constraint( model.d, model.j, rule=device_derivative_equalities ) diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 8c2dee2f83..89629bb1cf 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -7,7 +7,7 @@ }, "termsOfService": null, "title": "FlexMeasures", - "version": "0.30.0" + "version": "0.31.0" }, "externalDocs": { "description": "FlexMeasures runs on the open source FlexMeasures technology. Read the docs here.", From 485349e425b6efe40b5abbfed45aaf457bbd2246 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 26 Jan 2026 18:34:58 +0100 Subject: [PATCH 014/136] add commitments for multi group Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 105 +++++++++++++++--- 1 file changed, 91 insertions(+), 14 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 4d856c733c..26349f0ff6 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,8 +1,11 @@ import pandas as pd import numpy as np -from flexmeasures.data.models.planning import StockCommitment -from flexmeasures.data.models.planning.utils import initialize_index +from flexmeasures.data.models.planning import StockCommitment, FlowCommitment +from flexmeasures.data.models.planning.utils import ( + initialize_index, + add_tiny_price_slope, +) from flexmeasures.data.models.planning.linear_optimization import device_scheduler @@ -19,23 +22,36 @@ def test_multi_feed_device_scheduler_shared_buffer(): # ---- device grouping device_group = pd.Series( { - 0: "shared thermal buffer", # gas boiler - 1: "shared thermal buffer", # "heat pump power" - 2: "battery SoC", #"battery power" + 0: "shared thermal buffer", # gas boiler + 1: "shared thermal buffer", # "heat pump power" + 2: "battery SoC", # "battery power" } ) - + device_commodity = pd.Series( + { + 0: "gas", # gas boiler + 1: "electricity", # "heat pump power" + 2: "electricity", # "battery power" + } + ) + equals = pd.Series(np.nan, index=index) + equals[-1] = 100 # ---- trivial device constraints (stocks unconstrained individually) device_constraints = [] - for _ in devices: + for d, device_name in enumerate(devices): + # 0 and 1 : derivative min 0 + # 2 : derivative min = - production capacity + df = pd.DataFrame( { - "min": -np.inf, - "max": np.inf, + "min": 0, + "max": 100, "equals": np.nan, - "derivative min": -np.inf, - "derivative max": np.inf, + "derivative min": 0 if d in (0, 1) else -20, + "derivative max": 20, "derivative equals": np.nan, + "derivative down efficiency": 0.9, + "derivative up efficiency": 0.9, }, index=index, ) @@ -44,8 +60,8 @@ def test_multi_feed_device_scheduler_shared_buffer(): # ---- no EMS-level constraints ems_constraints = pd.DataFrame( { - "derivative min": -np.inf, - "derivative max": np.inf, + "derivative min": -40, + "derivative max": 40, }, index=index, ) @@ -53,9 +69,36 @@ def test_multi_feed_device_scheduler_shared_buffer(): # ---- shared buffer max = 100 (soft) max_soc = 100.0 breach_price = 1_000.0 + min_soc = pd.Series(0, index=index) + min_soc[-1] = 100 + + # default commodity: electricity + # choice: electricity or gas + gas_price = pd.Series(300, index=index) + electricity_price = pd.Series(600, index=index, name="event_value") + electricity_price.iloc[12:14] = 200 + prices = {"gas": gas_price, "electricity": electricity_price} + + sloped_prices = ( + add_tiny_price_slope(electricity_price.to_frame()) + - electricity_price.to_frame() + ) commitments = [] - for d,dev in enumerate(devices): + + # todo: fix this commitment for the group[boiler, heat pump] to reach 100 kW together + commitments.append( + StockCommitment( + name="buffer min", + index=index, + quantity=min_soc, + upwards_deviation_price=0, + downwards_deviation_price=-breach_price, + device=None, + device_group=device_group, + ) + ) + for d, dev in enumerate(devices): commitments.append( StockCommitment( name="buffer max", @@ -67,6 +110,29 @@ def test_multi_feed_device_scheduler_shared_buffer(): device_group=device_group, ) ) + commitments.append( + FlowCommitment( + name=device_commodity[d], + index=index, + quantity=0, + upwards_deviation_price=prices[device_commodity[d]], + downwards_deviation_price=prices[device_commodity[d]], + device=pd.Series(d, index=index), + device_group=device_commodity, + ) + ) + + commitments.append( + FlowCommitment( + name="preferred_charge_sooner", + index=index, + quantity=0, + upwards_deviation_price=sloped_prices, + downwards_deviation_price=sloped_prices, + device=pd.Series(d, index=index), + device_group=device_commodity, + ) + ) # ---- run scheduler planned_power, planned_costs, results, model = device_scheduler( @@ -85,6 +151,17 @@ def test_multi_feed_device_scheduler_shared_buffer(): # # i.e. NOT three (which would indicate per-device baselines) commitment_groups = set(commitments[0].device_group.values) + + # commitment_costs = [ + # { + # "name": "commitment_costs", + # "data": { + # c.name: costs + # for c, costs in zip(commitments, model.commitment_costs.values()) + # }, + # }, + # ] + assert commitment_groups == {"shared thermal buffer"} # ---- key behavioural check: From 81bb9ecd47e9d50512b18c04d7cafb0c9fcd6b63 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 27 Jan 2026 20:19:44 +0100 Subject: [PATCH 015/136] fix: get unique list of devices for a frame column Signed-off-by: Ahmad-Wahid --- .../data/models/planning/linear_optimization.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 38caeb127e..37bada383b 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -213,15 +213,20 @@ def convert_commitments_to_subcommitments( if "device_group" not in df.columns or "device" not in df.columns: continue - device_group_lookup[c] = {} + rows = df[["device", "device_group"]].dropna() - # keep only relevant rows - rows = df[["device", "device_group"]].dropna().drop_duplicates() + device_group_lookup[c] = {} for _, row in rows.iterrows(): g = row["device_group"] - d = row["device"] # NOTE: must match model.d indexing - device_group_lookup[c].setdefault(g, set()).add(d) + d = row["device"] + + if isinstance(d, (list, tuple, set, np.ndarray)): + devices = set(d) + else: + devices = {d} + + device_group_lookup[c].setdefault(g, set()).update(devices) # Oversimplified check for a convex cost curve df = pd.concat(commitments)[ From 334e4d38b289097a1d3cbf5f91f9c446c5013847 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 27 Jan 2026 20:23:28 +0100 Subject: [PATCH 016/136] fix: create util functions that extract devices for a list of values and map them to the respective group id Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/__init__.py | 51 +++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index df7e6dbd3f..a1e7cfa747 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -4,7 +4,7 @@ from datetime import datetime, timedelta from tabulate import tabulate from typing import Any, Dict, List, Type, Union - +from collections.abc import Iterable import pandas as pd from flask import current_app @@ -351,7 +351,7 @@ def _init_device_group(self): # Extract device universe if isinstance(self.device, pd.Series): - devices = self.device.unique() + devices = extract_devices(self.device) else: devices = [self.device] @@ -414,7 +414,7 @@ def to_frame(self) -> pd.DataFrame: ) # map device → device_group if self.device is not None: - df["device_group"] = self.device.map(self.device_group) + df["device_group"] = map_device_to_group(self.device, self.device_group) else: df["device_group"] = 0 @@ -440,3 +440,48 @@ class StockCommitment(Commitment): Scheduler.compute_schedule = deprecated(Scheduler.compute, "0.14")( Scheduler.compute_schedule ) + + +def extract_devices(device): + """ + Return a flat list of unique device identifiers from: + - scalar device + - Series of scalars + - Series of iterables (e.g. [0, 1]) + """ + if device is None: + return [] + + if isinstance(device, pd.Series): + values = device.dropna().values + else: + values = [device] + + devices = set() + for v in values: + if isinstance(v, Iterable) and not isinstance(v, (str, bytes)): + devices.update(v) + else: + devices.add(v) + + return list(devices) + + +def map_device_to_group(device_series, device_group_map): + """ + Map device identifiers to device_group. + + - scalar device → group label + - iterable of devices → group label (must be identical) + """ + + def resolve(v): + if isinstance(v, (list, tuple, set)): + groups = {device_group_map[d] for d in v} + if len(groups) != 1: + raise ValueError(f"Devices {v} map to multiple device groups: {groups}") + return groups.pop() + else: + return device_group_map[v] + + return device_series.apply(resolve) From f738d9f32a9604f72cf1ad92f8e466f4df9ea062 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 27 Jan 2026 20:25:13 +0100 Subject: [PATCH 017/136] fix: create a series for a list of grouped devices Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 26349f0ff6..fd67097da6 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -86,7 +86,6 @@ def test_multi_feed_device_scheduler_shared_buffer(): commitments = [] - # todo: fix this commitment for the group[boiler, heat pump] to reach 100 kW together commitments.append( StockCommitment( name="buffer min", @@ -94,7 +93,9 @@ def test_multi_feed_device_scheduler_shared_buffer(): quantity=min_soc, upwards_deviation_price=0, downwards_deviation_price=-breach_price, - device=None, + # instead of device=None, I considered to create a series for the devices that we need for this + # specific commitment. + device=pd.Series([[0, 1]] * len(index), index=index), device_group=device_group, ) ) From 620c6dc350b278053f9c79b5c01559517e1ea819 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Sat, 31 Jan 2026 17:23:27 +0100 Subject: [PATCH 018/136] drop outdated comments --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index fd67097da6..8d4df6e720 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -36,7 +36,6 @@ def test_multi_feed_device_scheduler_shared_buffer(): ) equals = pd.Series(np.nan, index=index) equals[-1] = 100 - # ---- trivial device constraints (stocks unconstrained individually) device_constraints = [] for d, device_name in enumerate(devices): # 0 and 1 : derivative min 0 @@ -57,7 +56,6 @@ def test_multi_feed_device_scheduler_shared_buffer(): ) device_constraints.append(df) - # ---- no EMS-level constraints ems_constraints = pd.DataFrame( { "derivative min": -40, From 40eb747b61321d1d47a9c448fcee2a9ca4ac0ca6 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Sat, 31 Jan 2026 19:59:01 +0100 Subject: [PATCH 019/136] use commitment costs and add asserts for electricity and gas Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 8d4df6e720..078ae3d502 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -151,15 +151,15 @@ def test_multi_feed_device_scheduler_shared_buffer(): # i.e. NOT three (which would indicate per-device baselines) commitment_groups = set(commitments[0].device_group.values) - # commitment_costs = [ - # { - # "name": "commitment_costs", - # "data": { - # c.name: costs - # for c, costs in zip(commitments, model.commitment_costs.values()) - # }, - # }, - # ] + commitment_costs = { + "name": "commitment_costs", + "data": { + c.name: costs + for c, costs in zip(commitments, model.commitment_costs.values()) + }, + } + assert commitment_costs["data"]["electricity"] == -11440.0 + assert round(commitment_costs["data"]["gas"], 2) == 21333.33 assert commitment_groups == {"shared thermal buffer"} From 708d86949420d6662c06818c4fe40e6cf4b5d046 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Sat, 31 Jan 2026 21:47:09 +0100 Subject: [PATCH 020/136] and an assert for commodity costs Signed-off-by: Ahmad-Wahid --- .../data/models/planning/tests/test_commitments.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 078ae3d502..56179ff8f2 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -150,6 +150,12 @@ def test_multi_feed_device_scheduler_shared_buffer(): # # i.e. NOT three (which would indicate per-device baselines) commitment_groups = set(commitments[0].device_group.values) + commodity_commitments = { + c.name + for c in commitments + if isinstance(c, FlowCommitment) and c.name in {"gas", "electricity"} + } + assert commodity_commitments == {"gas", "electricity"} commitment_costs = { "name": "commitment_costs", @@ -158,8 +164,10 @@ def test_multi_feed_device_scheduler_shared_buffer(): for c, costs in zip(commitments, model.commitment_costs.values()) }, } - assert commitment_costs["data"]["electricity"] == -11440.0 - assert round(commitment_costs["data"]["gas"], 2) == 21333.33 + commodity_costs = { + k: v for k, v in commitment_costs["data"].items() if k in {"gas", "electricity"} + } + assert set(commodity_costs.keys()) == {"gas", "electricity"} assert commitment_groups == {"shared thermal buffer"} From ce307f82474d690cb115d2664447cc344bc96666 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Sat, 31 Jan 2026 22:13:22 +0100 Subject: [PATCH 021/136] add an extra assert on costs Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 56179ff8f2..639813a090 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -177,3 +177,5 @@ def test_multi_feed_device_scheduler_shared_buffer(): # If baselines were duplicated, cost would be ~2x for the shared buffer. expected_max_cost = len(index) * breach_price * 2 assert planned_costs <= expected_max_cost + total_commodity_cost = sum(commodity_costs.values()) + assert total_commodity_cost <= planned_costs From 5be555b73838e5e3cf923f800a2ba45681e1042f Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 3 Feb 2026 01:42:08 +0100 Subject: [PATCH 022/136] support commodity based EMS flow commitments and grouped devices Signed-off-by: Ahmad-Wahid --- .../models/planning/linear_optimization.py | 82 +++++++++++-------- 1 file changed, 50 insertions(+), 32 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 37bada383b..0dfc3ebb2a 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -135,6 +135,20 @@ def device_scheduler( # noqa C901 df["group"] = group commitments.append(df) + # commodity → set(device indices) + commodity_devices = {} + + for df in commitments: + if "commodity" not in df.columns or "device" not in df.columns: + continue + + for _, row in df[["commodity", "device"]].dropna().iterrows(): + devices = row["device"] + if not isinstance(devices, (list, tuple, set)): + devices = [devices] + + commodity_devices.setdefault(row["commodity"], set()).update(devices) + # Check if commitments have the same time window and resolution as the constraints for commitment in commitments: start_c = commitment.index.to_pydatetime()[0] @@ -579,45 +593,33 @@ def device_stock_commitment_equalities(m, c, j, d): ) def ems_flow_commitment_equalities(m, c, j): - """Couple EMS flows (sum over devices) to each commitment. + """ + Enforce an EMS-level flow commitment for a given commodity. - - Creates an inequality for one-sided commitments. - - Creates an equality for two-sided commitments and for groups of size 1. + Couples the commitment baseline (plus deviation variables) to the sum of EMS + power over all devices belonging to the commitment’s commodity. Skips + non-flow commitments or commodities without associated devices. """ - if ( - "device" in commitments[c].columns - and not pd.isnull(commitments[c]["device"]).all() - ) or m.commitment_quantity[c, j] == -infinity: - # Commitment c does not concern EMS + if commitments[c]["class"].iloc[0] != FlowCommitment: return Constraint.Skip - if ( - "class" in commitments[c].columns - and not ( - commitments[c]["class"].apply(lambda cl: cl == FlowCommitment) - ).all() - ): - raise NotImplementedError( - "StockCommitment on an EMS level has not been implemented. Please file a GitHub ticket explaining your use case." - ) + + commodity = ( + commitments[c]["commodity"].iloc[0] + if "commodity" in commitments[c].columns + else None + ) + devices = commodity_devices.get(commodity, set()) + + if not devices: + return Constraint.Skip + return ( - ( - 0 - if len(commitments[c]) == 1 - or "upwards deviation price" in commitments[c].columns - else None - ), - # 0 if "upwards deviation price" in commitments[c].columns else None, # todo: possible simplification + None, m.commitment_quantity[c, j] + m.commitment_downwards_deviation[c] + m.commitment_upwards_deviation[c] - - sum(m.ems_power[:, j]), - ( - 0 - if len(commitments[c]) == 1 - or "downwards deviation price" in commitments[c].columns - else None - ), - # 0 if "downwards deviation price" in commitments[c].columns else None, # todo: possible simplification + - sum(m.ems_power[d, j] for d in devices), + None, ) def device_derivative_equalities(m, d, j): @@ -718,6 +720,22 @@ def cost_function(m): ) model.commitment_costs = commitment_costs + commodity_costs = {} + for c in model.c: + commodity = None + if "commodity" in commitments[c].columns: + commodity = commitments[c]["commodity"].iloc[0] + if commodity is None or (isinstance(commodity, float) and np.isnan(commodity)): + continue + + cost = value( + model.commitment_downwards_deviation[c] * model.down_price[c] + + model.commitment_upwards_deviation[c] * model.up_price[c] + ) + commodity_costs[commodity] = commodity_costs.get(commodity, 0) + cost + + model.commodity_costs = commodity_costs + # model.pprint() # model.display() # print(results.solver.termination_condition) From 6f2d7450b96637e9e68fe312cc5062bc3059ff01 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 3 Feb 2026 01:43:40 +0100 Subject: [PATCH 023/136] Add commodity field and support multi-device commitments Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/__init__.py | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index a1e7cfa747..32b83314ee 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -288,8 +288,24 @@ class Commitment: quantity: pd.Series = 0 upwards_deviation_price: pd.Series = 0 downwards_deviation_price: pd.Series = 0 + commodity: str | pd.Series | None = None def __post_init__(self): + if ( + isinstance(self, FlowCommitment) + and isinstance(self.commodity, pd.Series) + and self.device is not None + ): + devices = extract_devices(self.device) + missing = set(devices) - set(self.commodity.index) + if missing: + raise ValueError(f"commodity mapping missing for devices: {missing}") + + if isinstance(self, FlowCommitment) and self.commodity is None: + raise ValueError( + "FlowCommitment requires `commodity` (str or pd.Series mapping device→commodity)" + ) + series_attributes = [ attr for attr, _type in self.__annotations__.items() @@ -412,12 +428,25 @@ def to_frame(self) -> pd.DataFrame: ], axis=1, ) - # map device → device_group + # device_group if self.device is not None: df["device_group"] = map_device_to_group(self.device, self.device_group) else: df["device_group"] = 0 + # commodity + if getattr(self, "commodity", None) is None: + df["commodity"] = None + elif isinstance(self.commodity, pd.Series): + # commodity is a device→commodity mapping, like device_group + if self.device is None: + df["commodity"] = None + else: + df["commodity"] = map_device_to_group(self.device, self.commodity) + else: + # scalar commodity + df["commodity"] = self.commodity + return df From 1d63893eaf0e3a3e4fa91f82634203291c2709da Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 3 Feb 2026 01:45:35 +0100 Subject: [PATCH 024/136] test shared buffer and multi-comodity commitments Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 639813a090..7b99afcb88 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -118,6 +118,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): downwards_deviation_price=prices[device_commodity[d]], device=pd.Series(d, index=index), device_group=device_commodity, + commodity=device_commodity[d], ) ) @@ -130,6 +131,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): downwards_deviation_price=sloped_prices, device=pd.Series(d, index=index), device_group=device_commodity, + commodity=device_commodity[d], ) ) From 8c9b1b535c1121872b30139986093d2ec029ff2d Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 3 Feb 2026 02:09:28 +0100 Subject: [PATCH 025/136] remove hard check for commodity to make backward compatible Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/__init__.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 32b83314ee..2659faa3da 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -301,11 +301,6 @@ def __post_init__(self): if missing: raise ValueError(f"commodity mapping missing for devices: {missing}") - if isinstance(self, FlowCommitment) and self.commodity is None: - raise ValueError( - "FlowCommitment requires `commodity` (str or pd.Series mapping device→commodity)" - ) - series_attributes = [ attr for attr, _type in self.__annotations__.items() From a8f3b89a898345627e283320e3c32b40e7418f26 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 3 Feb 2026 02:10:42 +0100 Subject: [PATCH 026/136] update the function to support backward compatibility Signed-off-by: Ahmad-Wahid --- .../models/planning/linear_optimization.py | 27 +++++++++---------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 0dfc3ebb2a..4141042509 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -593,25 +593,22 @@ def device_stock_commitment_equalities(m, c, j, d): ) def ems_flow_commitment_equalities(m, c, j): - """ - Enforce an EMS-level flow commitment for a given commodity. + """Couple EMS flow commitments to device flows, optionally filtered by commodity.""" - Couples the commitment baseline (plus deviation variables) to the sum of EMS - power over all devices belonging to the commitment’s commodity. Skips - non-flow commitments or commodities without associated devices. - """ if commitments[c]["class"].iloc[0] != FlowCommitment: return Constraint.Skip - commodity = ( - commitments[c]["commodity"].iloc[0] - if "commodity" in commitments[c].columns - else None - ) - devices = commodity_devices.get(commodity, set()) - - if not devices: - return Constraint.Skip + # Legacy behavior: no commodity → sum over all devices + if "commodity" not in commitments[c].columns: + devices = m.d + else: + commodity = commitments[c]["commodity"].iloc[0] + if pd.isna(commodity): + devices = m.d + else: + devices = commodity_devices.get(commodity, set()) + if not devices: + return Constraint.Skip return ( None, From be09c4d88691cb68a6a8a72d5a3240be4b5122d0 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 10 Feb 2026 11:48:35 +0100 Subject: [PATCH 027/136] feat: add commodity field to the flexmodel and DBstorage-flex-model schemas Signed-off-by: Ahmad-Wahid --- .../data/schemas/scheduling/storage.py | 20 +++++++++++++++++++ flexmeasures/ui/static/openapi-specs.json | 9 +++++++++ 2 files changed, 29 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index 9eaf1dcf7c..1154c4c97f 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -25,6 +25,8 @@ is_energy_unit, ) +ALLOWED_COMMODITIES = {"electricity", "gas"} + # Telling type hints what to expect after schema parsing SoCTarget = TypedDict( "SoCTarget", @@ -222,6 +224,12 @@ class StorageFlexModelSchema(Schema): validate=validate.Length(min=1), metadata=metadata.SOC_USAGE.to_dict(), ) + commodity = fields.Str( + required=False, + load_default="electricity", + validate=OneOf(["electricity", "gas"]), + metadata=dict(description="Commodity label for this device/asset."), + ) def __init__( self, @@ -343,6 +351,11 @@ def check_redundant_efficiencies(self, data: dict, **kwargs): f"Fields `{field}` and `roundtrip_efficiency` are mutually exclusive." ) + @validates("commodity") + def validate_commodity(self, commodity: str, **kwargs): + if not isinstance(commodity, str) or not commodity.strip(): + raise ValidationError("commodity must be a non-empty string.") + @post_load def post_load_sequence(self, data: dict, **kwargs) -> dict: """Perform some checks and corrections after we loaded.""" @@ -492,6 +505,13 @@ class DBStorageFlexModelSchema(Schema): metadata={"deprecated field": "production_capacity"}, ) + commodity = fields.Str( + required=False, + load_default="electricity", + validate=OneOf(["electricity", "gas"]), + metadata=dict(description="Commodity label for this device/asset."), + ) + mapped_schema_keys: dict def __init__(self, *args, **kwargs): diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 89629bb1cf..f98ea620fb 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -5388,6 +5388,15 @@ } ], "items": {} + }, + "commodity": { + "type": "string", + "default": "electricity", + "enum": [ + "electricity", + "gas" + ], + "description": "Commodity label for this device/asset." } }, "additionalProperties": false From 07822eb8471db1f1bbbd7f0ad60e653f12b37ead Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 10 Feb 2026 12:18:08 +0100 Subject: [PATCH 028/136] fix: use devices as index rather than time series Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 2659faa3da..5f0e6fe2e9 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -304,7 +304,9 @@ def __post_init__(self): series_attributes = [ attr for attr, _type in self.__annotations__.items() - if _type == "pd.Series" and hasattr(self, attr) + if _type == "pd.Series" + and hasattr(self, attr) + and attr not in ("device_group", "commodity") ] for series_attr in series_attributes: val = getattr(self, series_attr) @@ -374,6 +376,10 @@ def _init_device_group(self): range(len(devices)), index=devices, name="device_group" ) else: + if not isinstance(self.device_group, pd.Series): + self.device_group = pd.Series( + self.device_group, index=devices, name="device_group" + ) # Validate custom grouping missing = set(devices) - set(self.device_group.index) if missing: From 92eabc763ae081ab741ab414d09d607b238d870a Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 10 Feb 2026 12:50:30 +0100 Subject: [PATCH 029/136] fix: exclude gas-power devices from electricity commitments Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 370 +++++++++++-------- 1 file changed, 206 insertions(+), 164 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index a0d6f8da19..2b866878a4 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -238,7 +238,10 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Set up commitments to optimise for commitments = self.convert_to_commitments( - query_window=(start, end), resolution=resolution, beliefs_before=belief_time + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + flex_model=flex_model, ) index = initialize_index(start, end, resolution) @@ -257,188 +260,220 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) # Set up commitments DataFrame - commitment = FlowCommitment( - name="energy", - quantity=commitment_quantities, - upwards_deviation_price=commitment_upwards_deviation_price, - downwards_deviation_price=commitment_downwards_deviation_price, - index=index, - ) - commitments.append(commitment) - - # Set up peak commitments - if self.flex_context.get("ems_peak_consumption_price") is not None: - ems_peak_consumption = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get("ems_peak_consumption_in_mw"), - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given - fill_sides=True, - ) - ems_peak_consumption_price = self.flex_context.get( - "ems_peak_consumption_price" - ) - ems_peak_consumption_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_peak_consumption_price, - unit=self.flex_context["shared_currency_unit"] + "/MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ) - - # Set up commitments DataFrame - commitment = FlowCommitment( - name="consumption peak", - quantity=ems_peak_consumption, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=ems_peak_consumption_price, - _type="any", - index=index, - ) - commitments.append(commitment) - if self.flex_context.get("ems_peak_production_price") is not None: - ems_peak_production = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get("ems_peak_production_in_mw"), - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given - fill_sides=True, - ) - ems_peak_production_price = self.flex_context.get( - "ems_peak_production_price" - ) - ems_peak_production_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_peak_production_price, - unit=self.flex_context["shared_currency_unit"] + "/MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ) + for d, flex_model_d in enumerate(flex_model): + # todo: use the right commodity prices for non-electricity devices + if flex_model_d["commodity"] != "electricity": + continue - # Set up commitments DataFrame commitment = FlowCommitment( - name="production peak", - quantity=-ems_peak_production, # production is negative quantity - # negative price because peaking in the downwards (production) direction is penalized - downwards_deviation_price=-ems_peak_production_price, - _type="any", + # todo: report aggregate energy costs, too (need to be backwards compatible) + name=f"energy {d}", + quantity=commitment_quantities, + upwards_deviation_price=commitment_upwards_deviation_price, + downwards_deviation_price=commitment_downwards_deviation_price, index=index, + device=d, + device_group=flex_model_d["commodity"], ) commitments.append(commitment) - # Set up capacity breach commitments and EMS capacity constraints - ems_consumption_breach_price = self.flex_context.get( - "ems_consumption_breach_price" - ) + # Set up peak commitments + if self.flex_context.get("ems_peak_consumption_price") is not None: + ems_peak_consumption = get_continuous_series_sensor_or_quantity( + variable_quantity=self.flex_context.get( + "ems_peak_consumption_in_mw" + ), + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given + fill_sides=True, + ) + ems_peak_consumption_price = self.flex_context.get( + "ems_peak_consumption_price" + ) + ems_peak_consumption_price = get_continuous_series_sensor_or_quantity( + variable_quantity=ems_peak_consumption_price, + unit=self.flex_context["shared_currency_unit"] + "/MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) - ems_production_breach_price = self.flex_context.get( - "ems_production_breach_price" - ) + # Set up commitments DataFrame + commitment = FlowCommitment( + name=f"consumption peak {d}", + quantity=ems_peak_consumption, + # positive price because breaching in the upwards (consumption) direction is penalized + upwards_deviation_price=ems_peak_consumption_price, + _type="any", + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) + if self.flex_context.get("ems_peak_production_price") is not None: + ems_peak_production = get_continuous_series_sensor_or_quantity( + variable_quantity=self.flex_context.get( + "ems_peak_production_in_mw" + ), + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given + fill_sides=True, + ) + ems_peak_production_price = self.flex_context.get( + "ems_peak_production_price" + ) + ems_peak_production_price = get_continuous_series_sensor_or_quantity( + variable_quantity=ems_peak_production_price, + unit=self.flex_context["shared_currency_unit"] + "/MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) - ems_constraints = initialize_df( - StorageScheduler.COLUMNS, start, end, resolution - ) - if ems_consumption_breach_price is not None: + # Set up commitments DataFrame + commitment = FlowCommitment( + name=f"production peak {d}", + quantity=-ems_peak_production, # production is negative quantity + # negative price because peaking in the downwards (production) direction is penalized + downwards_deviation_price=-ems_peak_production_price, + _type="any", + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) - # Convert to Series - any_ems_consumption_breach_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_consumption_breach_price, - unit=self.flex_context["shared_currency_unit"] + "/MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ) - all_ems_consumption_breach_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_consumption_breach_price, - unit=self.flex_context["shared_currency_unit"] - + "/MW*h", # from EUR/MWh to EUR/MW/resolution - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, + # Set up capacity breach commitments and EMS capacity constraints + ems_consumption_breach_price = self.flex_context.get( + "ems_consumption_breach_price" ) - # Set up commitments DataFrame to penalize any breach - commitment = FlowCommitment( - name="any consumption breach", - quantity=ems_consumption_capacity, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=any_ems_consumption_breach_price, - _type="any", - index=index, + ems_production_breach_price = self.flex_context.get( + "ems_production_breach_price" ) - commitments.append(commitment) - # Set up commitments DataFrame to penalize each breach - commitment = FlowCommitment( - name="all consumption breaches", - quantity=ems_consumption_capacity, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=all_ems_consumption_breach_price, - index=index, + ems_constraints = initialize_df( + StorageScheduler.COLUMNS, start, end, resolution ) - commitments.append(commitment) + if ems_consumption_breach_price is not None: - # Take the physical capacity as a hard constraint - ems_constraints["derivative max"] = ems_power_capacity_in_mw - else: - # Take the contracted capacity as a hard constraint - ems_constraints["derivative max"] = ems_consumption_capacity + # Convert to Series + any_ems_consumption_breach_price = ( + get_continuous_series_sensor_or_quantity( + variable_quantity=ems_consumption_breach_price, + unit=self.flex_context["shared_currency_unit"] + "/MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) + ) + all_ems_consumption_breach_price = ( + get_continuous_series_sensor_or_quantity( + variable_quantity=ems_consumption_breach_price, + unit=self.flex_context["shared_currency_unit"] + + "/MW*h", # from EUR/MWh to EUR/MW/resolution + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) + ) - if ems_production_breach_price is not None: + # Set up commitments DataFrame to penalize any breach + commitment = FlowCommitment( + name=f"any consumption breach {d}", + quantity=ems_consumption_capacity, + # positive price because breaching in the upwards (consumption) direction is penalized + upwards_deviation_price=any_ems_consumption_breach_price, + _type="any", + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) - # Convert to Series - any_ems_production_breach_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_production_breach_price, - unit=self.flex_context["shared_currency_unit"] + "/MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ) - all_ems_production_breach_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_production_breach_price, - unit=self.flex_context["shared_currency_unit"] - + "/MW*h", # from EUR/MWh to EUR/MW/resolution - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ) + # Set up commitments DataFrame to penalize each breach + commitment = FlowCommitment( + name=f"all consumption breaches {d}", + quantity=ems_consumption_capacity, + # positive price because breaching in the upwards (consumption) direction is penalized + upwards_deviation_price=all_ems_consumption_breach_price, + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) - # Set up commitments DataFrame to penalize any breach - commitment = FlowCommitment( - name="any production breach", - quantity=ems_production_capacity, - # negative price because breaching in the downwards (production) direction is penalized - downwards_deviation_price=-any_ems_production_breach_price, - _type="any", - index=index, - ) - commitments.append(commitment) + # Take the physical capacity as a hard constraint + ems_constraints["derivative max"] = ems_power_capacity_in_mw + else: + # Take the contracted capacity as a hard constraint + ems_constraints["derivative max"] = ems_consumption_capacity - # Set up commitments DataFrame to penalize each breach - commitment = FlowCommitment( - name="all production breaches", - quantity=ems_production_capacity, - # negative price because breaching in the downwards (production) direction is penalized - downwards_deviation_price=-all_ems_production_breach_price, - index=index, - ) - commitments.append(commitment) + if ems_production_breach_price is not None: - # Take the physical capacity as a hard constraint - ems_constraints["derivative min"] = -ems_power_capacity_in_mw - else: - # Take the contracted capacity as a hard constraint - ems_constraints["derivative min"] = ems_production_capacity + # Convert to Series + any_ems_production_breach_price = ( + get_continuous_series_sensor_or_quantity( + variable_quantity=ems_production_breach_price, + unit=self.flex_context["shared_currency_unit"] + "/MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) + ) + all_ems_production_breach_price = ( + get_continuous_series_sensor_or_quantity( + variable_quantity=ems_production_breach_price, + unit=self.flex_context["shared_currency_unit"] + + "/MW*h", # from EUR/MWh to EUR/MW/resolution + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ) + ) + + # Set up commitments DataFrame to penalize any breach + commitment = FlowCommitment( + name=f"any production breach {d}", + quantity=ems_production_capacity, + # negative price because breaching in the downwards (production) direction is penalized + downwards_deviation_price=-any_ems_production_breach_price, + _type="any", + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) + + # Set up commitments DataFrame to penalize each breach + commitment = FlowCommitment( + name=f"all production breaches {d}", + quantity=ems_production_capacity, + # negative price because breaching in the downwards (production) direction is penalized + downwards_deviation_price=-all_ems_production_breach_price, + index=index, + device=d, + device_group=flex_model_d["commodity"], + ) + commitments.append(commitment) + + # Take the physical capacity as a hard constraint + ems_constraints["derivative min"] = -ems_power_capacity_in_mw + else: + # Take the contracted capacity as a hard constraint + ems_constraints["derivative min"] = ems_production_capacity # Flow commitments per device @@ -932,6 +967,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 def convert_to_commitments( self, + flex_model, **timing_kwargs, ) -> list[FlowCommitment]: """Convert list of commitment specifications (dicts) to a list of FlowCommitments.""" @@ -969,7 +1005,13 @@ def convert_to_commitments( commitment_spec["index"] = initialize_index( start, end, timing_kwargs["resolution"] ) - commitments.append(FlowCommitment(**commitment_spec)) + for d, flex_model_d in enumerate(flex_model): + commitment = FlowCommitment( + device=d, + device_group=flex_model_d["commodity"], + **commitment_spec, + ) + commitments.append(commitment) return commitments From 325bfe8fdc879ce0d3826f9d3b65352fb8739261 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 10 Feb 2026 13:00:26 +0100 Subject: [PATCH 030/136] feat: add gas-price field to the Flex-context schema Signed-off-by: Ahmad-Wahid --- flexmeasures/data/schemas/scheduling/__init__.py | 7 +++++++ flexmeasures/data/schemas/scheduling/metadata.py | 5 +++++ flexmeasures/ui/static/openapi-specs.json | 7 +++++++ 3 files changed, 19 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index dd2caded11..148f095e44 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -299,6 +299,13 @@ class FlexContextSchema(Schema): data_key="aggregate-power", required=False, ) + gas_price = VariableQuantityField( + "/MWh", + data_key="gas-price", + required=False, + return_magnitude=False, + metadata=metadata.GAS_PRICE.to_dict(), + ) def set_default_breach_prices( self, data: dict, fields: list[str], price: ur.Quantity diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index e5a1f26f9d..0e5a2b3552 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -52,6 +52,11 @@ def to_dict(self): description="The electricity price applied to the site's aggregate production. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem, as long as the unit matches the ``consumption-price`` unit. [#old_production_price_field]_", example="0.12 EUR/kWh", ) +GAS_PRICE = MetaData( + description="The gas price applied to the site's aggregate gas consumption. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem", + example={"sensor": 6}, + # example="0.09 EUR/kWh", +) SITE_POWER_CAPACITY = MetaData( description="""Maximum achievable power at the site's grid connection point, in either direction. Becomes a hard constraint in the optimization problem, which is especially suitable for physical limitations. [#asymmetric]_ [#minimum_capacity_overlap]_ diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index da8db49566..6224dafe03 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -4106,6 +4106,13 @@ }, "aggregate-power": { "$ref": "#/components/schemas/SensorReference" + }, + "gas-price": { + "description": "The gas price applied to the site's aggregate gas consumption. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem", + "example": { + "sensor": 6 + }, + "$ref": "#/components/schemas/VariableQuantityOpenAPI" } }, "additionalProperties": false From 926162925150563e143b028f8bd0b033b749f2ee Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 10 Feb 2026 13:04:02 +0100 Subject: [PATCH 031/136] apply black Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index cf8c4cbe48..35e486e7cc 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,7 +1,11 @@ import pandas as pd import numpy as np -from flexmeasures.data.models.planning import Commitment, StockCommitment, FlowCommitment +from flexmeasures.data.models.planning import ( + Commitment, + StockCommitment, + FlowCommitment, +) from flexmeasures.data.models.planning.utils import ( initialize_index, add_tiny_price_slope, From 9dd58020efd5bd2165589e88e91689c323c3ef96 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 10 Feb 2026 22:55:05 +0100 Subject: [PATCH 032/136] feat: add a test case for two flexible devices with commodity Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 35e486e7cc..6e1e0a310e 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,6 +1,7 @@ import pandas as pd import numpy as np +from flexmeasures.data.services.utils import get_or_create_model from flexmeasures.data.models.planning import ( Commitment, StockCommitment, @@ -10,7 +11,10 @@ initialize_index, add_tiny_price_slope, ) +from flexmeasures.data.models.planning.storage import StorageScheduler +from flexmeasures.data.models.time_series import Sensor from flexmeasures.data.models.planning.linear_optimization import device_scheduler +from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType def test_multi_feed_device_scheduler_shared_buffer(): @@ -312,3 +316,100 @@ def test_each_type_assigns_unique_group_per_slot(): device=pd.Series("dev", index=idx), ) assert list(c.group) == list(range(len(idx))) + + +def test_two_flexible_assets_with_commodity(app, db): + """ + Test scheduling two flexible assets (battery + heat pump) + with explicit electricity commodity. + """ + # ---- asset types + battery_type = get_or_create_model(GenericAssetType, name="battery") + hp_type = get_or_create_model(GenericAssetType, name="heat-pump") + + # ---- time setup + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-02T00:00:00+01:00") + resolution = pd.Timedelta("1h") + + # ---- assets + battery = GenericAsset( + name="Battery", + generic_asset_type=battery_type, + attributes={"energy-capacity": "100 kWh"}, + ) + heat_pump = GenericAsset( + name="Heat Pump", + generic_asset_type=hp_type, + attributes={"energy-capacity": "50 kWh"}, + ) + db.session.add_all([battery, heat_pump]) + db.session.commit() + + # ---- sensors + battery_power = Sensor( + name="battery power", + unit="kW", + event_resolution=resolution, + generic_asset=battery, + ) + hp_power = Sensor( + name="heat pump power", + unit="kW", + event_resolution=resolution, + generic_asset=heat_pump, + ) + db.session.add_all([battery_power, hp_power]) + db.session.commit() + + # ---- flex-model (list = multi-asset) + flex_model = [ + { + # Battery as storage + "sensor": battery_power.id, + "commodity": "electricity", + "soc-at-start": 20.0, + "soc-min": 0.0, + "soc-max": 100.0, + "soc-targets": [{"datetime": "2024-01-01T23:00:00+01:00", "value": 80.0}], + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95, + }, + { + # Heat pump modeled as storage + "sensor": hp_power.id, + "commodity": "electricity", + "soc-at-start": 10.0, + "soc-min": 0.0, + "soc-max": 50.0, + "soc-targets": [{"datetime": "2024-01-01T23:00:00+01:00", "value": 40.0}], + "power-capacity": "10 kW", + "production-capacity": "0 kW", + "charging-efficiency": 0.95, + }, + ] + + # ---- flex-context (single electricity market) + flex_context = { + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh", + } + + # ---- run scheduler (use one asset as entry point) + scheduler = StorageScheduler( + asset_or_sensor=battery, + start=start, + end=end, + resolution=resolution, + belief_time=start, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + schedules = scheduler.compute(skip_validation=True) + + # ---- assertions + assert isinstance(schedules, list) + assert len(schedules) >= 2 # at least one schedule per device From b22c6d78bf2bc494b26d3d591023169908e202cc Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 16 Feb 2026 12:30:15 +0100 Subject: [PATCH 033/136] use expected datatypes Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 4caf906a60..6f0da76aa0 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -280,8 +280,8 @@ class Commitment: """ name: str - device: pd.Series = None - device_group: pd.Series = None + device: int | pd.Series = None + device_group: int | str | pd.Series = None index: pd.DatetimeIndex = field(repr=False, default=None) _type: str = field(repr=False, default="each") group: pd.Series = field(init=False) From c88af5c963ab8c1328385ef044ac9a533fcc72c2 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 16 Feb 2026 12:33:08 +0100 Subject: [PATCH 034/136] feat: split commitments per commodity Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/storage.py | 47 +++++++++++++++++--- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 48a593875a..0101b84d0f 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -159,12 +159,15 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Get info from flex-context consumption_price_sensor = self.flex_context.get("consumption_price_sensor") production_price_sensor = self.flex_context.get("production_price_sensor") + gas_price_sensor = self.flex_context.get("gas_price_sensor") + consumption_price = self.flex_context.get( "consumption_price", consumption_price_sensor ) production_price = self.flex_context.get( "production_price", production_price_sensor ) + gas_price = self.flex_context.get("gas_price", gas_price_sensor) # fallback to using the consumption price, for backwards compatibility if production_price is None: production_price = consumption_price @@ -175,6 +178,23 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Fetch the device's power capacity (required Sensor attribute) power_capacity_in_mw = self._get_device_power_capacity(flex_model, assets) + gas_deviation_prices = None + if gas_price is not None: + gas_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=gas_price, + unit=self.flex_context["shared_currency_unit"] + "/MWh", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ).to_frame(name="event_value") + ensure_prices_are_not_empty(gas_deviation_prices, gas_price) + gas_deviation_prices = ( + gas_deviation_prices.loc[start : end - resolution]["event_value"] + * resolution + / pd.Timedelta("1h") + ) + # Check for known prices or price forecasts up_deviation_prices = get_continuous_series_sensor_or_quantity( variable_quantity=consumption_price, @@ -262,19 +282,32 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Set up commitments DataFrame for d, flex_model_d in enumerate(flex_model): - # todo: use the right commodity prices for non-electricity devices - if flex_model_d["commodity"] != "electricity": - continue + commodity = flex_model_d.get("commodity", "electricity") + if commodity == "electricity": + up_price = commitment_upwards_deviation_price + down_price = commitment_downwards_deviation_price + elif commodity == "gas": + if gas_deviation_prices is None: + raise ValueError( + "Gas prices are required in the flex-context to set up gas flow commitments." + ) + up_price = gas_deviation_prices + down_price = gas_deviation_prices + else: + raise ValueError( + f"Unsupported commodity {commodity} in flex-model. Only 'electricity' and 'gas' are supported." + ) commitment = FlowCommitment( # todo: report aggregate energy costs, too (need to be backwards compatible) - name=f"energy {d}", + name=f"{commodity} energy {d}", quantity=commitment_quantities, - upwards_deviation_price=commitment_upwards_deviation_price, - downwards_deviation_price=commitment_downwards_deviation_price, + upwards_deviation_price=up_price, + downwards_deviation_price=down_price, + commodity=commodity, index=index, device=d, - device_group=flex_model_d["commodity"], + device_group=commodity, ) commitments.append(commitment) From c2341f1acff38b13e52b59706d04c079152bffa8 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 16 Feb 2026 12:33:08 +0100 Subject: [PATCH 035/136] feat: split commitments per commodity Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/storage.py | 47 +++++++++++++++++--- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 48a593875a..0101b84d0f 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -159,12 +159,15 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Get info from flex-context consumption_price_sensor = self.flex_context.get("consumption_price_sensor") production_price_sensor = self.flex_context.get("production_price_sensor") + gas_price_sensor = self.flex_context.get("gas_price_sensor") + consumption_price = self.flex_context.get( "consumption_price", consumption_price_sensor ) production_price = self.flex_context.get( "production_price", production_price_sensor ) + gas_price = self.flex_context.get("gas_price", gas_price_sensor) # fallback to using the consumption price, for backwards compatibility if production_price is None: production_price = consumption_price @@ -175,6 +178,23 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Fetch the device's power capacity (required Sensor attribute) power_capacity_in_mw = self._get_device_power_capacity(flex_model, assets) + gas_deviation_prices = None + if gas_price is not None: + gas_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=gas_price, + unit=self.flex_context["shared_currency_unit"] + "/MWh", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ).to_frame(name="event_value") + ensure_prices_are_not_empty(gas_deviation_prices, gas_price) + gas_deviation_prices = ( + gas_deviation_prices.loc[start : end - resolution]["event_value"] + * resolution + / pd.Timedelta("1h") + ) + # Check for known prices or price forecasts up_deviation_prices = get_continuous_series_sensor_or_quantity( variable_quantity=consumption_price, @@ -262,19 +282,32 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Set up commitments DataFrame for d, flex_model_d in enumerate(flex_model): - # todo: use the right commodity prices for non-electricity devices - if flex_model_d["commodity"] != "electricity": - continue + commodity = flex_model_d.get("commodity", "electricity") + if commodity == "electricity": + up_price = commitment_upwards_deviation_price + down_price = commitment_downwards_deviation_price + elif commodity == "gas": + if gas_deviation_prices is None: + raise ValueError( + "Gas prices are required in the flex-context to set up gas flow commitments." + ) + up_price = gas_deviation_prices + down_price = gas_deviation_prices + else: + raise ValueError( + f"Unsupported commodity {commodity} in flex-model. Only 'electricity' and 'gas' are supported." + ) commitment = FlowCommitment( # todo: report aggregate energy costs, too (need to be backwards compatible) - name=f"energy {d}", + name=f"{commodity} energy {d}", quantity=commitment_quantities, - upwards_deviation_price=commitment_upwards_deviation_price, - downwards_deviation_price=commitment_downwards_deviation_price, + upwards_deviation_price=up_price, + downwards_deviation_price=down_price, + commodity=commodity, index=index, device=d, - device_group=flex_model_d["commodity"], + device_group=commodity, ) commitments.append(commitment) From be05a199d6f61c56cf614b4658f39f91c4e1bfe8 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 16 Feb 2026 12:41:37 +0100 Subject: [PATCH 036/136] Revert "use expected datatypes" This reverts commit b22c6d78bf2bc494b26d3d591023169908e202cc. --- flexmeasures/data/models/planning/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 6f0da76aa0..4caf906a60 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -280,8 +280,8 @@ class Commitment: """ name: str - device: int | pd.Series = None - device_group: int | str | pd.Series = None + device: pd.Series = None + device_group: pd.Series = None index: pd.DatetimeIndex = field(repr=False, default=None) _type: str = field(repr=False, default="each") group: pd.Series = field(init=False) From f4ffd8a3f8ebe0dfe6902240ed7de5b39f751ad1 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 16 Feb 2026 12:50:54 +0100 Subject: [PATCH 037/136] feat: add a test case for different commodities Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 6e1e0a310e..055af2fb3e 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -413,3 +413,106 @@ def test_two_flexible_assets_with_commodity(app, db): # ---- assertions assert isinstance(schedules, list) assert len(schedules) >= 2 # at least one schedule per device + + +def test_mixed_gas_and_electricity_assets(app, db): + """ + Test scheduling two flexible assets with different commodities: + - Battery (electricity) + - Gas boiler (gas) + """ + + battery_type = get_or_create_model(GenericAssetType, name="battery") + boiler_type = get_or_create_model(GenericAssetType, name="gas-boiler") + + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-02T00:00:00+01:00") + resolution = pd.Timedelta("1h") + + battery = GenericAsset( + name="Battery", + generic_asset_type=battery_type, + attributes={"energy-capacity": "100 kWh"}, + ) + + gas_boiler = GenericAsset( + name="Gas Boiler", + generic_asset_type=boiler_type, + ) + + db.session.add_all([battery, gas_boiler]) + db.session.commit() + + battery_power = Sensor( + name="battery power", + unit="kW", + event_resolution=resolution, + generic_asset=battery, + ) + + boiler_power = Sensor( + name="boiler power", + unit="kW", + event_resolution=resolution, + generic_asset=gas_boiler, + ) + + db.session.add_all([battery_power, boiler_power]) + db.session.commit() + + flex_model = [ + { + # Electricity battery + "sensor": battery_power.id, + "commodity": "electricity", + "soc-at-start": 20.0, + "soc-min": 0.0, + "soc-max": 100.0, + "soc-targets": [{"datetime": "2024-01-01T23:00:00+01:00", "value": 80.0}], + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95, + }, + { + # Gas-powered device (no storage behavior) + "sensor": boiler_power.id, + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + }, + ] + + flex_context = { + "consumption-price": "100 EUR/MWh", # electricity price + "production-price": "100 EUR/MWh", + "gas-price": "50 EUR/MWh", # gas price + } + + scheduler = StorageScheduler( + asset_or_sensor=battery, + start=start, + end=end, + resolution=resolution, + belief_time=start, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + schedules = scheduler.compute(skip_validation=True) + + assert isinstance(schedules, list) + + scheduled_sensors = { + entry["sensor"] + for entry in schedules + if entry.get("name") == "storage_schedule" + } + + assert battery_power in scheduled_sensors + assert boiler_power in scheduled_sensors + + commitment_costs = [ + entry for entry in schedules if entry.get("name") == "commitment_costs" + ] + assert len(commitment_costs) == 1 From c43d2ad588fb6444e4084de0e6d9261293c4b748 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Feb 2026 11:06:37 +0100 Subject: [PATCH 038/136] fix: do not produce gas Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 1 + 1 file changed, 1 insertion(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 055af2fb3e..c288a4c619 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -479,6 +479,7 @@ def test_mixed_gas_and_electricity_assets(app, db): "commodity": "gas", "power-capacity": "30 kW", "consumption-capacity": "30 kW", + "production-capacity": "0 kW", }, ] From 0aa5b2bbd61ade2c841b6898126164a7a7b4ce61 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 26 Feb 2026 15:08:31 +0100 Subject: [PATCH 039/136] feat: create a flow commitment for prefering to charge sooner devices Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/storage.py | 30 +++++++++++++------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 0101b84d0f..836d81950e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -218,17 +218,6 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 start = pd.Timestamp(start).tz_convert("UTC") end = pd.Timestamp(end).tz_convert("UTC") - # Add tiny price slope to prefer charging now rather than later, and discharging later rather than now. - # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. - # todo: move to flow or stock commitment per device - if any(prefer_charging_sooner): - up_deviation_prices = add_tiny_price_slope( - up_deviation_prices, "event_value" - ) - down_deviation_prices = add_tiny_price_slope( - down_deviation_prices, "event_value" - ) - # Create Series with EMS capacities ems_power_capacity_in_mw = get_continuous_series_sensor_or_quantity( variable_quantity=self.flex_context.get("ems_power_capacity_in_mw"), @@ -531,6 +520,25 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) commitments.append(commitment) + # Add tiny price slope to prefer charging now rather than later, and discharging later rather than now. + # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. + for d, prefer_charging_sooner_d in enumerate(prefer_charging_sooner): + if prefer_charging_sooner_d: + tiny_price_slope = ( + add_tiny_price_slope(up_deviation_prices, "event_value") + - up_deviation_prices + ) + commitment = FlowCommitment( + name=f"prefer charging device {d} sooner", + # Prefer charging sooner by penalizing later consumption + upwards_deviation_price=tiny_price_slope, + # Prefer discharging later by penalizing earlier production + downwards_deviation_price=-tiny_price_slope, + index=index, + device=d, + ) + commitments.append(commitment) + # Set up device constraints: scheduled flexible devices for this EMS (from index 0 to D-1), plus the forecasted inflexible devices (at indices D to n). device_constraints = [ initialize_df(StorageScheduler.COLUMNS, start, end, resolution) From a48c6ba04c1ec637a554762b0481fc5a7018ace6 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 26 Feb 2026 18:53:47 +0100 Subject: [PATCH 040/136] add soc constraints for boiler Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index c288a4c619..9fde0f8cbf 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -480,6 +480,10 @@ def test_mixed_gas_and_electricity_assets(app, db): "power-capacity": "30 kW", "consumption-capacity": "30 kW", "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0, }, ] From c58965facd6b70d07bcb21fda98bfc7ecb011a47 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 26 Feb 2026 19:30:59 +0100 Subject: [PATCH 041/136] add some assert statments Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 51 +++++++++++++++---- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 9fde0f8cbf..0754329c2e 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -507,17 +507,50 @@ def test_mixed_gas_and_electricity_assets(app, db): schedules = scheduler.compute(skip_validation=True) assert isinstance(schedules, list) + assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs - scheduled_sensors = { - entry["sensor"] - for entry in schedules - if entry.get("name") == "storage_schedule" - } - - assert battery_power in scheduled_sensors - assert boiler_power in scheduled_sensors - + # Extract schedules by type + storage_schedules = [ + entry for entry in schedules if entry.get("name") == "storage_schedule" + ] commitment_costs = [ entry for entry in schedules if entry.get("name") == "commitment_costs" ] + + assert len(storage_schedules) == 2 assert len(commitment_costs) == 1 + + # Get battery schedule + battery_schedule = next( + entry for entry in storage_schedules if entry["sensor"] == battery_power + ) + battery_data = battery_schedule["data"] + + early_charging_hours = battery_data.iloc[:3] + assert (early_charging_hours > 0).all(), "Battery should charge early" + + assert battery_data.iloc[-1] < 0, "Battery should discharge at the end" + + middle_hours = battery_data.iloc[4:-2] + assert (middle_hours == 0).all(), "Battery should be idle during middle hours" + + boiler_schedule = next( + entry for entry in storage_schedules if entry["sensor"] == boiler_power + ) + boiler_data = boiler_schedule["data"] + + assert (boiler_data == 1.0).all(), "Boiler should have constant 1 kW consumption" + + costs_data = commitment_costs[0]["data"] + + assert ( + costs_data["electricity energy 0"] > 4.0 + ), "Battery electricity cost should be around 4.3 EUR" + assert costs_data["gas energy 1"] > 1.0, "Boiler gas cost should be around 1.2 EUR" + + # charging preference costs are roughly 2x curtailing preference costs + # (because curtailing uses 0.5x multiplier) + assert ( + costs_data["prefer charging device 0 sooner"] + > costs_data["prefer curtailing device 0 later"] + ), "Charging preference should have higher cost than curtailing (no 0.5x multiplier)" From 515f34a9cecc7d7d9ebd0a210c816d176eed76b5 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Wed, 4 Mar 2026 13:43:01 +0100 Subject: [PATCH 042/136] update and add new assertions with clear explanation Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 186 +++++++++++++++++- 1 file changed, 177 insertions(+), 9 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 0754329c2e..c9682418ce 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,5 +1,6 @@ import pandas as pd import numpy as np +import pytest from flexmeasures.data.services.utils import get_or_create_model from flexmeasures.data.models.planning import ( @@ -410,9 +411,127 @@ def test_two_flexible_assets_with_commodity(app, db): schedules = scheduler.compute(skip_validation=True) - # ---- assertions assert isinstance(schedules, list) - assert len(schedules) >= 2 # at least one schedule per device + assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs + + # Extract schedules by type + storage_schedules = [ + entry for entry in schedules if entry.get("name") == "storage_schedule" + ] + commitment_costs = [ + entry for entry in schedules if entry.get("name") == "commitment_costs" + ] + + assert len(storage_schedules) == 2 + assert len(commitment_costs) == 1 + + # Get battery schedule + battery_schedule = next( + entry for entry in storage_schedules if entry["sensor"] == battery_power + ) + battery_data = battery_schedule["data"] + + # Get heat pump schedule + hp_schedule = next( + entry for entry in storage_schedules if entry["sensor"] == hp_power + ) + hp_data = hp_schedule["data"] + + # Verify both devices charge to meet their targets + assert (battery_data > 0).any(), "Battery should charge at some point" + assert (hp_data > 0).any(), "Heat pump should charge at some point" + + costs_data = commitment_costs[0]["data"] + + # Battery: 60kWh Δ (20→80) / 0.95 eff × 100 EUR/MWh = 6.32 EUR (charge) + discharge loss ≈ 4.32 EUR + assert costs_data["electricity energy 0"] == pytest.approx(4.32, rel=1e-2), ( + f"Battery electricity cost (charges 60kWh with 95% efficiency + discharge): " + f"60kWh/0.95 × (100 EUR/MWh) = 4.32 EUR, " + f"got {costs_data['electricity energy 0']}" + ) + + # Heat pump: 30kWh Δ (10→40) / 0.95 eff × 100 EUR/MWh ≈ 3.16 EUR (no discharge, prod-cap=0) + assert costs_data["electricity energy 1"] == pytest.approx(3.16, rel=1e-2), ( + f"Heat pump electricity cost (charges 30kWh with 95% efficiency): " + f"30kWh/0.95 × (100 EUR/MWh) = 3.16 EUR, " + f"got {costs_data['electricity energy 1']}" + ) + + # Total electricity: battery (4.32) + heat pump (3.16) = 7.48 EUR + total_electricity_cost = sum( + v for k, v in costs_data.items() if k.startswith("electricity energy") + ) + assert total_electricity_cost == pytest.approx(7.47, rel=1e-2), ( + f"Total electricity cost (battery 4.32 + heat pump 3.16): " + f"= 7.48 EUR, got {total_electricity_cost}" + ) + + # Battery charges early (3-4h @20kW): tiny slope cost ≈ 2.30e-6 EUR (negligible tiebreaker) + assert costs_data["prefer charging device 0 sooner"] == pytest.approx( + 2.30e-6, rel=1e-2 + ), ( + f"Battery charging preference (charges early at low-slope cost): " + f"= 2.30e-6 EUR, got {costs_data['prefer charging device 0 sooner']}" + ) + + # Battery idle periods with 0.5× multiplier: = 0.5 × 2.30e-6 = 1.15e-6 EUR + assert costs_data["prefer curtailing device 0 later"] == pytest.approx( + 1.15e-6, rel=1e-2 + ), ( + f"Battery curtailing preference (idle periods with 0.5× multiplier): " + f"= 0.5 × 2.30e-6 = 1.15e-6 EUR, got {costs_data['prefer curtailing device 0 later']}" + ) + + # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) + assert ( + costs_data["prefer charging device 0 sooner"] + > costs_data["prefer curtailing device 0 later"] + ), ( + f"Battery charging preference should cost ~2× more than curtailing " + f"due to 0.5× multiplier on curtailing slopes. " + f"Ratio: {costs_data['prefer charging device 0 sooner'] / costs_data['prefer curtailing device 0 later']:.1f}×" + ) + + # Heat pump charges ~30 kWh (half of battery's 60 kWh) at 10 kW: tiny slope cost ≈ 1.51e-7 EUR + assert costs_data["prefer charging device 1 sooner"] == pytest.approx( + 1.51e-7, rel=1e-2 + ), ( + f"Heat pump charging preference (charges 30kWh, smaller than battery): " + f"= 1.51e-7 EUR, got {costs_data['prefer charging device 1 sooner']}" + ) + + # Heat pump idle periods with 0.5× multiplier should be positive + assert ( + costs_data["prefer curtailing device 1 later"] > 0 + ), "Heat pump curtailing preference cost should be positive" + # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) + assert ( + costs_data["prefer charging device 1 sooner"] + > costs_data["prefer curtailing device 1 later"] + ), ( + f"Heat pump charging preference should cost ~2× more than curtailing " + f"due to 0.5× multiplier. " + f"Ratio: {costs_data['prefer charging device 1 sooner'] / costs_data['prefer curtailing device 1 later']:.1f}×" + ) + + # ---- RELATIVE COSTS: Battery vs Heat Pump + # Battery moves 60 kWh, Heat Pump moves 30 kWh (2:1 ratio) + # Preference costs should roughly reflect this energy ratio + # Battery total preference: 2.30e-6 + 1.15e-6 = 3.45e-6 EUR + # Heat Pump total preference: 1.51e-7 + ~7.5e-8 ≈ 2.26e-7 EUR + # Ratio: 3.45e-6 / 2.26e-7 ≈ 15× (battery has much higher preference costs) + battery_total_pref = ( + costs_data["prefer charging device 0 sooner"] + + costs_data["prefer curtailing device 0 later"] + ) + hp_total_pref = ( + costs_data["prefer charging device 1 sooner"] + + costs_data["prefer curtailing device 1 later"] + ) + assert battery_total_pref > hp_total_pref, ( + f"Battery preference costs ({battery_total_pref:.2e}) should be higher than " + f"heat pump ({hp_total_pref:.2e}) since battery moves more energy (60 kWh vs 30 kWh)" + ) def test_mixed_gas_and_electricity_assets(app, db): @@ -543,14 +662,63 @@ def test_mixed_gas_and_electricity_assets(app, db): costs_data = commitment_costs[0]["data"] - assert ( - costs_data["electricity energy 0"] > 4.0 - ), "Battery electricity cost should be around 4.3 EUR" - assert costs_data["gas energy 1"] > 1.0, "Boiler gas cost should be around 1.2 EUR" + # Battery: 60kWh Δ (20→80) / 0.95 eff × 100 EUR/MWh + discharge loss ≈ 4.32 EUR + assert costs_data["electricity energy 0"] == pytest.approx(4.32, rel=1e-2), ( + f"Electricity energy cost (battery charging phase ~3h at 20kW with 95% efficiency " + f"+ discharge at end): 60kWh/0.95 × (100 EUR/MWh) = 4.32 EUR, " + f"got {costs_data['electricity energy 0']}" + ) + + # Boiler: constant 1kW × 24h = 24 kWh = 0.024 MWh × 50 EUR/MWh = 1.20 EUR (no efficiency loss) + assert costs_data["gas energy 1"] == pytest.approx(1.20, rel=1e-2), ( + f"Gas energy cost (boiler constant 1kW for 24h): " + f"1 kW × 24h = 24 kWh = 0.024 MWh × 50 EUR/MWh = 1.20 EUR, " + f"got {costs_data['gas energy 1']}" + ) + + # Battery charges early (3h @20kW): tiny slope cost = 3h × 20kW × (24/1e6) = 2.30e-6 EUR + assert costs_data["prefer charging device 0 sooner"] == pytest.approx( + 2.30e-6, rel=1e-2 + ), ( + f"Charging preference (battery charges early at low-slope cost): " + f"accumulates tiny slope penalty over charging period = 2.30e-6 EUR, " + f"got {costs_data['prefer charging device 0 sooner']}" + ) - # charging preference costs are roughly 2x curtailing preference costs - # (because curtailing uses 0.5x multiplier) + # Battery idle periods with 0.5× multiplier = 0.5 × 2.30e-6 = 1.15e-6 EUR (prioritizes early charge) + assert costs_data["prefer curtailing device 0 later"] == pytest.approx( + 1.15e-6, rel=1e-2 + ), ( + f"Curtailing preference (battery idle periods with 0.5× multiplier): " + f"= 0.5 × charging preference = 1.15e-6 EUR (weaker to prioritize early charging), " + f"got {costs_data['prefer curtailing device 0 later']}" + ) + + # Boiler: constant 1kW × 24h × tiny_slope = 24h × 1kW × (24/1e6) = 1.20e-6 EUR (no flexibility) + assert costs_data["prefer charging device 1 sooner"] == pytest.approx( + 1.20e-6, rel=1e-2 + ), ( + f"Charging preference (boiler 1kW constant load, 24h duration): " + f"1 kW × 24h × tiny_slope = 1.20e-6 EUR (degenerate: no flexibility), " + f"got {costs_data['prefer charging device 1 sooner']}" + ) + + # Boiler curtailing with 0.5× multiplier = 0.5 × 1.20e-6 = 6.00e-7 EUR (no flexibility) + assert costs_data["prefer curtailing device 1 later"] == pytest.approx( + 6.00e-7, rel=1e-2 + ), ( + f"Curtailing preference (boiler with 0.5× multiplier, no flexibility): " + f"= 0.5 × charging preference = 6.00e-7 EUR, " + f"got {costs_data['prefer curtailing device 1 later']}" + ) + + # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) assert ( costs_data["prefer charging device 0 sooner"] > costs_data["prefer curtailing device 0 later"] - ), "Charging preference should have higher cost than curtailing (no 0.5x multiplier)" + ), ( + f"Battery charging preference (2.30e-6) should cost ~2× more than curtailing " + f"(1.15e-6) due to 0.5× multiplier on curtailing slopes. " + f"This ensures optimizer prioritizes filling battery early over idling. " + f"Ratio: {costs_data['prefer charging device 0 sooner'] / costs_data['prefer curtailing device 0 later']:.1f}×" + ) From 4d77344063fe449d78ba94f21b22e7903cbfdc67 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Wed, 4 Mar 2026 13:45:39 +0100 Subject: [PATCH 043/136] update the docstring Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index c9682418ce..9f2bf36fda 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -536,9 +536,8 @@ def test_two_flexible_assets_with_commodity(app, db): def test_mixed_gas_and_electricity_assets(app, db): """ - Test scheduling two flexible assets with different commodities: - - Battery (electricity) - - Gas boiler (gas) + Test scheduling with mixed commodities: battery (electricity) and boiler (gas). + Verify cost calculations for both commodity types. """ battery_type = get_or_create_model(GenericAssetType, name="battery") From 2becd028513a24a727effbcaf18a8bbd25dd3c88 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 02:02:30 +0100 Subject: [PATCH 044/136] refactor: move tiny-price-slope decleration out of the for loop Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 9f2bf36fda..d40fadc43b 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -623,7 +623,7 @@ def test_mixed_gas_and_electricity_assets(app, db): ) schedules = scheduler.compute(skip_validation=True) - + breakpoint() assert isinstance(schedules, list) assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs From d9d4f94e92ffbd1c42f741d9fe085032e8e5123f Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 02:03:12 +0100 Subject: [PATCH 045/136] Revert "refactor: move tiny-price-slope decleration out of the for loop" This reverts commit 2becd028513a24a727effbcaf18a8bbd25dd3c88. --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index d40fadc43b..9f2bf36fda 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -623,7 +623,7 @@ def test_mixed_gas_and_electricity_assets(app, db): ) schedules = scheduler.compute(skip_validation=True) - breakpoint() + assert isinstance(schedules, list) assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs From 2a22fae1234da55463905d1cf0309100c924ef9d Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 02:03:49 +0100 Subject: [PATCH 046/136] refactor: move tiny-price-slope decleration out of the for loop Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/storage.py | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 836d81950e..d74466e716 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -502,19 +502,18 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Add tiny price slope to prefer curtailing later rather than now. # The price slope is half of the slope to prefer charging sooner + tiny_price_slope = ( + add_tiny_price_slope(up_deviation_prices, "event_value") + - up_deviation_prices + ) for d, prefer_curtailing_later_d in enumerate(prefer_curtailing_later): if prefer_curtailing_later_d: - tiny_price_slope = ( - add_tiny_price_slope(up_deviation_prices, "event_value") - - up_deviation_prices - ) - tiny_price_slope *= 0.5 commitment = FlowCommitment( name=f"prefer curtailing device {d} later", # Prefer curtailing consumption later by penalizing later consumption - upwards_deviation_price=tiny_price_slope, + upwards_deviation_price=tiny_price_slope * 0.5, # Prefer curtailing production later by penalizing later production - downwards_deviation_price=-tiny_price_slope, + downwards_deviation_price=-tiny_price_slope * 0.5, index=index, device=d, ) @@ -524,10 +523,6 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. for d, prefer_charging_sooner_d in enumerate(prefer_charging_sooner): if prefer_charging_sooner_d: - tiny_price_slope = ( - add_tiny_price_slope(up_deviation_prices, "event_value") - - up_deviation_prices - ) commitment = FlowCommitment( name=f"prefer charging device {d} sooner", # Prefer charging sooner by penalizing later consumption From f38d6d85f3c8bfa417c0ba4e14622f1eacbb9133 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 20:08:04 +0100 Subject: [PATCH 047/136] fix: add data_key attr Signed-off-by: Ahmad-Wahid --- flexmeasures/data/schemas/scheduling/storage.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index 1154c4c97f..19a752f3c9 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -226,6 +226,7 @@ class StorageFlexModelSchema(Schema): ) commodity = fields.Str( required=False, + data_key="commodity", load_default="electricity", validate=OneOf(["electricity", "gas"]), metadata=dict(description="Commodity label for this device/asset."), @@ -507,6 +508,7 @@ class DBStorageFlexModelSchema(Schema): commodity = fields.Str( required=False, + data_key="commodity", load_default="electricity", validate=OneOf(["electricity", "gas"]), metadata=dict(description="Commodity label for this device/asset."), From 007e51497b8f532e8312c838d92e55d7d1219272 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 20:09:52 +0100 Subject: [PATCH 048/136] add missing commodity description and it's field in ui flexmodel schema Signed-off-by: Ahmad-Wahid --- flexmeasures/data/schemas/scheduling/__init__.py | 9 +++++++++ flexmeasures/data/schemas/scheduling/metadata.py | 8 +++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 148f095e44..21333884c7 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -738,6 +738,15 @@ def _to_currency_per_mwh(price_unit: str) -> str: }, "example-units": EXAMPLE_UNIT_TYPES["power"], }, + "commodity": { + "default": "electricity", + "description": rst_to_openapi(metadata.COMMODITY.description), + "types": { + "backend": "typeOne", + "ui": "One fixed value only.", + }, + "options": ["electricity", "gas"], + }, } diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index 0e5a2b3552..4165cb003e 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -189,7 +189,13 @@ def to_dict(self): # FLEX-MODEL - +COMMODITY = MetaData( + description="""Commodity type for this storage flex-model. +Allowed values are ``electricity`` and ``gas``. +Defaults to ``electricity``. +""", + example="electricity", +) STATE_OF_CHARGE = MetaData( description="Sensor used to record the scheduled state of charge.", example={"sensor": 12}, From 0bff8bcc8fd87836fbdeb38ac4327ec76b28c7dc Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Fri, 6 Mar 2026 20:17:21 +0100 Subject: [PATCH 049/136] fix: add missing gas-price field in UI Flexcontext schema Signed-off-by: Ahmad-Wahid --- flexmeasures/data/schemas/scheduling/__init__.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 21333884c7..2ca56c152e 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -582,6 +582,11 @@ def _to_currency_per_mwh(price_unit: str) -> str: "description": rst_to_openapi(metadata.AGGREGATE_POWER.description), "example-units": EXAMPLE_UNIT_TYPES["power"], }, + "gas-price": { + "default": None, + "description": rst_to_openapi(metadata.GAS_PRICE.description), + "example-units": EXAMPLE_UNIT_TYPES["energy-price"], + }, } UI_FLEX_MODEL_SCHEMA: Dict[str, Dict[str, Any]] = { From 82f807ec91da001378a9a7a822e5e03f4bb59ccc Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 16:21:40 +0100 Subject: [PATCH 050/136] fix: wrong timezone; the test relied on the preference to charge sooner and discharge later, rather than on the EPEX price transition, as the inline test documentation advertised Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_solver.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 919936d315..1f0a5baa27 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -2235,11 +2235,14 @@ def test_battery_storage_different_units( battery_name="Test battery", power_sensor_name=power_sensor_name, ) - tz = pytz.timezone("Europe/Amsterdam") + tz = pytz.timezone(epex_da.timezone) # transition from cheap to expensive (90 -> 100) start = tz.localize(datetime(2015, 1, 2, 14, 0, 0)) end = tz.localize(datetime(2015, 1, 2, 16, 0, 0)) + assert len(epex_da.search_beliefs(start, end)) == 2 + assert epex_da.search_beliefs(start, end).values[0][0] == 90 + assert epex_da.search_beliefs(start, end).values[1][0] == 100 resolution = timedelta(minutes=15) flex_model = { From 128550f5b03f76e50bae7ec8f87b31162fb25d9d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 17:17:24 +0100 Subject: [PATCH 051/136] feat: move preference to charge sooner and discharge later into a StockCommitment to prefer being full Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 35 +++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 6fb7aa5e94..dd53db19fc 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -198,17 +198,6 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 start = pd.Timestamp(start).tz_convert("UTC") end = pd.Timestamp(end).tz_convert("UTC") - # Add tiny price slope to prefer charging now rather than later, and discharging later rather than now. - # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. - # todo: move to flow or stock commitment per device - if any(prefer_charging_sooner): - up_deviation_prices = add_tiny_price_slope( - up_deviation_prices, "event_value" - ) - down_deviation_prices = add_tiny_price_slope( - down_deviation_prices, "event_value" - ) - # Create Series with EMS capacities ems_power_capacity_in_mw = get_continuous_series_sensor_or_quantity( variable_quantity=self.flex_context.get("ems_power_capacity_in_mw"), @@ -463,6 +452,28 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) commitments.append(commitment) + # Use a tiny price slope to prefer a fuller SoC sooner rather than later + # This corresponds to a preference for charging now rather than later, and discharging later rather than now. + # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. + for d, prefer_charging_sooner_d in enumerate(prefer_charging_sooner): + if prefer_charging_sooner_d: + tiny_price_slope = ( + add_tiny_price_slope(up_deviation_prices, "event_value") + - up_deviation_prices + ) + commitment = StockCommitment( + name=f"prefer charging device {d} sooner", + quantity=soc_max[d] - soc_at_start[d], + # Prefer curtailing consumption later by penalizing later consumption + upwards_deviation_price=0, + # Prefer curtailing production later by penalizing later production + downwards_deviation_price=-0.00000001, + # downwards_deviation_price=-tiny_price_slope / 1000000,#0.00000001, + index=index, + device=d, + ) + commitments.append(commitment) + # Set up device constraints: scheduled flexible devices for this EMS (from index 0 to D-1), plus the forecasted inflexible devices (at indices D to n). device_constraints = [ initialize_df(StorageScheduler.COLUMNS, start, end, resolution) @@ -940,7 +951,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 def convert_to_commitments( self, **timing_kwargs, - ) -> list[FlowCommitment]: + ) -> list[FlowCommitment | StockCommitment]: """Convert list of commitment specifications (dicts) to a list of FlowCommitments.""" commitment_specs = self.flex_context.get("commitments", []) if len(commitment_specs) == 0: From 130b9dd6528291903cb7582d2763e843ca94241b Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 17:38:42 +0100 Subject: [PATCH 052/136] fix: test case no longer relies on arbitrage opportunity coming from artificial price slope Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_solver.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 1f0a5baa27..d85a45243b 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -816,8 +816,8 @@ def compute_schedule(flex_model): # soc maxima and soc minima soc_maxima = [ - {"datetime": "2015-01-02T15:00:00+01:00", "value": 1.0}, - {"datetime": "2015-01-02T16:00:00+01:00", "value": 1.0}, + {"datetime": "2015-01-02T12:00:00+01:00", "value": 1.0}, + {"datetime": "2015-01-02T13:00:00+01:00", "value": 1.0}, ] soc_minima = [{"datetime": "2015-01-02T08:00:00+01:00", "value": 3.5}] @@ -853,7 +853,7 @@ def compute_schedule(flex_model): # test for soc_maxima # check that the local maximum constraint is respected - assert soc_schedule_2.loc["2015-01-02T15:00:00+01:00"] <= 1.0 + assert soc_schedule_2.loc["2015-01-02T13:00:00+01:00"] <= 1.0 # test for soc_targets # check that the SOC target (at 19 pm, local time) is met From 1eab8282406d85d343ed6b22f73f781f68e45777 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 17:40:42 +0100 Subject: [PATCH 053/136] feat: check for optimal schedule Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_solver.py | 1 + 1 file changed, 1 insertion(+) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index d85a45243b..6ee32f8b59 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -270,6 +270,7 @@ def run_test_charge_discharge_sign( for soc_at_start_d in soc_at_start ], ) + assert results.solver.termination_condition == "optimal" device_power_sign = pd.Series(model.device_power_sign.extract_values())[0] device_power_up = pd.Series(model.device_power_up.extract_values())[0] From b9bc4a9c0786907c25737a2b5c20492501cf79ec Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 17:58:51 +0100 Subject: [PATCH 054/136] feat: prefer a full storage earlier over later Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 6 ++++-- flexmeasures/data/models/planning/utils.py | 19 +++++++++++++++---- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index dd53db19fc..b473ed650a 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -458,7 +458,9 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 for d, prefer_charging_sooner_d in enumerate(prefer_charging_sooner): if prefer_charging_sooner_d: tiny_price_slope = ( - add_tiny_price_slope(up_deviation_prices, "event_value") + add_tiny_price_slope( + up_deviation_prices, "event_value", order="desc" + ) - up_deviation_prices ) commitment = StockCommitment( @@ -467,7 +469,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Prefer curtailing consumption later by penalizing later consumption upwards_deviation_price=0, # Prefer curtailing production later by penalizing later production - downwards_deviation_price=-0.00000001, + downwards_deviation_price=-tiny_price_slope, # downwards_deviation_price=-tiny_price_slope / 1000000,#0.00000001, index=index, device=d, diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index 549ef6a01a..c7e20860f0 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -2,6 +2,7 @@ from packaging import version from datetime import date, datetime, timedelta +from typing import Literal from flask import current_app import pandas as pd @@ -67,7 +68,10 @@ def initialize_index( def add_tiny_price_slope( - orig_prices: pd.DataFrame, col_name: str = "event_value", d: float = 10**-4 + orig_prices: pd.DataFrame, + col_name: str = "event_value", + d: float = 10**-4, + order: Literal["asc", "desc"] = "asc", ) -> pd.DataFrame: """Add tiny price slope to col_name to represent e.g. inflation as a simple linear price increase. This is meant to break ties, when multiple time slots have equal prices, in favour of acting sooner. @@ -79,9 +83,16 @@ def add_tiny_price_slope( max_penalty = price_spread * d else: max_penalty = d - prices[col_name] = prices[col_name] + np.linspace( - 0, max_penalty, prices[col_name].size - ) + if order == "asc": + prices[col_name] = prices[col_name] + np.linspace( + 0, max_penalty, prices[col_name].size + ) + elif order == "desc": + prices[col_name] = prices[col_name] + np.linspace( + max_penalty, 0, prices[col_name].size + ) + else: + raise ValueError("order must be 'asc' or 'desc'") return prices From 57df5c31d9f6e3e9a868c3e35bff4db7fda2d003 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 13 Mar 2026 18:04:18 +0100 Subject: [PATCH 055/136] docs: update commitment name and inline comments Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index b473ed650a..3c2388d435 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -464,11 +464,10 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 - up_deviation_prices ) commitment = StockCommitment( - name=f"prefer charging device {d} sooner", + name=f"prefer a full storage {d} sooner", quantity=soc_max[d] - soc_at_start[d], - # Prefer curtailing consumption later by penalizing later consumption upwards_deviation_price=0, - # Prefer curtailing production later by penalizing later production + # Penalize not being full, with lower penalties later downwards_deviation_price=-tiny_price_slope, # downwards_deviation_price=-tiny_price_slope / 1000000,#0.00000001, index=index, From ce71637238b1d4c1eb81a400edb1fce9a7c9a67c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 14 Mar 2026 11:30:46 +0100 Subject: [PATCH 056/136] docs: touch up test explanation Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_solver.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 6ee32f8b59..b46ee0f0b1 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -1788,7 +1788,7 @@ def test_battery_stock_delta_sensor( - Battery of size 2 MWh. - Consumption capacity of the battery is 2 MW. - The battery cannot discharge. - With these settings, the battery needs to charge at a power or greater than the usage forecast + With these settings, the battery needs to charge at a power equal or greater than the usage forecast to keep the SOC within bounds ([0, 2 MWh]). """ _, battery = get_sensors_from_db(db, add_battery_assets) From f6183df2df24639858907f7ae681636673d8140b Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 14 Mar 2026 11:43:55 +0100 Subject: [PATCH 057/136] fix: update test case given preference for a full battery Signed-off-by: F.N. Claessen --- .../data/models/planning/tests/test_solver.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index b46ee0f0b1..e810e8c854 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -1791,7 +1791,7 @@ def test_battery_stock_delta_sensor( With these settings, the battery needs to charge at a power equal or greater than the usage forecast to keep the SOC within bounds ([0, 2 MWh]). """ - _, battery = get_sensors_from_db(db, add_battery_assets) + epex_da, battery = get_sensors_from_db(db, add_battery_assets) tz = pytz.timezone("Europe/Amsterdam") start = tz.localize(datetime(2015, 1, 1)) end = tz.localize(datetime(2015, 1, 2)) @@ -1836,9 +1836,21 @@ def test_battery_stock_delta_sensor( with pytest.raises(InfeasibleProblemException): scheduler.compute() elif stock_delta_sensor is None: - # No usage -> the battery does not charge + # No usage -> the battery only charges when energy is free + free_hour = "2015-01-01 17:00:00+00:00" + prices = epex_da.search_beliefs(start, end) + zero_prices = prices[prices.event_value == 0] + assert len(zero_prices) == 1, "this test assumes a single hour of free energy" + assert ( + len(zero_prices[zero_prices.event_starts == free_hour]) == 1 + ), "this test assumes free energy from 5 to 6 PM UTC" schedule = scheduler.compute() - assert all(schedule == 0) + assert all( + schedule[schedule.index != free_hour] == 0 + ), "no charging expected when energy is not free, given no soc-usage" + assert all( + schedule[schedule.index == free_hour] == capacity + ), "max charging expected when energy is free, because of preference to have a full SoC" else: # Some usage -> the battery needs to charge schedule = scheduler.compute() From ed471a8b836c60da2964e2a666c4613db98bcf5d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 14 Mar 2026 11:46:52 +0100 Subject: [PATCH 058/136] delete: clean up comment Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 1 - 1 file changed, 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 3c2388d435..4d1e551c95 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -469,7 +469,6 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 upwards_deviation_price=0, # Penalize not being full, with lower penalties later downwards_deviation_price=-tiny_price_slope, - # downwards_deviation_price=-tiny_price_slope / 1000000,#0.00000001, index=index, device=d, ) From 7611fb9eadbde61efa317f8ace3e7adbb9d971de Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 14 Mar 2026 11:59:56 +0100 Subject: [PATCH 059/136] feat: model the preference to curtail later within the same StockCommitment, using a tiny price slope to prefer a fuller SoC sooner rather than later, by lowering penalties later Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 38 +++++++------------- 1 file changed, 12 insertions(+), 26 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 4d1e551c95..44024c0cb8 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -430,32 +430,13 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Take the contracted capacity as a hard constraint ems_constraints["derivative min"] = ems_production_capacity - # Flow commitments per device + # Commitments per device - # Add tiny price slope to prefer curtailing later rather than now. - # The price slope is half of the slope to prefer charging sooner - for d, prefer_curtailing_later_d in enumerate(prefer_curtailing_later): - if prefer_curtailing_later_d: - tiny_price_slope = ( - add_tiny_price_slope(up_deviation_prices, "event_value") - - up_deviation_prices - ) - tiny_price_slope *= 0.5 - commitment = FlowCommitment( - name=f"prefer curtailing device {d} later", - # Prefer curtailing consumption later by penalizing later consumption - upwards_deviation_price=tiny_price_slope, - # Prefer curtailing production later by penalizing later production - downwards_deviation_price=-tiny_price_slope, - index=index, - device=d, - ) - commitments.append(commitment) - - # Use a tiny price slope to prefer a fuller SoC sooner rather than later + # StockCommitment per device to prefer a full storage by penalizing not being full # This corresponds to a preference for charging now rather than later, and discharging later rather than now. - # We penalise future consumption and reward future production with at most 1 per thousand times the energy price spread. - for d, prefer_charging_sooner_d in enumerate(prefer_charging_sooner): + for d, (prefer_charging_sooner_d, prefer_curtailing_later_d) in enumerate( + zip(prefer_charging_sooner, prefer_curtailing_later) + ): if prefer_charging_sooner_d: tiny_price_slope = ( add_tiny_price_slope( @@ -463,12 +444,17 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) - up_deviation_prices ) + if prefer_curtailing_later: + # Use a tiny price slope to prefer a fuller SoC sooner rather than later, by lowering penalties later + penalty = tiny_price_slope + else: + # Constant penalty + penalty = tiny_price_slope[0] commitment = StockCommitment( name=f"prefer a full storage {d} sooner", quantity=soc_max[d] - soc_at_start[d], upwards_deviation_price=0, - # Penalize not being full, with lower penalties later - downwards_deviation_price=-tiny_price_slope, + downwards_deviation_price=-penalty, index=index, device=d, ) From bf16e63606ed5a2889fe25b45c0fd4f28a40b486 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 14 Mar 2026 12:17:35 +0100 Subject: [PATCH 060/136] fix: reduce tiny price slope Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 44024c0cb8..5ef5517b36 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -440,7 +440,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 if prefer_charging_sooner_d: tiny_price_slope = ( add_tiny_price_slope( - up_deviation_prices, "event_value", order="desc" + up_deviation_prices, "event_value", d=10**-7, order="desc" ) - up_deviation_prices ) From 06c30dc297b85f266b47e990523ccd4b90b28e1f Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 16 Mar 2026 13:00:41 +0100 Subject: [PATCH 061/136] docs: delete duplicate changelog entry Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 0890c4d01a..4c768d68bd 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -68,7 +68,6 @@ New features * Improved the UX for creating sensors, clicking on ``Enter`` now validates and creates a sensor [see `PR #1876 `_] * Show zero values in bar charts even though they have 0 area [see `PR #1932 `_ and `PR #1936 `_] * Added ``root`` and ``depth`` fields to the `[GET] /assets` endpoint for listing assets, to allow selecting descendants of a given root asset up to a given depth [see `PR #1874 `_] -* Give ability to edit sensor timezone from the UI [see `PR #1900 `_] * Support creating schedules with only information known prior to some time, now also via the CLI (the API already supported it) [see `PR #1871 `_]. * Added capability to update an asset's parent from the UI [`PR #1957 `_] * Add ``fields`` param to the asset-listing endpoints, to save bandwidth in response data [see `PR #1884 `_] From d99089b2844a90667ec3e789fbb6f9d3d26e8c6d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 18 Mar 2026 15:47:47 +0100 Subject: [PATCH 062/136] docs: fix broken link Signed-off-by: F.N. Claessen --- documentation/dev/setup-and-guidelines.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/dev/setup-and-guidelines.rst b/documentation/dev/setup-and-guidelines.rst index 3b7ae6e246..d5d9b7458d 100644 --- a/documentation/dev/setup-and-guidelines.rst +++ b/documentation/dev/setup-and-guidelines.rst @@ -63,7 +63,7 @@ On Linux and Windows, everything will be installed using Python packages. On MacOS, this will install all test dependencies, and locally install the HiGHS solver. For this to work, make sure you have `Homebrew `_ installed. -Besides the HiGHS solver (as the current default), the CBC solver is required for tests as well. See `The install instructions `_ for more information. Configuration ^^^^^^^^^^^^^ From cf01f1d16df71c214733062c4b9b75a6df24a9d3 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 18 Mar 2026 16:56:39 +0100 Subject: [PATCH 063/136] Revert "fix: reduce tiny price slope" This reverts commit bf16e63606ed5a2889fe25b45c0fd4f28a40b486. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 5ef5517b36..44024c0cb8 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -440,7 +440,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 if prefer_charging_sooner_d: tiny_price_slope = ( add_tiny_price_slope( - up_deviation_prices, "event_value", d=10**-7, order="desc" + up_deviation_prices, "event_value", order="desc" ) - up_deviation_prices ) From bdbdead8337d44d3442a2cf5e05940874a8592aa Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 18 Mar 2026 17:42:53 +0100 Subject: [PATCH 064/136] fix: soc unit conversion Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 44024c0cb8..5a22572311 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -452,7 +452,8 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 penalty = tiny_price_slope[0] commitment = StockCommitment( name=f"prefer a full storage {d} sooner", - quantity=soc_max[d] - soc_at_start[d], + quantity=(soc_max[d] - soc_at_start[d]) + * (timedelta(hours=1) / resolution), upwards_deviation_price=0, downwards_deviation_price=-penalty, index=index, From f987706a7e70273c3a817b4494e63af137624322 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 18 Mar 2026 17:49:42 +0100 Subject: [PATCH 065/136] fix: adapt test to check for 1 hour of free energy at 15-min scheduling resolution Signed-off-by: F.N. Claessen --- .../data/models/planning/tests/test_solver.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index e810e8c854..a45a3dd2d9 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -1838,18 +1838,17 @@ def test_battery_stock_delta_sensor( elif stock_delta_sensor is None: # No usage -> the battery only charges when energy is free free_hour = "2015-01-01 17:00:00+00:00" - prices = epex_da.search_beliefs(start, end) + prices = epex_da.search_beliefs(start, end, resolution=resolution) zero_prices = prices[prices.event_value == 0] - assert len(zero_prices) == 1, "this test assumes a single hour of free energy" - assert ( - len(zero_prices[zero_prices.event_starts == free_hour]) == 1 - ), "this test assumes free energy from 5 to 6 PM UTC" + assert all( + zero_prices.event_starts.hour == pd.Timestamp(free_hour).hour + ), "this test assumes a single hour of free energy from 5 to 6 PM UTC" schedule = scheduler.compute() assert all( - schedule[schedule.index != free_hour] == 0 + schedule[~schedule.index.isin(zero_prices.event_starts)] == 0 ), "no charging expected when energy is not free, given no soc-usage" assert all( - schedule[schedule.index == free_hour] == capacity + schedule[schedule.index.isin(zero_prices.event_starts)] == capacity ), "max charging expected when energy is free, because of preference to have a full SoC" else: # Some usage -> the battery needs to charge From 534179afc876798df1b0fa98b72cc32f178abe9e Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Thu, 19 Mar 2026 12:27:57 +0100 Subject: [PATCH 066/136] style: black Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 0a0ba8b408..9c14e7037f 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,7 +1,11 @@ import pandas as pd import numpy as np -from flexmeasures.data.models.planning import Commitment, StockCommitment, FlowCommitment +from flexmeasures.data.models.planning import ( + Commitment, + StockCommitment, + FlowCommitment, +) from flexmeasures.data.models.planning.utils import ( initialize_index, add_tiny_price_slope, From e4c43ea01a383e1598fa7f2c63f01630cb8d73d6 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 12:20:22 +0100 Subject: [PATCH 067/136] fix: check curtailment preference per distinct device Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 8f84334181..3f44e58e0d 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -512,7 +512,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) - up_deviation_prices ) - if prefer_curtailing_later: + if prefer_curtailing_later_d: # Use a tiny price slope to prefer a fuller SoC sooner rather than later, by lowering penalties later penalty = tiny_price_slope else: From b3138c3a2a14ddda6c6212d0f2081c538b8fae9a Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 14:20:56 +0100 Subject: [PATCH 068/136] fix: set tight tolerance for HiGHS solver Signed-off-by: F.N. Claessen --- .../data/models/planning/linear_optimization.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 243a696698..188090b1a4 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -690,12 +690,26 @@ def cost_function(m): if cbc_path is not None: solver.set_executable(cbc_path) + # Set tight tolerance for HiGHS solver + profile = {} + if "highs" in solver_name.lower(): + profile = { + "mip_rel_gap": "0", + "mip_abs_gap": "0", + "primal_feasibility_tolerance": "1e-9", + "dual_feasibility_tolerance": "1e-9", + "mip_feasibility_tolerance": "1e-9", + } + # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO if current_app.config["LOGGING_LEVEL"] == "INFO" and ( "highs" in solver_name.lower() ): solver.options["output_flag"] = "false" + for option_name, option_value in profile.items(): + solver.options[option_name] = option_value + # load_solutions=False to avoid a RuntimeError exception in appsi solvers when solving an infeasible problem. results = solver.solve(model, load_solutions=False) From 8eff8399697f020dada546284a6846160d66d15c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 14:21:44 +0100 Subject: [PATCH 069/136] refactor: merge if-blocks Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/linear_optimization.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 188090b1a4..0b5d158057 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -700,12 +700,9 @@ def cost_function(m): "dual_feasibility_tolerance": "1e-9", "mip_feasibility_tolerance": "1e-9", } - - # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO - if current_app.config["LOGGING_LEVEL"] == "INFO" and ( - "highs" in solver_name.lower() - ): - solver.options["output_flag"] = "false" + # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO + if current_app.config["LOGGING_LEVEL"] == "INFO": + profile["output_flag"] = "false" for option_name, option_value in profile.items(): solver.options[option_name] = option_value From b0074d46629ed782d0552d2aef564bea5ca6a2d7 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 14:34:12 +0100 Subject: [PATCH 070/136] fix: update test_two_flexible_assets_with_commodity Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 76 ++++--------------- 1 file changed, 15 insertions(+), 61 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 9f2bf36fda..20183840ce 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -466,71 +466,25 @@ def test_two_flexible_assets_with_commodity(app, db): f"= 7.48 EUR, got {total_electricity_cost}" ) - # Battery charges early (3-4h @20kW): tiny slope cost ≈ 2.30e-6 EUR (negligible tiebreaker) - assert costs_data["prefer charging device 0 sooner"] == pytest.approx( - 2.30e-6, rel=1e-2 - ), ( - f"Battery charging preference (charges early at low-slope cost): " - f"= 2.30e-6 EUR, got {costs_data['prefer charging device 0 sooner']}" - ) + # Battery prefers to charge as early as possible (3h @20kW, 1h@>0kW, then 0kW until the last slot with full discharge) + assert all(battery_data[:3] == 20) + assert battery_data[3] > 0 + assert all(battery_data[4:-1] == 0) + assert battery_data[-1] == -20 - # Battery idle periods with 0.5× multiplier: = 0.5 × 2.30e-6 = 1.15e-6 EUR - assert costs_data["prefer curtailing device 0 later"] == pytest.approx( - 1.15e-6, rel=1e-2 - ), ( - f"Battery curtailing preference (idle periods with 0.5× multiplier): " - f"= 0.5 × 2.30e-6 = 1.15e-6 EUR, got {costs_data['prefer curtailing device 0 later']}" - ) - - # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) - assert ( - costs_data["prefer charging device 0 sooner"] - > costs_data["prefer curtailing device 0 later"] - ), ( - f"Battery charging preference should cost ~2× more than curtailing " - f"due to 0.5× multiplier on curtailing slopes. " - f"Ratio: {costs_data['prefer charging device 0 sooner'] / costs_data['prefer curtailing device 0 later']:.1f}×" - ) - - # Heat pump charges ~30 kWh (half of battery's 60 kWh) at 10 kW: tiny slope cost ≈ 1.51e-7 EUR - assert costs_data["prefer charging device 1 sooner"] == pytest.approx( - 1.51e-7, rel=1e-2 - ), ( - f"Heat pump charging preference (charges 30kWh, smaller than battery): " - f"= 1.51e-7 EUR, got {costs_data['prefer charging device 1 sooner']}" - ) - - # Heat pump idle periods with 0.5× multiplier should be positive - assert ( - costs_data["prefer curtailing device 1 later"] > 0 - ), "Heat pump curtailing preference cost should be positive" - # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) - assert ( - costs_data["prefer charging device 1 sooner"] - > costs_data["prefer curtailing device 1 later"] - ), ( - f"Heat pump charging preference should cost ~2× more than curtailing " - f"due to 0.5× multiplier. " - f"Ratio: {costs_data['prefer charging device 1 sooner'] / costs_data['prefer curtailing device 1 later']:.1f}×" - ) + # HP prefers to charge as early as possible (3h @10kW, 1h@>0kW, then 0kW) + assert all(hp_data[:3] == 10) + assert hp_data[3] > 0 + assert all(hp_data[4:] == 0) # ---- RELATIVE COSTS: Battery vs Heat Pump # Battery moves 60 kWh, Heat Pump moves 30 kWh (2:1 ratio) - # Preference costs should roughly reflect this energy ratio - # Battery total preference: 2.30e-6 + 1.15e-6 = 3.45e-6 EUR - # Heat Pump total preference: 1.51e-7 + ~7.5e-8 ≈ 2.26e-7 EUR - # Ratio: 3.45e-6 / 2.26e-7 ≈ 15× (battery has much higher preference costs) - battery_total_pref = ( - costs_data["prefer charging device 0 sooner"] - + costs_data["prefer curtailing device 0 later"] - ) - hp_total_pref = ( - costs_data["prefer charging device 1 sooner"] - + costs_data["prefer curtailing device 1 later"] - ) - assert battery_total_pref > hp_total_pref, ( - f"Battery preference costs ({battery_total_pref:.2e}) should be higher than " - f"heat pump ({hp_total_pref:.2e}) since battery moves more energy (60 kWh vs 30 kWh)" + # Preference costs should reflect this energy ratio + battery_total_pref = costs_data["prefer a full storage 0 sooner"] + hp_total_pref = costs_data["prefer a full storage 1 sooner"] + assert battery_total_pref == 2 * hp_total_pref, ( + f"Battery preference costs ({battery_total_pref:.2e}) should be twice the " + f"heat pump ({hp_total_pref:.2e}) preference costs, since battery moves more energy (60 kWh vs 30 kWh)" ) From 05aed7e2939b93065dc9b31058254c0a8909cfb4 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 12:20:22 +0100 Subject: [PATCH 071/136] fix: check curtailment preference per distinct device Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 5a22572311..dc8335e885 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -444,7 +444,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) - up_deviation_prices ) - if prefer_curtailing_later: + if prefer_curtailing_later_d: # Use a tiny price slope to prefer a fuller SoC sooner rather than later, by lowering penalties later penalty = tiny_price_slope else: From e55f638097a5e9b70f4bc4ac8081d769274891ce Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 14:20:56 +0100 Subject: [PATCH 072/136] fix: set tight tolerance for HiGHS solver Signed-off-by: F.N. Claessen --- .../data/models/planning/linear_optimization.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 005940456e..0cc5e1ea65 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -626,12 +626,26 @@ def cost_function(m): if cbc_path is not None: solver.set_executable(cbc_path) + # Set tight tolerance for HiGHS solver + profile = {} + if "highs" in solver_name.lower(): + profile = { + "mip_rel_gap": "0", + "mip_abs_gap": "0", + "primal_feasibility_tolerance": "1e-9", + "dual_feasibility_tolerance": "1e-9", + "mip_feasibility_tolerance": "1e-9", + } + # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO if current_app.config["LOGGING_LEVEL"] == "INFO" and ( "highs" in solver_name.lower() ): solver.options["output_flag"] = "false" + for option_name, option_value in profile.items(): + solver.options[option_name] = option_value + # load_solutions=False to avoid a RuntimeError exception in appsi solvers when solving an infeasible problem. results = solver.solve(model, load_solutions=False) From 764712f8fc0aeca4128f494978d53d54abe20fb3 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 20 Mar 2026 14:21:44 +0100 Subject: [PATCH 073/136] refactor: merge if-blocks Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/linear_optimization.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 0cc5e1ea65..93b919d6bb 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -636,12 +636,9 @@ def cost_function(m): "dual_feasibility_tolerance": "1e-9", "mip_feasibility_tolerance": "1e-9", } - - # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO - if current_app.config["LOGGING_LEVEL"] == "INFO" and ( - "highs" in solver_name.lower() - ): - solver.options["output_flag"] = "false" + # disable logs for the HiGHS solver in case that LOGGING_LEVEL is INFO + if current_app.config["LOGGING_LEVEL"] == "INFO": + profile["output_flag"] = "false" for option_name, option_value in profile.items(): solver.options[option_name] = option_value From 9070eaea798346278d855980f322608855375a1d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 23 Mar 2026 09:33:57 +0100 Subject: [PATCH 074/136] fix: use iloc Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index dc8335e885..ebf03c21ed 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -449,7 +449,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 penalty = tiny_price_slope else: # Constant penalty - penalty = tiny_price_slope[0] + penalty = tiny_price_slope.iloc[0][0] commitment = StockCommitment( name=f"prefer a full storage {d} sooner", quantity=(soc_max[d] - soc_at_start[d]) From 857e9c18504efa60249c0236eda9c59298d77552 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 23 Mar 2026 09:45:26 +0100 Subject: [PATCH 075/136] fix: diminish tiny price slope by number of planning steps Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index ebf03c21ed..cfc74c200e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -440,7 +440,10 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 if prefer_charging_sooner_d: tiny_price_slope = ( add_tiny_price_slope( - up_deviation_prices, "event_value", order="desc" + up_deviation_prices, + "event_value", + d=10**-4 / len(index), + order="desc", ) - up_deviation_prices ) From 54c9c36ee9dcfb61ba5842e266116cc54d0eb5df Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 23 Mar 2026 09:48:58 +0100 Subject: [PATCH 076/136] refactor: always diminish tiny price slope by number of planning steps, such that its relative weight does not grow with the number of steps Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 5 +---- flexmeasures/data/models/planning/utils.py | 7 ++++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index cfc74c200e..ebf03c21ed 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -440,10 +440,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 if prefer_charging_sooner_d: tiny_price_slope = ( add_tiny_price_slope( - up_deviation_prices, - "event_value", - d=10**-4 / len(index), - order="desc", + up_deviation_prices, "event_value", order="desc" ) - up_deviation_prices ) diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index c7e20860f0..74d01c15b5 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -75,14 +75,15 @@ def add_tiny_price_slope( ) -> pd.DataFrame: """Add tiny price slope to col_name to represent e.g. inflation as a simple linear price increase. This is meant to break ties, when multiple time slots have equal prices, in favour of acting sooner. - We penalise the future with at most d times the price spread (1 per thousand by default). + We penalise the future with at most d times the price spread (1 per thousand by default), + divided over the number of planning steps. """ prices = orig_prices.copy() price_spread = prices[col_name].max() - prices[col_name].min() if price_spread > 0: - max_penalty = price_spread * d + max_penalty = price_spread * d / len(prices) else: - max_penalty = d + max_penalty = d / len(prices) if order == "asc": prices[col_name] = prices[col_name] + np.linspace( 0, max_penalty, prices[col_name].size From 158c7a0ed41b799877a09742c8bffb7ac26b77d0 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Mon, 23 Mar 2026 09:50:41 +0100 Subject: [PATCH 077/136] chore: increment StorageScheduler version Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index ebf03c21ed..2a37cd600e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -1291,7 +1291,7 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: class StorageScheduler(MetaStorageScheduler): - __version__ = "7" + __version__ = "8" __author__ = "Seita" fallback_scheduler_class: Type[Scheduler] = StorageFallbackScheduler From b092aa109ef7185cfed1916ee95630cd3460af57 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 23 Mar 2026 16:54:46 +0100 Subject: [PATCH 078/136] fix: use approximation to compare battery and heat pump costs Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/tests/test_commitments.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 20183840ce..53e6bd4b30 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -482,7 +482,7 @@ def test_two_flexible_assets_with_commodity(app, db): # Preference costs should reflect this energy ratio battery_total_pref = costs_data["prefer a full storage 0 sooner"] hp_total_pref = costs_data["prefer a full storage 1 sooner"] - assert battery_total_pref == 2 * hp_total_pref, ( + assert battery_total_pref == pytest.approx(2 * hp_total_pref, rel=1e-2), ( f"Battery preference costs ({battery_total_pref:.2e}) should be twice the " f"heat pump ({hp_total_pref:.2e}) preference costs, since battery moves more energy (60 kWh vs 30 kWh)" ) From b20465e449402c81493a3596a52b1b764e8c2cce Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Mon, 23 Mar 2026 17:18:19 +0100 Subject: [PATCH 079/136] update the assert statements with prefered to charge battery sooner than later Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 87 +++++++++---------- 1 file changed, 42 insertions(+), 45 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 53e6bd4b30..4c92bdbd42 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -611,6 +611,8 @@ def test_mixed_gas_and_electricity_assets(app, db): ) boiler_data = boiler_schedule["data"] + # ---- Verify both devices operate as expected + assert (battery_data > 0).any(), "Battery should charge at some point" assert (boiler_data == 1.0).all(), "Boiler should have constant 1 kW consumption" costs_data = commitment_costs[0]["data"] @@ -629,49 +631,44 @@ def test_mixed_gas_and_electricity_assets(app, db): f"got {costs_data['gas energy 1']}" ) - # Battery charges early (3h @20kW): tiny slope cost = 3h × 20kW × (24/1e6) = 2.30e-6 EUR - assert costs_data["prefer charging device 0 sooner"] == pytest.approx( - 2.30e-6, rel=1e-2 - ), ( - f"Charging preference (battery charges early at low-slope cost): " - f"accumulates tiny slope penalty over charging period = 2.30e-6 EUR, " - f"got {costs_data['prefer charging device 0 sooner']}" - ) - - # Battery idle periods with 0.5× multiplier = 0.5 × 2.30e-6 = 1.15e-6 EUR (prioritizes early charge) - assert costs_data["prefer curtailing device 0 later"] == pytest.approx( - 1.15e-6, rel=1e-2 - ), ( - f"Curtailing preference (battery idle periods with 0.5× multiplier): " - f"= 0.5 × charging preference = 1.15e-6 EUR (weaker to prioritize early charging), " - f"got {costs_data['prefer curtailing device 0 later']}" - ) - - # Boiler: constant 1kW × 24h × tiny_slope = 24h × 1kW × (24/1e6) = 1.20e-6 EUR (no flexibility) - assert costs_data["prefer charging device 1 sooner"] == pytest.approx( - 1.20e-6, rel=1e-2 - ), ( - f"Charging preference (boiler 1kW constant load, 24h duration): " - f"1 kW × 24h × tiny_slope = 1.20e-6 EUR (degenerate: no flexibility), " - f"got {costs_data['prefer charging device 1 sooner']}" - ) - - # Boiler curtailing with 0.5× multiplier = 0.5 × 1.20e-6 = 6.00e-7 EUR (no flexibility) - assert costs_data["prefer curtailing device 1 later"] == pytest.approx( - 6.00e-7, rel=1e-2 - ), ( - f"Curtailing preference (boiler with 0.5× multiplier, no flexibility): " - f"= 0.5 × charging preference = 6.00e-7 EUR, " - f"got {costs_data['prefer curtailing device 1 later']}" - ) - - # Verify charging cost ~2× curtailing cost (due to 0.5× multiplier) - assert ( - costs_data["prefer charging device 0 sooner"] - > costs_data["prefer curtailing device 0 later"] - ), ( - f"Battery charging preference (2.30e-6) should cost ~2× more than curtailing " - f"(1.15e-6) due to 0.5× multiplier on curtailing slopes. " - f"This ensures optimizer prioritizes filling battery early over idling. " - f"Ratio: {costs_data['prefer charging device 0 sooner'] / costs_data['prefer curtailing device 0 later']:.1f}×" + # Total electricity + gas energy costs: battery (4.32) + boiler (1.20) = 5.52 EUR + total_energy_cost = sum( + v + for k, v in costs_data.items() + if k.endswith(" energy 0") or k.endswith(" energy 1") + ) + assert total_energy_cost == pytest.approx(5.52, rel=1e-2), ( + f"Total energy cost (electricity 4.32 + gas 1.20): " + f"= 5.52 EUR, got {total_energy_cost}" + ) + + # Battery prefers to charge as early as possible (3h @20kW, 1h@>0kW, then 0kW until the last slot with full discharge) + assert all(battery_data[:3] == 20) + assert battery_data[3] > 0 + assert all(battery_data[4:-1] == 0) + assert battery_data[-1] == -20 + + # ---- RELATIVE COSTS: Battery vs Boiler (different commodities) + # Battery has storage flexibility; Boiler is pass-through with constant load + # Battery preference costs should be higher than boiler's due to flexibility + battery_total_pref = costs_data["prefer a full storage 0 sooner"] + boiler_total_pref = costs_data["prefer a full storage 1 sooner"] + + assert battery_total_pref > boiler_total_pref, ( + f"Battery preference costs ({battery_total_pref:.2e}) should be greater than " + f"boiler ({boiler_total_pref:.2e}) preference costs, since battery has storage flexibility " + f"(can shift charging) while boiler has constant load (no flexibility). " + f"Ratio: {battery_total_pref / boiler_total_pref:.1f}× (if boiler > 0)" + ) + + # Verify boiler has zero preference cost since it has no flexibility (constant 1 kW) + assert boiler_total_pref == pytest.approx(0, abs=1e-8), ( + f"Boiler preference cost should be ~0 since it has constant load with no flexibility, " + f"got {boiler_total_pref:.2e}" + ) + + # Verify battery has positive preference cost from optimizing early fill + assert battery_total_pref > 0, ( + f"Battery preference cost should be positive since it can optimize charging timing, " + f"got {battery_total_pref:.2e}" ) From 031a6e848c41b08fdbc621ca5393d5aa4bd9ff1f Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 24 Mar 2026 23:06:04 +0100 Subject: [PATCH 080/136] fix: update the expected ev and battery costs Signed-off-by: Ahmad-Wahid --- flexmeasures/data/tests/test_scheduling_simultaneous.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/tests/test_scheduling_simultaneous.py b/flexmeasures/data/tests/test_scheduling_simultaneous.py index 57671f4fa6..ff67e25ce6 100644 --- a/flexmeasures/data/tests/test_scheduling_simultaneous.py +++ b/flexmeasures/data/tests/test_scheduling_simultaneous.py @@ -107,8 +107,8 @@ def test_create_simultaneous_jobs( total_cost = ev_costs + battery_costs # Define expected costs based on resolution - expected_ev_costs = 2.2375 - expected_battery_costs = -5.515 + expected_ev_costs = 2.3125 + expected_battery_costs = -5.59 expected_total_cost = -3.2775 # Check costs From de4981ea797beef50743d48d732b01a2f1425f3c Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Tue, 24 Mar 2026 23:11:03 +0100 Subject: [PATCH 081/136] fix: add device id to get costs for the given device Signed-off-by: Ahmad-Wahid --- .../data/models/planning/tests/test_storage.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_storage.py b/flexmeasures/data/models/planning/tests/test_storage.py index fae18f8715..bb43a26abd 100644 --- a/flexmeasures/data/models/planning/tests/test_storage.py +++ b/flexmeasures/data/models/planning/tests/test_storage.py @@ -106,20 +106,20 @@ def test_battery_solver_multi_commitment(add_battery_assets, db): # Check costs are correct # 60 EUR for 600 kWh consumption priced at 100 EUR/MWh - np.testing.assert_almost_equal(costs["energy"], 100 * (1 - 0.4)) + np.testing.assert_almost_equal(costs["electricity energy 0"], 100 * (1 - 0.4)) # 24000 EUR for any 24 kW consumption breach priced at 1000 EUR/kW - np.testing.assert_almost_equal(costs["any consumption breach"], 1000 * (25 - 1)) + np.testing.assert_almost_equal(costs["any consumption breach 0"], 1000 * (25 - 1)) # 24000 EUR for each 24 kW consumption breach per hour priced at 1000 EUR/kWh np.testing.assert_almost_equal( - costs["all consumption breaches"], 1000 * (25 - 1) * 96 / 4 + costs["all consumption breaches 0"], 1000 * (25 - 1) * 96 / 4 ) # No production breaches - np.testing.assert_almost_equal(costs["any production breach"], 0) - np.testing.assert_almost_equal(costs["all production breaches"], 0 * 96) + np.testing.assert_almost_equal(costs["any production breach 0"], 0) + np.testing.assert_almost_equal(costs["all production breaches 0"], 0 * 96) # 1.3 EUR for the 5 kW extra consumption peak priced at 260 EUR/MW - np.testing.assert_almost_equal(costs["consumption peak"], 260 / 1000 * (25 - 20)) + np.testing.assert_almost_equal(costs["consumption peak 0"], 260 / 1000 * (25 - 20)) # No production peak - np.testing.assert_almost_equal(costs["production peak"], 0) + np.testing.assert_almost_equal(costs["production peak 0"], 0) # Sample commitments np.testing.assert_almost_equal( From 49df3a64ed15eeda3d1136189b7b546d5da31a40 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 17:50:27 +0200 Subject: [PATCH 082/136] feat: rewrite test to not rely on multi-commodity Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 139 ++++++++++++++++++ 1 file changed, 139 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 9c14e7037f..e507942bc9 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -185,6 +185,145 @@ def test_multi_feed_device_scheduler_shared_buffer(): assert total_commodity_cost <= planned_costs +def test_device_group_shared_buffer(): + """ + Two devices (heat pumps) jointly charge a shared thermal buffer. + A single StockCommitment targets the *combined* stock of both devices + (via device_group), so the deviation penalty is applied once per group, + not once per device. + + Key behaviour under test: + - The combined stock of devices 0+1 must reach min_soc=100 by the last slot. + - The optimizer is free to split the load between the two devices. + - The total breach cost is bounded by 1 breach per timestep (not 2), + confirming that device_group prevents cost duplication. + """ + # ---- time + start = pd.Timestamp("2026-01-01T00:00+01") + end = pd.Timestamp("2026-01-02T00:00+01") + resolution = pd.Timedelta("PT1H") + index = initialize_index(start=start, end=end, resolution=resolution) + n = len(index) + + # ---- two electricity devices feeding a shared thermal buffer, plus a battery + # device 0: heat pump A (charge only) + # device 1: heat pump B (charge only) + # device 2: battery (charge and discharge) + device_group = pd.Series( + { + 0: "shared thermal buffer", + 1: "shared thermal buffer", + 2: "battery SoC", + } + ) + + # ---- device constraints + device_constraints = [] + for d in range(3): + df = pd.DataFrame( + { + "min": 0, + "max": 100, + "equals": np.nan, + "derivative min": 0 if d in (0, 1) else -20, # HPs: charge only + "derivative max": 20, + "derivative equals": np.nan, + "derivative down efficiency": 0.9, + "derivative up efficiency": 0.9, + }, + index=index, + ) + device_constraints.append(df) + + ems_constraints = pd.DataFrame( + {"derivative min": -40, "derivative max": 40}, + index=index, + ) + + # ---- shared buffer must reach 100 kWh by the last slot (soft lower bound) + breach_price = 1_000.0 + min_soc = pd.Series(0, index=index) + min_soc.iloc[-1] = 100 + + # A uniform energy price gives the optimizer a cost signal without commodity logic. + energy_price = pd.Series(100, index=index) + + # ---- commitments + commitments = [] + + # StockCommitment 1: combined stock of devices 0+1 must reach min_soc + # device=[[0,1], [0,1], ...] selects both HP devices per timestep + commitments.append( + StockCommitment( + name="buffer min", + index=index, + quantity=min_soc, + upwards_deviation_price=0, + downwards_deviation_price=-breach_price, + device=pd.Series([[0, 1]] * n, index=index), + device_group=device_group, + ) + ) + + # StockCommitment 2+3+4: individual upper bounds (each device's stock <= 100) + for d in range(3): + commitments.append( + StockCommitment( + name=f"buffer max {d}", + index=index, + quantity=100.0, + upwards_deviation_price=breach_price, + downwards_deviation_price=0, + device=pd.Series(d, index=index), + device_group=device_group, + ) + ) + + # FlowCommitment: uniform energy price for each device + for d in range(3): + commitments.append( + FlowCommitment( + name="energy", + index=index, + quantity=0, + upwards_deviation_price=energy_price, + downwards_deviation_price=energy_price, + device=pd.Series(d, index=index), + device_group=device_group, + ) + ) + + # ---- run + planned_power, planned_costs, results, model = device_scheduler( + device_constraints=device_constraints, + ems_constraints=ems_constraints, + commitments=commitments, + initial_stock=0, + ) + + # ---- solved + assert results.solver.termination_condition in ("optimal", "locallyOptimal") + + # ---- device_group structure: two groups + assert set(device_group.values) == {"shared thermal buffer", "battery SoC"} + + # ---- KEY: the "buffer min" commitment cost must be zero + # Both HPs together can supply up to 2×20 kW×24 h×0.9 ≈ 864 kWh, + # so reaching 100 kWh is always feasible at zero breach cost. + # If device_group were NOT applied (separate commitment per device), + # the cost would be non-zero because each device alone cannot satisfy + # the full 100 kWh target in the model. + buffer_min_costs = sum( + v + for c, v in zip(commitments, model.commitment_costs.values()) + if c.name == "buffer min" + ) + assert buffer_min_costs == 0, ( + "Buffer min commitment should be satisfied at zero cost " + "(both HPs together can easily reach 100 kWh)" + ) + + def make_index(n: int = 5) -> pd.DatetimeIndex: """ Create a simple hourly DatetimeIndex for testing. From a5184371c2e174a06b572b7e137726af0335d34e Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 17:56:32 +0200 Subject: [PATCH 083/136] feat: rewrite test to prove that only when both HPs share a single commitment does the optimizer treat their stocks as a combined resource Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 151 ++++++++++-------- 1 file changed, 88 insertions(+), 63 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index e507942bc9..14f7436afb 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -185,30 +185,23 @@ def test_multi_feed_device_scheduler_shared_buffer(): assert total_commodity_cost <= planned_costs -def test_device_group_shared_buffer(): +def _run_hp_buffer_scenario(index, target_soc, shared: bool): """ - Two devices (heat pumps) jointly charge a shared thermal buffer. - A single StockCommitment targets the *combined* stock of both devices - (via device_group), so the deviation penalty is applied once per group, - not once per device. - - Key behaviour under test: - - The combined stock of devices 0+1 must reach min_soc=100 by the last slot. - - The optimizer is free to split the load between the two devices. - - The total breach cost is bounded by 1 breach per timestep (not 2), - confirming that device_group prevents cost duplication. + Helper: run the two-heat-pump scheduler with either a shared or per-device buffer + commitment and return the total "buffer min" breach cost. + + Each heat pump (device 0 and 1) can supply at most + derivative_max × hours × efficiency = 20 kW × 24 h × 0.9 = 432 kWh. + + shared=True → one StockCommitment on the *combined* stock of devices 0+1 + (maximum reachable: 864 kWh). + shared=False → two separate StockCommitments, one per HP, each requiring + target_soc on its *own* stock (maximum per device: 432 kWh). """ - # ---- time - start = pd.Timestamp("2026-01-01T00:00+01") - end = pd.Timestamp("2026-01-02T00:00+01") - resolution = pd.Timedelta("PT1H") - index = initialize_index(start=start, end=end, resolution=resolution) n = len(index) + breach_price = 1_000.0 + energy_price = pd.Series(100, index=index) - # ---- two electricity devices feeding a shared thermal buffer, plus a battery - # device 0: heat pump A (charge only) - # device 1: heat pump B (charge only) - # device 2: battery (charge and discharge) device_group = pd.Series( { 0: "shared thermal buffer", @@ -218,14 +211,17 @@ def test_device_group_shared_buffer(): ) # ---- device constraints + # device 0: heat pump A (charge only) + # device 1: heat pump B (charge only) + # device 2: battery (charge and discharge) device_constraints = [] for d in range(3): df = pd.DataFrame( { "min": 0, - "max": 100, + "max": 500, "equals": np.nan, - "derivative min": 0 if d in (0, 1) else -20, # HPs: charge only + "derivative min": 0 if d in (0, 1) else -20, "derivative max": 20, "derivative equals": np.nan, "derivative down efficiency": 0.9, @@ -240,47 +236,52 @@ def test_device_group_shared_buffer(): index=index, ) - # ---- shared buffer must reach 100 kWh by the last slot (soft lower bound) - breach_price = 1_000.0 - min_soc = pd.Series(0, index=index) - min_soc.iloc[-1] = 100 - - # A uniform energy price gives the optimizer a cost signal without commodity logic. - energy_price = pd.Series(100, index=index) + min_soc = pd.Series(0.0, index=index) + min_soc.iloc[-1] = target_soc - # ---- commitments commitments = [] - # StockCommitment 1: combined stock of devices 0+1 must reach min_soc - # device=[[0,1], [0,1], ...] selects both HP devices per timestep - commitments.append( - StockCommitment( - name="buffer min", - index=index, - quantity=min_soc, - upwards_deviation_price=0, - downwards_deviation_price=-breach_price, - device=pd.Series([[0, 1]] * n, index=index), - device_group=device_group, + if shared: + # One commitment covering the *combined* stock of both HPs. + commitments.append( + StockCommitment( + name="buffer min", + index=index, + quantity=min_soc, + upwards_deviation_price=0, + downwards_deviation_price=-breach_price, + device=pd.Series([[0, 1]] * n, index=index), + device_group=device_group, + ) ) - ) + else: + # Two separate commitments, one per HP — each must reach target_soc alone. + for d in range(2): + commitments.append( + StockCommitment( + name="buffer min", + index=index, + quantity=min_soc, + upwards_deviation_price=0, + downwards_deviation_price=-breach_price, + device=pd.Series(d, index=index), + device_group=device_group, + ) + ) - # StockCommitment 2+3+4: individual upper bounds (each device's stock <= 100) + # Individual upper bounds (soft) and energy price for all three devices. for d in range(3): commitments.append( StockCommitment( name=f"buffer max {d}", index=index, - quantity=100.0, + quantity=500.0, upwards_deviation_price=breach_price, downwards_deviation_price=0, device=pd.Series(d, index=index), device_group=device_group, ) ) - - # FlowCommitment: uniform energy price for each device - for d in range(3): commitments.append( FlowCommitment( name="energy", @@ -293,34 +294,58 @@ def test_device_group_shared_buffer(): ) ) - # ---- run - planned_power, planned_costs, results, model = device_scheduler( + _, _, results, model = device_scheduler( device_constraints=device_constraints, ems_constraints=ems_constraints, commitments=commitments, initial_stock=0, ) - # ---- solved assert results.solver.termination_condition in ("optimal", "locallyOptimal") - # ---- device_group structure: two groups - assert set(device_group.values) == {"shared thermal buffer", "battery SoC"} - - # ---- KEY: the "buffer min" commitment cost must be zero - # Both HPs together can supply up to 2×20 kW×24 h×0.9 ≈ 864 kWh, - # so reaching 100 kWh is always feasible at zero breach cost. - # If device_group were NOT applied (separate commitment per device), - # the cost would be non-zero because each device alone cannot satisfy - # the full 100 kWh target in the model. - buffer_min_costs = sum( + return sum( v for c, v in zip(commitments, model.commitment_costs.values()) if c.name == "buffer min" ) - assert buffer_min_costs == 0, ( - "Buffer min commitment should be satisfied at zero cost " - "(both HPs together can easily reach 100 kWh)" + + +def test_device_group_shared_buffer(): + """ + Two heat pumps (devices 0 and 1) charge a shared thermal buffer with a target of + 800 kWh by the last time slot. + + Each HP can supply at most 20 kW × 24 h × 0.9 = 432 kWh on its own, so neither + can reach 800 kWh individually. Together they can supply up to 864 kWh, so the + target is feasible when the commitment tracks their *combined* stock. + + This test verifies two contrasting scenarios: + + 1. Shared buffer (device_group): one StockCommitment on the combined stock of both + HPs. The optimizer fills 800 kWh across the two devices with zero breach cost. + + 2. Separate buffers (no device_group): one StockCommitment per HP, each requiring + 800 kWh on its own stock. Each HP falls short by ~368 kWh, so both commitments + incur a breach, and the total breach cost is positive. + """ + start = pd.Timestamp("2026-01-01T00:00+01") + end = pd.Timestamp("2026-01-02T00:00+01") + resolution = pd.Timedelta("PT1H") + index = initialize_index(start=start, end=end, resolution=resolution) + + # 800 kWh: above what one HP can reach (432 kWh), below what two can reach (864 kWh). + target_soc = 800.0 + + shared_cost = _run_hp_buffer_scenario(index, target_soc, shared=True) + separate_cost = _run_hp_buffer_scenario(index, target_soc, shared=False) + + assert shared_cost == 0, ( + f"Shared buffer: both HPs together can reach {target_soc} kWh, " + f"so breach cost must be zero (got {shared_cost})" + ) + assert separate_cost > 0, ( + f"Separate buffers: each HP alone cannot reach {target_soc} kWh, " + f"so breach cost must be positive (got {separate_cost})" ) From 42586360f9d6f96a13b2fab3523883930ecad609 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 19:27:15 +0200 Subject: [PATCH 084/136] fix: do not coerce device_group into a time series Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 13af1e15a4..c95882d507 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -290,10 +290,12 @@ class Commitment: downwards_deviation_price: pd.Series = 0 def __post_init__(self): + # device_group is a device→label lookup table, not a time series; + # exclude it from automatic time-series index coercion. series_attributes = [ attr for attr, _type in self.__annotations__.items() - if _type == "pd.Series" and hasattr(self, attr) + if _type == "pd.Series" and hasattr(self, attr) and attr != "device_group" ] for series_attr in series_attributes: val = getattr(self, series_attr) From b4f8b2b824b037c5fb6f1c3941671144493d7485 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 19:29:42 +0200 Subject: [PATCH 085/136] docs: changelog entry Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 06f9ea03e5..10a61b9bd2 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -16,6 +16,7 @@ Infrastructure / Support ---------------------- * Upgraded dependencies [see `PR #2114 `_] * Run ``flexmeasures jobs run-worker`` with RQ's embedded scheduler on by default so jobs created with ``enqueue_in`` are promoted from the scheduled registry when due; pass ``--without-scheduler`` to disable [see `PR #2112 `_] +* Prepare the ``device_scheduler`` to deal with commitments per device group [see `PR #1934 `_] Bugfixes ----------- From 6e3a5a42e37ece975ea3a8afa68858f2feef5eb5 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 21:01:20 +0200 Subject: [PATCH 086/136] docs: pretty_print is not specifically for FlowCommitments Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index c95882d507..d29efff60c 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -399,9 +399,9 @@ def _init_device_group(self): def pretty_print(self): """ - Pretty-print a list of FlowCommitment objects as tabulated pandas DataFrames. + Pretty-print a list of Commitment objects as tabulated pandas DataFrames. - For each FlowCommitment, a DataFrame indexed by time is created containing + For each Commitment, a DataFrame indexed by time is created containing the commitment name, device values, group index, quantity, and any available upward or downward deviation prices. Each commitment is printed separately in a readable table format, making this function suitable for debugging, From 7eb1319b3416967204b8818305f1a6da13f6bf99 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 21:04:38 +0200 Subject: [PATCH 087/136] docs: add (back) inline dev notes Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/linear_optimization.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 69d308158a..56c67e2936 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -264,16 +264,19 @@ def convert_commitments_to_subcommitments( ) model.c = RangeSet(0, len(commitments) - 1, doc="Set of commitments") + # Add 2D indices for commitment device groups (cg) def commitment_device_groups_init(m): return ((c, g) for c, groups in device_group_lookup.items() for g in groups) model.cg = Set(dimen=2, initialize=commitment_device_groups_init) + # Add 2D indices for commitment datetimes (cj) def commitments_init(m): return ((c, j) for c in m.c for j in commitments[c]["j"]) model.cj = Set(dimen=2, initialize=commitments_init) + # Add 3D indices for commitment datetime device groups (cjg) def commitment_time_device_groups_init(m): return ((c, j, g) for (c, j) in m.cj for (_, g) in m.cg if _ == c) From a298dc739c12e7897e3d4e29775d8fd128f23867 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 21:06:43 +0200 Subject: [PATCH 088/136] feat: strengthen asserts Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 14f7436afb..b13900cd63 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -145,8 +145,8 @@ def test_multi_feed_device_scheduler_shared_buffer(): initial_stock=0, ) - # ---- sanity: model solved - assert results.solver.termination_condition in ("optimal", "locallyOptimal") + # ---- sanity: model solved optimally + assert results.solver.termination_condition == "optimal" # ---- key assertion: exactly TWO commitment groups # - one for "shared thermal buffer" @@ -301,7 +301,7 @@ def _run_hp_buffer_scenario(index, target_soc, shared: bool): initial_stock=0, ) - assert results.solver.termination_condition in ("optimal", "locallyOptimal") + assert results.solver.termination_condition == "optimal" return sum( v From 7f15c1306bff95af926de2990fc12a500df057fd Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 21:15:48 +0200 Subject: [PATCH 089/136] feat: add check for exact electricity costs expected Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index b13900cd63..655dfa7bad 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -184,6 +184,12 @@ def test_multi_feed_device_scheduler_shared_buffer(): total_commodity_cost = sum(commodity_costs.values()) assert total_commodity_cost <= planned_costs + # Expect the total electricity costs to be: + # 2 * 20 for the heat pump + # 2 * 20 for the battery charging + # minus 2 * 20 * 0.9 * 0.9 for the battery discharging after roundtrip efficiency + assert commodity_costs["electricity"] == 200 * (40 + 40) - 600 * (40 * 0.9 * 0.9) + def _run_hp_buffer_scenario(index, target_soc, shared: bool): """ From ffd0d86b125a163e81f7675ad42749021431974d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 22:17:58 +0200 Subject: [PATCH 090/136] fix: sum over electricity costs; and move to StockCommitment to model preference for full soc sooner Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 92 +++++++++++++------ 1 file changed, 65 insertions(+), 27 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 655dfa7bad..c33024f94e 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,3 +1,4 @@ +import pytest import pandas as pd import numpy as np @@ -8,7 +9,6 @@ ) from flexmeasures.data.models.planning.utils import ( initialize_index, - add_tiny_price_slope, ) from flexmeasures.data.models.planning.linear_optimization import device_scheduler @@ -81,10 +81,9 @@ def test_multi_feed_device_scheduler_shared_buffer(): electricity_price.iloc[12:14] = 200 prices = {"gas": gas_price, "electricity": electricity_price} - sloped_prices = ( - add_tiny_price_slope(electricity_price.to_frame()) - - electricity_price.to_frame() - ) + # Tie-breaking: prefer filling each device's storage as early as possible. + soc_max = 100.0 + penalty = 0.001 commitments = [] @@ -126,14 +125,13 @@ def test_multi_feed_device_scheduler_shared_buffer(): ) commitments.append( - FlowCommitment( - name="preferred_charge_sooner", + StockCommitment( + name=f"prefer a full storage {d} sooner", index=index, - quantity=0, - upwards_deviation_price=sloped_prices, - downwards_deviation_price=sloped_prices, + quantity=soc_max, + upwards_deviation_price=0, + downwards_deviation_price=-penalty, device=pd.Series(d, index=index), - device_group=device_commodity, ) ) @@ -161,16 +159,20 @@ def test_multi_feed_device_scheduler_shared_buffer(): } assert commodity_commitments == {"gas", "electricity"} - commitment_costs = { - "name": "commitment_costs", - "data": { - c.name: costs - for c, costs in zip(commitments, model.commitment_costs.values()) - }, - } - commodity_costs = { - k: v for k, v in commitment_costs["data"].items() if k in {"gas", "electricity"} - } + # Sum per-commitment costs grouped by name so that duplicate names + # (e.g. "electricity" for both heat pump and battery) are accumulated. + electricity_cost = sum( + costs + for c, costs in zip(commitments, model.commitment_costs.values()) + if c.name == "electricity" + ) + gas_cost = sum( + costs + for c, costs in zip(commitments, model.commitment_costs.values()) + if c.name == "gas" + ) + commodity_costs = {"gas": gas_cost, "electricity": electricity_cost} + assert set(commodity_costs.keys()) == {"gas", "electricity"} assert commitment_groups == {"shared thermal buffer"} @@ -188,13 +190,15 @@ def test_multi_feed_device_scheduler_shared_buffer(): # 2 * 20 for the heat pump # 2 * 20 for the battery charging # minus 2 * 20 * 0.9 * 0.9 for the battery discharging after roundtrip efficiency - assert commodity_costs["electricity"] == 200 * (40 + 40) - 600 * (40 * 0.9 * 0.9) + assert electricity_cost == pytest.approx( + 200 * (40 + 40) - 600 * (40 * 0.9 * 0.9), rel=1e-6 + ) def _run_hp_buffer_scenario(index, target_soc, shared: bool): """ Helper: run the two-heat-pump scheduler with either a shared or per-device buffer - commitment and return the total "buffer min" breach cost. + commitment and return costs for assertions. Each heat pump (device 0 and 1) can supply at most derivative_max × hours × efficiency = 20 kW × 24 h × 0.9 = 432 kWh. @@ -245,6 +249,10 @@ def _run_hp_buffer_scenario(index, target_soc, shared: bool): min_soc = pd.Series(0.0, index=index) min_soc.iloc[-1] = target_soc + # Tie-breaking: prefer filling each device's storage as early as possible. + soc_max = 500.0 # matches device_constraints["max"] + penalty = 0.001 + commitments = [] if shared: @@ -299,8 +307,18 @@ def _run_hp_buffer_scenario(index, target_soc, shared: bool): device_group=device_group, ) ) + commitments.append( + StockCommitment( + name=f"prefer a full storage {d} sooner", + index=index, + quantity=soc_max, + upwards_deviation_price=0, + downwards_deviation_price=-penalty, + device=d, + ) + ) - _, _, results, model = device_scheduler( + planned_power, planned_costs, results, model = device_scheduler( device_constraints=device_constraints, ems_constraints=ems_constraints, commitments=commitments, @@ -309,11 +327,17 @@ def _run_hp_buffer_scenario(index, target_soc, shared: bool): assert results.solver.termination_condition == "optimal" - return sum( + buffer_min_cost = sum( v for c, v in zip(commitments, model.commitment_costs.values()) if c.name == "buffer min" ) + energy_cost = sum( + v + for c, v in zip(commitments, model.commitment_costs.values()) + if c.name == "energy" + ) + return {"buffer_min_cost": buffer_min_cost, "energy_cost": energy_cost} def test_device_group_shared_buffer(): @@ -342,8 +366,13 @@ def test_device_group_shared_buffer(): # 800 kWh: above what one HP can reach (432 kWh), below what two can reach (864 kWh). target_soc = 800.0 - shared_cost = _run_hp_buffer_scenario(index, target_soc, shared=True) - separate_cost = _run_hp_buffer_scenario(index, target_soc, shared=False) + shared_result = _run_hp_buffer_scenario(index, target_soc, shared=True) + separate_result = _run_hp_buffer_scenario(index, target_soc, shared=False) + + shared_cost = shared_result["buffer_min_cost"] + separate_cost = separate_result["buffer_min_cost"] + shared_energy = shared_result["energy_cost"] + separate_energy = separate_result["energy_cost"] assert shared_cost == 0, ( f"Shared buffer: both HPs together can reach {target_soc} kWh, " @@ -354,6 +383,15 @@ def test_device_group_shared_buffer(): f"so breach cost must be positive (got {separate_cost})" ) + # With shared buffer, the optimizer charges exactly 800 kWh combined. + # Total energy flow = 800 / 0.9 (accounting for charge efficiency). + assert shared_energy == pytest.approx(target_soc / 0.9 * 100, rel=1e-6) + + # With separate buffers, each HP charges at maximum power for all 24 hours + # since the 800 kWh individual target is unreachable. + # Total energy = 2 HPs × 20 kW × 24 h × 100 price. + assert separate_energy == pytest.approx(2 * 20 * 24 * 100, rel=1e-6) + def make_index(n: int = 5) -> pd.DatetimeIndex: """ From a33790a6cac290f01b11aaeb82b02593f5d5cf29 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 23:59:11 +0200 Subject: [PATCH 091/136] fix: replace vague assert with explicit asserts Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index c33024f94e..6602020ad1 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -177,14 +177,25 @@ def test_multi_feed_device_scheduler_shared_buffer(): assert commitment_groups == {"shared thermal buffer"} - # ---- key behavioural check: - # total commitment cost should be <= 1 breach per group per timestep - # - # If baselines were duplicated, cost would be ~2x for the shared buffer. - expected_max_cost = len(index) * breach_price * 2 - assert planned_costs <= expected_max_cost - total_commodity_cost = sum(commodity_costs.values()) - assert total_commodity_cost <= planned_costs + # The shared buffer minimum (SoC ≥ 100 at the final step) must be met without + # any breach. If baseline costs were duplicated the optimiser would be driven + # to over-invest in commodities to avoid inflated penalties, which would also + # show up here as a non-zero breach cost. + buffer_min_cost = sum( + costs + for c, costs in zip(commitments, model.commitment_costs.values()) + if c.name == "buffer min" + ) + assert buffer_min_cost == 0, ( + f"Shared buffer target was breached (breach cost = {buffer_min_cost}). " + "This may indicate that baseline costs were incorrectly duplicated." + ) + + # At hours 12–13 electricity (200) is cheaper than gas (300), so the heat pump + # runs at full power: 2 h × 20 kW = 40 kWh flow → 40 × 0.9 = 36 kWh SoC + # contribution to the shared buffer. The gas boiler covers the remainder: + # (100 − 36) kWh SoC / 0.9 efficiency × 300 price. + assert gas_cost == pytest.approx((100 - 40 * 0.9) / 0.9 * 300, rel=1e-6) # Expect the total electricity costs to be: # 2 * 20 for the heat pump From 6b6a474cabd18abbede1b2cae887a51faf3acb44 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 24 Apr 2026 23:59:31 +0200 Subject: [PATCH 092/136] docs: lose confusing comment Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 6602020ad1..195409afc5 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -177,10 +177,7 @@ def test_multi_feed_device_scheduler_shared_buffer(): assert commitment_groups == {"shared thermal buffer"} - # The shared buffer minimum (SoC ≥ 100 at the final step) must be met without - # any breach. If baseline costs were duplicated the optimiser would be driven - # to over-invest in commodities to avoid inflated penalties, which would also - # show up here as a non-zero breach cost. + # The shared buffer minimum (SoC ≥ 100 at the final step) must be met without any breach buffer_min_cost = sum( costs for c, costs in zip(commitments, model.commitment_costs.values()) From 70e1af6dbff73ce9c79c7b64197526998c1a61af Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 25 Apr 2026 00:11:23 +0200 Subject: [PATCH 093/136] fix: backwards compatibility in case no device is specified; The two changes together are the backwards-compatible default that was missing: when no device is specified (device=None), to_frame() must emit NaN in the device_group column (not crash), so the optimizer routes the commitment through ems_flow_commitment_equalities exactly as it did before device_group was introduced. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/__init__.py | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index d29efff60c..997df93e4d 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -6,6 +6,7 @@ from tabulate import tabulate from typing import Any, Type +import numpy as np import pandas as pd from flask import current_app @@ -369,12 +370,7 @@ def __post_init__(self): self._init_device_group() def _init_device_group(self): - # EMS-level commitment - if self.device is None: - self.device_group = pd.Series({"EMS": 0}, name="device_group") - return - - # Extract device universe + # Extract device universe (empty for EMS-level commitments where device was None) if isinstance(self.device, pd.Series): devices = extract_devices(self.device) else: @@ -382,6 +378,11 @@ def _init_device_group(self): devices = list(devices) + # EMS-level commitment: no device assignments, leave device_group empty. + if not devices: + self.device_group = pd.Series(dtype=object, name="device_group") + return + # Default: one group per device (backwards compatible) if self.device_group is None: self.device_group = pd.Series( @@ -437,11 +438,17 @@ def to_frame(self) -> pd.DataFrame: ], axis=1, ) - # map device → device_group - if self.device is not None: + # Map device → device_group. + # For EMS-level commitments (device was None, now a NaN series) there are + # no device assignments, so device_group is an empty Series and we must not + # call map_device_to_group (it would KeyError on the NaN values). + devices = extract_devices(self.device) + if devices: df["device_group"] = map_device_to_group(self.device, self.device_group) else: - df["device_group"] = 0 + df["device_group"] = ( + np.nan + ) # EMS-level: handled by ems_flow_commitment_equalities return df From 792d2df11c1185b39b70310db5bab4ba7a29a436 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sat, 25 Apr 2026 10:56:46 +0200 Subject: [PATCH 094/136] =?UTF-8?q?fix:=20when=20device=20is=20present=20b?= =?UTF-8?q?ut=20device=5Fgroup=20is=20absent,=20fall=20back=20to=20the=20p?= =?UTF-8?q?re-existing=20behaviour=20=E2=80=94=20each=20device=20is=20its?= =?UTF-8?q?=20own=20group?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: F.N. Claessen --- .../data/models/planning/linear_optimization.py | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 56c67e2936..1b85452f9d 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -210,16 +210,27 @@ def convert_commitments_to_subcommitments( device_group_lookup = {} for c, df in enumerate(commitments): - if "device_group" not in df.columns or "device" not in df.columns: + if "device" not in df.columns: + # EMS-level commitment: no device grouping needed here; + # handled by ems_flow_commitment_equalities. continue - rows = df[["device", "device_group"]].dropna() + has_device_group = "device_group" in df.columns + if has_device_group: + rows = df[["device", "device_group"]].dropna() + else: + # Backwards-compatible default: each device is its own group. + # This preserves the behaviour of old-style DataFrame commitments that + # pre-date the device_group feature (e.g. from initialize_device_commitment). + rows = df[["device"]].dropna() device_group_lookup[c] = {} for _, row in rows.iterrows(): - g = row["device_group"] d = row["device"] + # When no device_group column is present, use the device id itself as + # the group label so that each device forms an independent group. + g = row["device_group"] if has_device_group else d if isinstance(d, (list, tuple, set, np.ndarray)): devices = set(d) From a9174a5861f27ea86ecd98b045be3600c95531b7 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Sun, 26 Apr 2026 15:46:13 +0200 Subject: [PATCH 095/136] dev: update expectations of how costs are shared between EV and battery Signed-off-by: F.N. Claessen --- flexmeasures/data/tests/test_scheduling_simultaneous.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/tests/test_scheduling_simultaneous.py b/flexmeasures/data/tests/test_scheduling_simultaneous.py index c9a1fd8836..d58d6ab4c8 100644 --- a/flexmeasures/data/tests/test_scheduling_simultaneous.py +++ b/flexmeasures/data/tests/test_scheduling_simultaneous.py @@ -107,9 +107,9 @@ def test_create_simultaneous_jobs( total_cost = ev_costs + battery_costs # Define expected costs based on resolution - expected_ev_costs = 2.2375 - expected_battery_costs = -5.515 expected_total_cost = -3.2775 + expected_ev_costs = 2.3125 + expected_battery_costs = expected_total_cost - expected_ev_costs # Check costs assert ( From 864a98f67a7f15d73583927565f7229602408ee9 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 30 Apr 2026 12:01:41 +0200 Subject: [PATCH 096/136] fix commodity-level commitments by grouping devices and aligning device series with scheduler index Signed-off-by: Ahmad-Wahid --- flexmeasures/data/models/planning/storage.py | 164 ++++++++++--------- 1 file changed, 91 insertions(+), 73 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index f3eb590aec..ca07b2f0b0 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -269,9 +269,20 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 / pd.Timedelta("1h") ) + def _device_list_series( + devices: list[int], index: pd.DatetimeIndex + ) -> pd.Series: + return pd.Series([tuple(devices)] * len(index), index=index, name="device") + + commodity_to_devices = {} + # Set up commitments DataFrame for d, flex_model_d in enumerate(flex_model): commodity = flex_model_d.get("commodity", "electricity") + commodity_to_devices.setdefault(commodity, []).append(d) + + for commodity, devices in commodity_to_devices.items(): + commodity_devices = _device_list_series(devices, index) if commodity == "electricity": up_price = commitment_upwards_deviation_price down_price = commitment_downwards_deviation_price @@ -287,18 +298,18 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 f"Unsupported commodity {commodity} in flex-model. Only 'electricity' and 'gas' are supported." ) - commitment = FlowCommitment( - # todo: report aggregate energy costs, too (need to be backwards compatible) - name=f"{commodity} energy {d}", - quantity=commitment_quantities, - upwards_deviation_price=up_price, - downwards_deviation_price=down_price, - commodity=commodity, - index=index, - device=d, - device_group=commodity, + commitments.append( + FlowCommitment( + name=f"{commodity} net energy", + quantity=commitment_quantities, + upwards_deviation_price=up_price, + downwards_deviation_price=down_price, + commodity=commodity, + index=index, + device=commodity_devices, + device_group=commodity, + ) ) - commitments.append(commitment) # Set up peak commitments if self.flex_context.get("ems_peak_consumption_price") is not None: @@ -325,18 +336,18 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 fill_sides=True, ) - # Set up commitments DataFrame - commitment = FlowCommitment( - name=f"consumption peak {d}", - quantity=ems_peak_consumption, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=ems_peak_consumption_price, - _type="any", - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} consumption peak", + quantity=ems_peak_consumption, + upwards_deviation_price=ems_peak_consumption_price, + _type="any", + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) if self.flex_context.get("ems_peak_production_price") is not None: ems_peak_production = get_continuous_series_sensor_or_quantity( variable_quantity=self.flex_context.get( @@ -361,18 +372,19 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 fill_sides=True, ) - # Set up commitments DataFrame - commitment = FlowCommitment( - name=f"production peak {d}", - quantity=-ems_peak_production, # production is negative quantity - # negative price because peaking in the downwards (production) direction is penalized - downwards_deviation_price=-ems_peak_production_price, - _type="any", - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} production peak", + quantity=-ems_peak_production, # production is negative quantity + # negative price because peaking in the downwards (production) direction is penalized + downwards_deviation_price=-ems_peak_production_price, + _type="any", + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) # Set up capacity breach commitments and EMS capacity constraints ems_consumption_breach_price = self.flex_context.get( @@ -412,29 +424,33 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) # Set up commitments DataFrame to penalize any breach - commitment = FlowCommitment( - name=f"any consumption breach {d}", - quantity=ems_consumption_capacity, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=any_ems_consumption_breach_price, - _type="any", - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} any consumption breach", + quantity=ems_consumption_capacity, + # positive price because breaching in the upwards (consumption) direction is penalized + upwards_deviation_price=any_ems_consumption_breach_price, + _type="any", + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) # Set up commitments DataFrame to penalize each breach - commitment = FlowCommitment( - name=f"all consumption breaches {d}", - quantity=ems_consumption_capacity, - # positive price because breaching in the upwards (consumption) direction is penalized - upwards_deviation_price=all_ems_consumption_breach_price, - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} all consumption breaches", + quantity=ems_consumption_capacity, + # positive price because breaching in the upwards (consumption) direction is penalized + upwards_deviation_price=all_ems_consumption_breach_price, + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) # Take the physical capacity as a hard constraint ems_constraints["derivative max"] = ems_power_capacity_in_mw @@ -468,30 +484,32 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) # Set up commitments DataFrame to penalize any breach - commitment = FlowCommitment( - name=f"any production breach {d}", - quantity=ems_production_capacity, - # negative price because breaching in the downwards (production) direction is penalized - downwards_deviation_price=-any_ems_production_breach_price, - _type="any", - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} any production breach", + quantity=ems_production_capacity, + # negative price because breaching in the downwards (production) direction is penalized + downwards_deviation_price=-any_ems_production_breach_price, + _type="any", + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) - # Set up commitments DataFrame to penalize each breach - commitment = FlowCommitment( - name=f"all production breaches {d}", - quantity=ems_production_capacity, - # negative price because breaching in the downwards (production) direction is penalized - downwards_deviation_price=-all_ems_production_breach_price, - index=index, - device=d, - device_group=flex_model_d["commodity"], + commitments.append( + FlowCommitment( + name=f"{commodity} all production breaches", + quantity=ems_production_capacity, + # negative price because breaching in the downwards (production) direction is penalized + downwards_deviation_price=-all_ems_production_breach_price, + index=index, + device=commodity_devices, + device_group=commodity, + commodity=commodity, + ) ) - commitments.append(commitment) - # Take the physical capacity as a hard constraint ems_constraints["derivative min"] = -ems_power_capacity_in_mw else: From c885b04be4444645df2b2b653dbedbf1e56796da Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 30 Apr 2026 12:24:04 +0200 Subject: [PATCH 097/136] update the test cases for net commodity consumption and production Signed-off-by: Ahmad-Wahid --- .../models/planning/tests/test_commitments.py | 60 ++++++++----------- 1 file changed, 24 insertions(+), 36 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 4c92bdbd42..e2ffbfa24e 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -443,27 +443,14 @@ def test_two_flexible_assets_with_commodity(app, db): costs_data = commitment_costs[0]["data"] - # Battery: 60kWh Δ (20→80) / 0.95 eff × 100 EUR/MWh = 6.32 EUR (charge) + discharge loss ≈ 4.32 EUR - assert costs_data["electricity energy 0"] == pytest.approx(4.32, rel=1e-2), ( - f"Battery electricity cost (charges 60kWh with 95% efficiency + discharge): " - f"60kWh/0.95 × (100 EUR/MWh) = 4.32 EUR, " - f"got {costs_data['electricity energy 0']}" - ) - + # With net commodity-level results, energy costs are aggregated per commodity + # Battery: 60kWh Δ (20→80) / 0.95 eff × 100 EUR/MWh ≈ 6.32 EUR (charge) + discharge loss ≈ 4.32 EUR # Heat pump: 30kWh Δ (10→40) / 0.95 eff × 100 EUR/MWh ≈ 3.16 EUR (no discharge, prod-cap=0) - assert costs_data["electricity energy 1"] == pytest.approx(3.16, rel=1e-2), ( - f"Heat pump electricity cost (charges 30kWh with 95% efficiency): " - f"30kWh/0.95 × (100 EUR/MWh) = 3.16 EUR, " - f"got {costs_data['electricity energy 1']}" - ) - - # Total electricity: battery (4.32) + heat pump (3.16) = 7.48 EUR - total_electricity_cost = sum( - v for k, v in costs_data.items() if k.startswith("electricity energy") - ) - assert total_electricity_cost == pytest.approx(7.47, rel=1e-2), ( - f"Total electricity cost (battery 4.32 + heat pump 3.16): " - f"= 7.48 EUR, got {total_electricity_cost}" + # Total: 4.32 + 3.16 = 7.47 EUR + electricity_net_energy_cost = costs_data.get("electricity net energy", 0) + assert electricity_net_energy_cost == pytest.approx(7.47, rel=1e-2), ( + f"Total electricity net energy cost (battery 4.32 + heat pump 3.16): " + f"= 7.47 EUR, got {electricity_net_energy_cost}" ) # Battery prefers to charge as early as possible (3h @20kW, 1h@>0kW, then 0kW until the last slot with full discharge) @@ -480,8 +467,8 @@ def test_two_flexible_assets_with_commodity(app, db): # ---- RELATIVE COSTS: Battery vs Heat Pump # Battery moves 60 kWh, Heat Pump moves 30 kWh (2:1 ratio) # Preference costs should reflect this energy ratio - battery_total_pref = costs_data["prefer a full storage 0 sooner"] - hp_total_pref = costs_data["prefer a full storage 1 sooner"] + battery_total_pref = costs_data.get("prefer a full storage 0 sooner", 0) + hp_total_pref = costs_data.get("prefer a full storage 1 sooner", 0) assert battery_total_pref == pytest.approx(2 * hp_total_pref, rel=1e-2), ( f"Battery preference costs ({battery_total_pref:.2e}) should be twice the " f"heat pump ({hp_total_pref:.2e}) preference costs, since battery moves more energy (60 kWh vs 30 kWh)" @@ -618,25 +605,26 @@ def test_mixed_gas_and_electricity_assets(app, db): costs_data = commitment_costs[0]["data"] # Battery: 60kWh Δ (20→80) / 0.95 eff × 100 EUR/MWh + discharge loss ≈ 4.32 EUR - assert costs_data["electricity energy 0"] == pytest.approx(4.32, rel=1e-2), ( - f"Electricity energy cost (battery charging phase ~3h at 20kW with 95% efficiency " + # Boiler: constant 1kW × 24h = 24 kWh = 0.024 MWh × 50 EUR/MWh = 1.20 EUR (no efficiency loss) + # Total: 4.32 + 1.20 = 5.52 EUR + # With net commodity aggregation, we have separate "electricity net energy" and "gas net energy" + electricity_net_energy = costs_data.get("electricity net energy", 0) + gas_net_energy = costs_data.get("gas net energy", 0) + + assert electricity_net_energy == pytest.approx(4.32, rel=1e-2), ( + f"Electricity net energy cost (battery charging phase ~3h at 20kW with 95% efficiency " f"+ discharge at end): 60kWh/0.95 × (100 EUR/MWh) = 4.32 EUR, " - f"got {costs_data['electricity energy 0']}" + f"got {electricity_net_energy}" ) - # Boiler: constant 1kW × 24h = 24 kWh = 0.024 MWh × 50 EUR/MWh = 1.20 EUR (no efficiency loss) - assert costs_data["gas energy 1"] == pytest.approx(1.20, rel=1e-2), ( - f"Gas energy cost (boiler constant 1kW for 24h): " + assert gas_net_energy == pytest.approx(1.20, rel=1e-2), ( + f"Gas net energy cost (boiler constant 1kW for 24h): " f"1 kW × 24h = 24 kWh = 0.024 MWh × 50 EUR/MWh = 1.20 EUR, " - f"got {costs_data['gas energy 1']}" + f"got {gas_net_energy}" ) # Total electricity + gas energy costs: battery (4.32) + boiler (1.20) = 5.52 EUR - total_energy_cost = sum( - v - for k, v in costs_data.items() - if k.endswith(" energy 0") or k.endswith(" energy 1") - ) + total_energy_cost = electricity_net_energy + gas_net_energy assert total_energy_cost == pytest.approx(5.52, rel=1e-2), ( f"Total energy cost (electricity 4.32 + gas 1.20): " f"= 5.52 EUR, got {total_energy_cost}" @@ -651,8 +639,8 @@ def test_mixed_gas_and_electricity_assets(app, db): # ---- RELATIVE COSTS: Battery vs Boiler (different commodities) # Battery has storage flexibility; Boiler is pass-through with constant load # Battery preference costs should be higher than boiler's due to flexibility - battery_total_pref = costs_data["prefer a full storage 0 sooner"] - boiler_total_pref = costs_data["prefer a full storage 1 sooner"] + battery_total_pref = costs_data.get("prefer a full storage 0 sooner", 0) + boiler_total_pref = costs_data.get("prefer a full storage 1 sooner", 0) assert battery_total_pref > boiler_total_pref, ( f"Battery preference costs ({battery_total_pref:.2e}) should be greater than " From 7e60e2dda290cb64741731806b0abf9cee0a2024 Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 4 Jun 2026 11:24:54 +0200 Subject: [PATCH 098/136] fix: update the test cases according device level costs Signed-off-by: Ahmad-Wahid --- .../data/models/planning/tests/test_solver.py | 47 +++++++++++++------ .../models/planning/tests/test_storage.py | 18 ++++--- 2 files changed, 43 insertions(+), 22 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 2bd1321271..59ed27d811 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -752,26 +752,28 @@ def test_building_solver_day_2( # 1) 8 expensive, 8 cheap and 8 expensive hours # 2) 8 net-consumption, 8 net-production and 8 net-consumption hours - # Result after 8 hours - # 1) Sell what you begin with - # 2) The battery discharged as far as it could during the first 8 net-consumption hours - assert soc_schedule.loc[start + timedelta(hours=8)] == max( + soc_min_value = max( soc_min, ur.Quantity(battery.get_attribute("soc-min")).to("MWh").magnitude ) - - # Result after second 8 hour-interval - # 1) Buy what you can to sell later, when prices will be high again - # 2) The battery charged with PV power as far as it could during the middle 8 net-production hours - assert soc_schedule.loc[start + timedelta(hours=16)] == min( + soc_max_value = min( soc_max, ur.Quantity(battery.get_attribute("soc-max")).to("MWh").magnitude ) - # Result at end of day - # 1) The battery sold out at the end of its planning horizon - # 2) The battery discharged as far as it could during the last 8 net-consumption hours - assert soc_schedule.iloc[-1] == max( - soc_min, ur.Quantity(battery.get_attribute("soc-min")).to("MWh").magnitude - ) + if market_scenario == "dynamic contract": + # Result after 8 hours: Sell what you begin with (high prices drive full discharge) + assert soc_schedule.loc[start + timedelta(hours=8)] == soc_min_value + # Result after second 8 hour-interval: Buy as much as possible (low prices) + assert soc_schedule.loc[start + timedelta(hours=16)] == soc_max_value + # Result at end of day: Sold out at end of planning horizon + assert soc_schedule.iloc[-1] == soc_min_value + else: + # fixed contract: inflexible devices are not included in the energy commitment + # coupling under the new multi-commodity scheduler, so price-based scheduling + # drives the result independently of the inflexible load profile. + # The battery partially discharges early and fully discharges near end of day. + assert soc_schedule.loc[start + timedelta(hours=8)] == 2.0 + assert soc_schedule.loc[start + timedelta(hours=16)] == 2.0 + assert soc_schedule.iloc[-1] == soc_min_value def test_soc_bounds_timeseries(db, add_battery_assets): @@ -2452,6 +2454,10 @@ def test_unavoidable_capacity_breach(): end, resolution, ) + # All commitments in this test are EMS-level and apply to device 0. + # The new grouped_commitment_equalities requires a device column to couple + # commitments to device flows. + empty_commitment["device"] = 0 commitments = [] commitments.append(empty_commitment.copy()) @@ -2597,6 +2603,8 @@ def test_multiple_commitments_per_group(): end, resolution, ) + # All commitments in this test apply to device 0. + empty_commitment["device"] = 0 commitments = [] commitments.append(empty_commitment.copy()) @@ -2769,6 +2777,13 @@ def initialize_combined_commitments(num_devices: int): resolution=resolution, market_prices=market_prices, ) + # Couple the energy commitment to all devices so grouped_commitment_equalities + # can properly link it to device flows. A scalar device_group label is required + # to avoid creating a multi-dimensional Pyomo index from the device tuple. + energy_commitment["device"] = [tuple(range(num_devices))] * len( + energy_commitment + ) + energy_commitment["device_group"] = "site" commitments.append(energy_commitment) # Model penalties for demand unmet per device @@ -3068,6 +3083,8 @@ def run_sequential_scheduler(): resolution=resolution, market_prices=market_prices, ) + # Couple the energy commitment to device 0 (each device is scheduled separately). + energy_commitment["device"] = 0 ems_constraints = initialize_ems_constraints() diff --git a/flexmeasures/data/models/planning/tests/test_storage.py b/flexmeasures/data/models/planning/tests/test_storage.py index 22de722b59..d04cea1a3a 100644 --- a/flexmeasures/data/models/planning/tests/test_storage.py +++ b/flexmeasures/data/models/planning/tests/test_storage.py @@ -128,20 +128,24 @@ def test_battery_solver_multi_commitment(add_battery_assets, db): # Check costs are correct # 60 EUR for 600 kWh consumption priced at 100 EUR/MWh - np.testing.assert_almost_equal(costs["electricity energy 0"], 100 * (1 - 0.4)) + np.testing.assert_almost_equal(costs["electricity net energy"], 100 * (1 - 0.4)) # 24000 EUR for any 24 kW consumption breach priced at 1000 EUR/kW - np.testing.assert_almost_equal(costs["any consumption breach 0"], 1000 * (25 - 1)) + np.testing.assert_almost_equal( + costs["electricity any consumption breach"], 1000 * (25 - 1) + ) # 24000 EUR for each 24 kW consumption breach per hour priced at 1000 EUR/kWh np.testing.assert_almost_equal( - costs["all consumption breaches 0"], 1000 * (25 - 1) * 96 / 4 + costs["electricity all consumption breaches"], 1000 * (25 - 1) * 96 / 4 ) # No production breaches - np.testing.assert_almost_equal(costs["any production breach 0"], 0) - np.testing.assert_almost_equal(costs["all production breaches 0"], 0 * 96) + np.testing.assert_almost_equal(costs["electricity any production breach"], 0) + np.testing.assert_almost_equal(costs["electricity all production breaches"], 0 * 96) # 1.3 EUR for the 5 kW extra consumption peak priced at 260 EUR/MW - np.testing.assert_almost_equal(costs["consumption peak 0"], 260 / 1000 * (25 - 20)) + np.testing.assert_almost_equal( + costs["electricity consumption peak"], 260 / 1000 * (25 - 20) + ) # No production peak - np.testing.assert_almost_equal(costs["production peak 0"], 0) + np.testing.assert_almost_equal(costs["electricity production peak"], 0) # Sample commitments np.testing.assert_almost_equal( From 6c82e1cb43bdf5a8a3a670cb989a4bfa116801ed Mon Sep 17 00:00:00 2001 From: Ahmad-Wahid Date: Thu, 4 Jun 2026 11:58:19 +0200 Subject: [PATCH 099/136] docs/scheduling: add COMMODITY and GAS_PRICE metadata field documentation Context: - Test test_all_metadata_fields_are_documented was failing because these fields were not documented - Part of multi-commodity feature development Change: - Added ``commodity`` field to the storage flex-model table - Added ``gas-price`` field to the flex-context table --- documentation/features/scheduling.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 1643330a6d..4cc319c7fd 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -70,6 +70,9 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul * - ``production-price`` - |PRODUCTION_PRICE.example| - .. include:: ../_autodoc/PRODUCTION_PRICE.rst + * - ``gas-price`` + - |GAS_PRICE.example| + - .. include:: ../_autodoc/GAS_PRICE.rst * - ``site-power-capacity`` - |SITE_POWER_CAPACITY.example| - .. include:: ../_autodoc/SITE_POWER_CAPACITY.rst @@ -183,6 +186,9 @@ For more details on the possible formats for field values, see :ref:`variable_qu * - Field - Example value - Description + * - ``commodity`` + - |COMMODITY.example| + - .. include:: ../_autodoc/COMMODITY.rst * - ``consumption`` - |CONSUMPTION.example| - .. include:: ../_autodoc/CONSUMPTION.rst From 581ff0569880f3a5b4d24ca341db4e19883209f9 Mon Sep 17 00:00:00 2001 From: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Date: Tue, 7 Jul 2026 14:37:01 +0200 Subject: [PATCH 100/136] =?UTF-8?q?Feat:=20Support=20shared=20storage=20(m?= =?UTF-8?q?ulti=E2=80=91feed=20stock)=20(#2001)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add stock-id field in Storage and DB flex model schemas Signed-off-by: Ahmad-Wahid * feat: build stock groups Signed-off-by: Ahmad-Wahid * feat: get stock groups Signed-off-by: Ahmad-Wahid * feat: add a test case for multi feed stock Signed-off-by: Ahmad-Wahid * feat: add support for shared storage Signed-off-by: Ahmad-Wahid * remove the breakpoint Signed-off-by: Ahmad-Wahid * feat: update the test case for two devices with shared stock Signed-off-by: Ahmad-Wahid * feat: add assertions with clear reasons Signed-off-by: Ahmad-Wahid * Add support for multi-device charging of shared storage Introduce stock_groups mapping to link multiple devices to a shared SOC. Aggregate stock delta across devices sharing the same battery. Update stock change calculation to use combined device flows. Add device-to-group and group-to-devices lookup for efficient shared stock computation. Signed-off-by: Ahmad-Wahid * fix: sum all devices soc contribution, and use individual device efficiencies Signed-off-by: Ahmad-Wahid * update test case for multi feed stock Signed-off-by: Ahmad-Wahid * expect to charge the battery early to see the effect of fully discharge Signed-off-by: Ahmad-Wahid * fix: update the assert statements according to the scheduler results Signed-off-by: Ahmad-Wahid * dev: first step in resolving merge conflicts Signed-off-by: F.N. Claessen * chore: code annotation Signed-off-by: F.N. Claessen * fix: not all flex-models have sensors Signed-off-by: F.N. Claessen * fix: static method has no self Signed-off-by: F.N. Claessen * delete: remove inapplicable fields for stock model Signed-off-by: F.N. Claessen * fix: fix interpretation of test results Signed-off-by: F.N. Claessen * fix: move initialization of ems_constraints Signed-off-by: F.N. Claessen * fix: resolve merge conflicts on _build_soc_schedule, copied from Ahmad Signed-off-by: F.N. Claessen * fix: remove redundant code block Signed-off-by: F.N. Claessen * dev: use "state-of-charge" key instead of "sensor" key for stock models Signed-off-by: F.N. Claessen * fix: skip StockCommitment for device models that outsource their stock model to a separately modeled device Signed-off-by: F.N. Claessen * fix: old flex models that describe a device that serves both as a feeder and stock are both categorized as device models and stock models Signed-off-by: F.N. Claessen * fix: model stock devices using the state-of-charge field instead of the sensor field Signed-off-by: F.N. Claessen * fix: identify asset to merge with db flex-model Signed-off-by: F.N. Claessen * fix: validation Signed-off-by: F.N. Claessen * fix: flex-model setup in test Signed-off-by: F.N. Claessen * fix: create stock group Signed-off-by: Ahmad-Wahid * use soc-sensor in case of missing power sensor and also correct stock groups Signed-off-by: Ahmad-Wahid * fix: create stock model for a model which has itself stock * update the assert statements Signed-off-by: Ahmad-Wahid * remove stock-id field Signed-off-by: Ahmad-Wahid * fix: correct the stock groups Signed-off-by: Ahmad-Wahid * refactor: remove unneccessary test function Signed-off-by: Ahmad-Wahid * fix: shared soc-gain, soc-usage, soc-minima and soc-maxima Signed-off-by: F.N. Claessen * fix: shared StockCommitment for preferring a full SoC Signed-off-by: F.N. Claessen * dev: todo Signed-off-by: F.N. Claessen * dev: add "test" test case Signed-off-by: F.N. Claessen * fix commodity-level commitments by grouping devices and aligning device series with scheduler index Signed-off-by: Ahmad-Wahid * fix: use net energy costs instead of individual device costs Signed-off-by: Ahmad-Wahid * fix: comment out the buggy lines Signed-off-by: Ahmad-Wahid * fix: merge conflicts Signed-off-by: F.N. Claessen * fix: add device-model in groups if it's missing Signed-off-by: Ahmad-Wahid * fix: restore SOC constraints and state-of-charge handling broken by multi-feed-stock refactor Signed-off-by: Ahmad-Wahid * feat: Split flex-context settings by commodity for dynamic capacity scheduling (#2172) * dev: Support commodity-specific prices and site capacities in storage scheduler Signed-off-by: Ahmad-Wahid * dev: Add commodity-specific flex-context schema Signed-off-by: Ahmad-Wahid * dev: Add dynamic commodity prices and split flex-context settings to capacity scheduling test Signed-off-by: Ahmad-Wahid * feat: create a shared schema for flex-context and commodity-context Signed-off-by: Ahmad-Wahid * update the test case to have inflexible-devices-sensors for each commodity-flex-context Signed-off-by: Ahmad-Wahid * refactor: loop over flex-context fields and choose all fields except 'gas-price' for electricity as commodity Signed-off-by: Ahmad-Wahid * fix: add inflexible-device-sensors to the gas commodity model Signed-off-by: Ahmad-Wahid * fix: remove self Signed-off-by: Ahmad-Wahid * fix: fall back to deprecated price fields Signed-off-by: F.N. Claessen * fix: typo Signed-off-by: F.N. Claessen * feat: store commitment costs on job meta Signed-off-by: F.N. Claessen * refactor: clarify which job is which Signed-off-by: F.N. Claessen * fix: update test expectation: the battery could save more? Signed-off-by: F.N. Claessen * dev: add todo Signed-off-by: F.N. Claessen * fix: price window should match scheduling window Signed-off-by: F.N. Claessen * fix: comment out unreasoned check Signed-off-by: F.N. Claessen * fix: update test expectation; apparently the battery could save more? Signed-off-by: F.N. Claessen * dev: add todo Signed-off-by: F.N. Claessen * chore: flake8 Signed-off-by: F.N. Claessen * dev: exclude commodities field from flex-context schema referencing a dedicated issue Signed-off-by: F.N. Claessen * fix: only save commitment costs on job if we have a job to save it on Signed-off-by: F.N. Claessen * fix: inflexible devices are electricity devices by default Signed-off-by: F.N. Claessen * delete: no more need for backwards-compatibility of the temporary gas-price field (only used during development) Signed-off-by: F.N. Claessen * chore: black Signed-off-by: F.N. Claessen * fix: optional dict key Signed-off-by: F.N. Claessen * fix: keep ems-constraints and fix the test cases (#2233) * fix: keep ems-constraints and fix the test cases Signed-off-by: Ahmad-Wahid * Update flexmeasures/data/models/planning/storage.py Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * fix: update the comment and raise value error if ems_constraints_group is not passed Signed-off-by: Ahmad-Wahid --------- Signed-off-by: Ahmad-Wahid Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: F.N. Claessen * fix: only raise in case of multiple EMS constraint DataFrames Signed-off-by: F.N. Claessen * chore: the wait for https://github.com/marshmallow-code/apispec/pull/999 is over Signed-off-by: F.N. Claessen * feat: allow any commodity, with electricity and gas serving as examples Signed-off-by: F.N. Claessen * delete: remove unreleased flex-context field for gas price Signed-off-by: F.N. Claessen * fix: add all relaxation fields to the list of fields to ignore when moving old flex-context fields into the electricity commodity context Signed-off-by: F.N. Claessen * delete: gas_price is no longer a field (remove reference to unreleased field) Signed-off-by: F.N. Claessen * delete: just treat the whole old flex-context as the electricity flex-context Signed-off-by: F.N. Claessen * chore: update openapi-specs.json Signed-off-by: F.N. Claessen * feat: list the commodity field first rather than last Signed-off-by: F.N. Claessen * feat: commodity is a field in both flex-model and flex-context Signed-off-by: F.N. Claessen * feat: flex-model commodity can also be more than just electricity and gas Signed-off-by: F.N. Claessen * docs: remove mention of gas-price field Signed-off-by: F.N. Claessen * docs: adjust scheduling section for multi-commodity Signed-off-by: F.N. Claessen * docs: adjust field descriptions for multi-commodity Signed-off-by: F.N. Claessen * feat: list the commodity field first rather than last, also in flex-model Signed-off-by: F.N. Claessen * fix: wrong data-key due to wrong indentation Signed-off-by: F.N. Claessen * docs: explain the "commodities" field Signed-off-by: F.N. Claessen * docs: no need for ill-formatted in-line code formatting Signed-off-by: F.N. Claessen * remove: devices do not have to be storages anymore Signed-off-by: F.N. Claessen * fix: set default flex-context commodity to electricity Signed-off-by: F.N. Claessen * fix: preserve field order in case schema is made OpenAPI compatible Signed-off-by: F.N. Claessen * fix: default flex-model and flex-context to empty list and empty dict, respectively, because it is possible that the entire flex-config is described in the db instead of the trigger message Signed-off-by: F.N. Claessen * Feature: flex-context per commodity as list (#2235) * feat: support passing flex-context as a list (one flex-context per commodity) Signed-off-by: F.N. Claessen * fix: set default flex-context commodity to electricity Signed-off-by: F.N. Claessen * fix: preserve field order in case schema is made OpenAPI compatible Signed-off-by: F.N. Claessen * feat: reduce documented nesting when defining a flex-context per commodity Signed-off-by: F.N. Claessen * dev: add todos Signed-off-by: F.N. Claessen * style: flake8 Signed-off-by: F.N. Claessen * scheduling: complete schema refactoring per PR #2235 - Add SensorReferenceSchema import - Override relax_constraints default to True in CommodityFlexContextSchema - Remove duplicate breach price and relax fields from FlexContextSchema - Remove duplicate set_default_breach_prices method from FlexContextSchema Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Signed-off-by: F.N. Claessen * tests: add comprehensive tests for schema refactoring - Test aggregate-consumption and aggregate-production fields - Test SharedSchema fields accessible in FlexContextSchema - Test CommodityFlexContextSchema relax_constraints defaults to True - Test shared currency logic for flex-context listings - Test breach prices in both schemas - Add noqa comment for existing unused variable Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> * scheduling: add shared currency validation for commodity contexts - Add validator to check prices share same currency across all commodity contexts - Fix test data to use actual fixture sensor names - All new tests now passing Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> * docs: add aggregate fields to documentation and update tests - Add aggregate-consumption and aggregate-production to scheduling.rst - Exclude COMMODITY_FLEX_CONTEXT and COMMODITY_FLEX_MODEL from doc test (already documented as "commodity") - Exclude aggregate fields from UI test (not yet supported in UI) Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> * chore: upgrade openapi-specs.json Signed-off-by: F.N. Claessen * fix: move _try_to_convert_price_units to SharedSchema Signed-off-by: F.N. Claessen * fix: some fields cannot use source filters; now instead of just having validators in place to forbid that, we actually stop having those fields in the schemas of those fields Signed-off-by: F.N. Claessen * chore: mypy Signed-off-by: F.N. Claessen * delete: remove redundant tests Signed-off-by: F.N. Claessen * fix: minimize diff Signed-off-by: F.N. Claessen * fix: default flex-model and flex-context to empty lists, because it is possible that the entire flex-config is described in the db instead of the trigger message (this cherry-pick was adapted, because the flex-context moved from being documented as a dict to a list) Signed-off-by: F.N. Claessen * docs: prefer unique sensor IDs in examples Signed-off-by: F.N. Claessen * docs: clarify cross-reference Signed-off-by: F.N. Claessen * feat: UI support for picking a sensor for aggregate-consumption sensor and/or aggregate-production Signed-off-by: F.N. Claessen * fix: for some reason, for flex-context fields, the sensor-only property is defined in the html in 3 places Signed-off-by: F.N. Claessen * feat: test coverage for UI support of aggregate-consumption and aggregate-production Signed-off-by: F.N. Claessen * docs: add deprecation instructions for aggregate-power Signed-off-by: F.N. Claessen * docs: move down the aggregate-power field documentation Signed-off-by: F.N. Claessen * use different compatible units. Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * update comment Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * update comment Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * update comment Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * update the comment Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * chore: update openapi-specs.json again Signed-off-by: F.N. Claessen * feat: add multi feed stock tutorial Signed-off-by: Ahmad-Wahid * remove extra chart explanation Signed-off-by: Ahmad-Wahid * fix: update flexmeasures version in openapi-specs Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> * feat: add multi commodity tutorial Signed-off-by: Ahmad-Wahid * Apply my own suggestions from code review Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> * docs: changelog entry Signed-off-by: F.N. Claessen * test: add comprehensive coverage for aggregate-consumption/production flex-context fields - Add test_asset_trigger_and_get_aggregate_schedule to verify aggregate sensors get populated (currently xfail as StorageScheduler doesn't yet populate them) - Add test_asset_trigger_with_multi_commodity_flex_context to verify aggregate sensors work with multi-commodity flex-context format (list of commodity dicts) - Document that aggregate sensor population is missing logic in StorageScheduler - Both tests verify the infrastructure and schema support works correctly * fix: compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors Signed-off-by: F.N. Claessen * Include per-commodity inflexible devices in aggregate schedules _compute_commodity_aggregate_schedules only added the top-level inflexible devices to the electricity commodity, so a commodity's own inflexible demand (e.g. a heat load) was excluded from its aggregate-consumption schedule. The aggregate then reflected only that commodity's flexible devices. Mirror the device enumeration used in _prepare so each commodity context's inflexible-device-sensors are appended to that commodity at the matching ems_schedule indices. * test: update aggregate sensor tests to verify implementation - Remove xfail markers from both tests now that StorageScheduler populates aggregate sensors - Update test docstrings to reflect actual functionality - Tests now verify that aggregate-consumption and aggregate-production sensors receive data - First test covers single aggregate pair with dual devices - Second test covers aggregates with multiple devices scheduling together * feat: test commodities referenced in only the flex-model or only the flex-context Signed-off-by: F.N. Claessen * feat: test logs should explain why response was unexpected Signed-off-by: F.N. Claessen * style: black Signed-off-by: F.N. Claessen * feat: AssetTriggerSchema accepts list-like flex-contexts Signed-off-by: F.N. Claessen * fix: skip aggregate computation for unused commodities When a multi-commodity flex_context includes a commodity (e.g., heat) but no devices exist for that commodity, the sum() of an empty list returns integer 0 instead of a pandas Series. This caused AttributeError when calling .round() on the result later in compute(). Fixed by checking if any devices contribute to the commodity before attempting to aggregate. If no devices exist for a commodity, skip creating an aggregate schedule for it. Signed-off-by: F.N. Claessen --------- Signed-off-by: F.N. Claessen Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Signed-off-by: Ahmad-Wahid Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+copilot@users.noreply.github.com> Co-authored-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Co-authored-by: Ahmad-Wahid * docs: prefer storage when talking about the device, and stock when talking about the device state Signed-off-by: F.N. Claessen * docs: discuss a more realistic case for the multi-feed-storage example; add cross-references between tutorials Signed-off-by: F.N. Claessen --------- Signed-off-by: Ahmad-Wahid Signed-off-by: F.N. Claessen Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: F.N. Claessen Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> * style: black Signed-off-by: F.N. Claessen --------- Signed-off-by: Ahmad-Wahid Signed-off-by: F.N. Claessen Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: F.N. Claessen Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+copilot@users.noreply.github.com> --- documentation/changelog.rst | 3 +- documentation/features/scheduling.rst | 26 +- documentation/index.rst | 2 + documentation/tut/flex-model-v2g.rst | 5 +- documentation/tut/multi-commodity.rst | 262 +++++ documentation/tut/multi-feed-storage.rst | 234 +++++ flexmeasures/api/common/schemas/scheduling.py | 4 +- flexmeasures/api/common/schemas/utils.py | 6 +- flexmeasures/api/v3_0/assets.py | 20 +- .../tests/test_asset_schedules_fresh_db.py | 487 ++++++++++ flexmeasures/data/models/planning/__init__.py | 63 +- .../models/planning/linear_optimization.py | 149 ++- flexmeasures/data/models/planning/storage.py | 905 +++++++++++++----- .../models/planning/tests/test_commitments.py | 686 ++++++++++++- .../data/models/planning/tests/test_solver.py | 38 +- .../data/schemas/scheduling/__init__.py | 429 ++++++--- .../data/schemas/scheduling/metadata.py | 63 +- .../data/schemas/scheduling/storage.py | 60 +- flexmeasures/data/schemas/sensors.py | 20 +- .../data/schemas/tests/test_scheduling.py | 289 ++++-- flexmeasures/data/services/scheduling.py | 5 +- flexmeasures/data/tests/conftest.py | 3 +- .../data/tests/test_scheduling_sequential.py | 31 +- .../tests/test_scheduling_simultaneous.py | 83 +- flexmeasures/ui/static/openapi-specs.json | 197 ++-- .../ui/templates/assets/asset_context.html | 6 +- flexmeasures/ui/tests/test_utils.py | 6 +- flexmeasures/utils/coding_utils.py | 29 + tests/documentation/test_schemas.py | 2 + 29 files changed, 3421 insertions(+), 692 deletions(-) create mode 100644 documentation/tut/multi-commodity.rst create mode 100644 documentation/tut/multi-feed-storage.rst diff --git a/documentation/changelog.rst b/documentation/changelog.rst index aa1166e382..c2970d3cf6 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -11,7 +11,8 @@ New features ------------- * Floor off-clock API datetimes to a non-instantaneous sensor's resolution by default when ingesting sensor data, uploading sensor data, and handling scheduler flex-model timed events; configurable with the ``floor_datetimes_to_resolution`` sensor attribute [see `PR #2146 `_] * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 `_] - +* Support multiple feeders to a shared storage [see `PR #2001 `_ ] +* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #2172 `_ and `PR #2235 `_] Infrastructure / Support ---------------------- diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 4cc319c7fd..ba40903dc9 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -5,8 +5,8 @@ Scheduling Scheduling is the main value-drive of FlexMeasures. We have two major types of schedulers built-in, for storage devices (usually batteries or hot water storage) and processes (usually in industry). -FlexMeasures computes schedules for energy systems that consist of multiple devices that consume and/or produce electricity. -We model a device as an asset with a power sensor, and compute schedules only for flexible devices, while taking into account inflexible devices. +FlexMeasures computes schedules for energy systems that consist of multiple devices that consume and/or produce a commodity (e.g. electricity or gas). +We model a device as an asset with a consumption/production sensor recording power values, and compute schedules only for flexible devices, while taking into account inflexible devices. .. contents:: :local: @@ -39,6 +39,7 @@ The flex-context The ``flex-context`` is independent of the type of flexible device that is optimized, or which scheduler is used. With the flexibility context, we aim to describe the system in which the flexible assets operate, such as its physical and contractual limitations. +For multi-commodity scheduling problems, the flex-context can be defined separately per commodity (e.g. electricity and gas). See :ref:`tut_multi_commodity` for a hands-on example. Fields can have fixed values, but some fields can also point to sensors, so they will always represent the dynamics of the asset's environment (as long as that sensor has current data). The full list of flex-context fields follows below. @@ -46,7 +47,7 @@ For more details on the possible formats for field values, see :ref:`variable_qu Where should you set these fields? Within requests to the API or by editing the relevant asset in the UI. -If they are not sent in via the API (one of the endpoints triggering schedule computation), the scheduler will look them up on the `flex-context` field of the asset. +If they are not sent in via the API (one of the endpoints triggering schedule computation), the scheduler will look them up on the flex-context field of the asset. And if the asset belongs to a larger system (a hierarchy of assets), the scheduler will also search if parent assets have them set. @@ -58,9 +59,18 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul * - Field - Example value - Description + * - ``commodity`` + - |COMMODITY_FLEX_CONTEXT.example| + - .. include:: ../_autodoc/COMMODITY_FLEX_CONTEXT.rst * - ``inflexible-device-sensors`` - |INFLEXIBLE_DEVICE_SENSORS.example| - .. include:: ../_autodoc/INFLEXIBLE_DEVICE_SENSORS.rst + * - ``aggregate-consumption`` + - |AGGREGATE_CONSUMPTION.example| + - .. include:: ../_autodoc/AGGREGATE_CONSUMPTION.rst + * - ``aggregate-production`` + - |AGGREGATE_PRODUCTION.example| + - .. include:: ../_autodoc/AGGREGATE_PRODUCTION.rst * - ``aggregate-power`` - |AGGREGATE_POWER.example| - .. include:: ../_autodoc/AGGREGATE_POWER.rst @@ -70,9 +80,6 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul * - ``production-price`` - |PRODUCTION_PRICE.example| - .. include:: ../_autodoc/PRODUCTION_PRICE.rst - * - ``gas-price`` - - |GAS_PRICE.example| - - .. include:: ../_autodoc/GAS_PRICE.rst * - ``site-power-capacity`` - |SITE_POWER_CAPACITY.example| - .. include:: ../_autodoc/SITE_POWER_CAPACITY.rst @@ -187,8 +194,8 @@ For more details on the possible formats for field values, see :ref:`variable_qu - Example value - Description * - ``commodity`` - - |COMMODITY.example| - - .. include:: ../_autodoc/COMMODITY.rst + - |COMMODITY_FLEX_MODEL.example| + - .. include:: ../_autodoc/COMMODITY_FLEX_MODEL.rst * - ``consumption`` - |CONSUMPTION.example| - .. include:: ../_autodoc/CONSUMPTION.rst @@ -274,6 +281,8 @@ However, here are some tips to model a buffer correctly: - Set ``charging-efficiency`` to the sensor describing the :abbr:`COP (coefficient of performance)` values. - Set ``storage-efficiency`` to a value below 100% to model (heat) loss. + For a hands-on example of a heat buffer fed by multiple devices, see :ref:`tut_multi_feed_storage`. + What happens if the flex model describes an infeasible problem for the storage scheduler? Excellent question! It is highly important for a robust operation that these situations still lead to a somewhat good outcome. From our practical experience, we derived a ``StorageFallbackScheduler``. @@ -283,6 +292,7 @@ depending on the first target state of charge and the capabilities of the asset. Of course, we also log a failure in the scheduling job, so it's important to take note of these failures. Often, mis-configured flex models are the reason. For a hands-on tutorial on using some of the storage flex-model fields, head over to :ref:`tut_v2g` use case and `the API documentation for triggering schedules <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_. +For further hands-on examples, see :ref:`tut_multi_feed_storage` (multiple devices feeding one shared storage) and :ref:`tut_multi_commodity` (devices on different commodities scheduled together). Finally, are you interested in the linear programming details behind the storage scheduler? Then head over to :ref:`storage_device_scheduler`! diff --git a/documentation/index.rst b/documentation/index.rst index a771ccb589..c111d19a41 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -175,6 +175,8 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum tut/toy-example-expanded tut/toy-example-multiasset-curtailment tut/flex-model-v2g + tut/multi-feed-storage + tut/multi-commodity tut/toy-example-process tut/toy-example-reporter tut/posting_data diff --git a/documentation/tut/flex-model-v2g.rst b/documentation/tut/flex-model-v2g.rst index 25b0ebd040..068fcc5e4b 100644 --- a/documentation/tut/flex-model-v2g.rst +++ b/documentation/tut/flex-model-v2g.rst @@ -144,5 +144,6 @@ To provide an incentive for cycling the battery in response to market prices, th } -We hope this demonstration helped to illustrate the flex-model of the storage scheduler. Until now, optimizing storage (like batteries) has been the sole focus of these tutorial series. -In :ref:`tut_toy_schedule_process`, we'll turn to something different: the optimal timing of processes with fixed energy work and duration. \ No newline at end of file +We hope this demonstration helped to illustrate the flex-model of the storage scheduler. +Until now, optimizing a single storage device (like a battery) has been the sole focus of these tutorial series. +In :ref:`tut_multi_feed_storage`, we'll cover scheduling several devices that feed into one shared storage. \ No newline at end of file diff --git a/documentation/tut/multi-commodity.rst b/documentation/tut/multi-commodity.rst new file mode 100644 index 0000000000..3861a0c7cf --- /dev/null +++ b/documentation/tut/multi-commodity.rst @@ -0,0 +1,262 @@ +.. _tut_multi_commodity: + +A flex-modeling tutorial for storage: Multiple commodities (gas & electricity) +------------------------------------------------------------------------------ + +The :ref:`multi-feed storage tutorial ` showed that the ``flex-model`` can be a *list*, so that several devices are scheduled together in one call. +Those devices all acted on the same commodity (electricity). But many real sites mix commodities — electricity *and* gas, for instance — each with its own price. + +FlexMeasures handles this with two ingredients: + +- a ``commodity`` field on each device in the ``flex-model``, and +- a per-commodity price listing in the ``flex-context``. + +In this tutorial we schedule a small hybrid site with one device on each commodity, and read back a cost breakdown that is tracked *per commodity*. +(For a more general introduction to flex modeling, see :ref:`describing_flexibility`. For the single-commodity, multi-device case, see :ref:`tut_multi_feed_storage`.) + + +The use case +============ + +A site has two flexible-ish devices, each acting on a different commodity: + +- A **battery** on the ``electricity`` commodity: 20 kW power, 100 kWh capacity, 95% charging and discharging efficiency. It starts at 20 kWh and must reach 80 kWh by 23:00. +- A **gas boiler** on the ``gas`` commodity: it draws a **constant 1 kW** of gas every hour, modelled as a fixed load (it is not really flexible, but it still incurs a commodity cost we want to account for). + +Prices are flat, but *different per commodity*: + +- Electricity: **100 EUR/MWh** (consumption and production) +- Gas: **50 EUR/MWh** + +We want the scheduler to optimise the battery against the electricity price, run the boiler at its fixed gas baseline, and report electricity and gas costs separately. + + +Building the flex model +======================= + +As in the multi-feed tutorial, the ``flex-model`` is a **list** with one entry per device. +What is new here is the ``commodity`` field, which tells the scheduler *which price signal* applies to each device. It defaults to ``"electricity"``. + +.. code-block:: json + + { + "flex-model": [ + { + "sensor": 1, + "commodity": "electricity", + "state-of-charge": {"sensor": 3}, + "soc-at-start": 20.0, + "soc-min": 0.0, + "soc-max": 100.0, + "soc-targets": [ + {"datetime": "2024-01-01T23:00:00+01:00", "value": 80.0} + ], + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95 + }, + { + "sensor": 2, + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0 + } + ] + } + +Here, sensor ``1`` is the battery's power sensor, sensor ``2`` is the boiler's power sensor, and sensor ``3`` is the battery's instantaneous ``state-of-charge`` sensor (referenced from the battery entry so the scheduler records its charge level). + +A few things to note: + +- **The battery is a normal storage device** (``soc-at-start``, ``soc-min``, ``soc-max``, ``soc-targets``), tagged with ``"commodity": "electricity"``. +- **The boiler is modelled as a fixed load.** With ``soc-min`` and ``soc-max`` both 0, it can store nothing; ``soc-usage`` of ``1 kW`` forces it to consume exactly 1 kW of gas every hour, which the optimiser cannot change. ``production-capacity`` of 0 kW means it can never produce gas. + +The prices live in the ``flex-context``. For a single commodity you would pass ``consumption-price`` and ``production-price`` directly. For **multiple commodities**, you instead provide a ``commodities`` list, one entry per commodity: + +.. code-block:: json + + { + "flex-context": [ + { + "commodity": "electricity", + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh" + }, + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh" + } + ] + } + +Each device's costs are then evaluated against the prices of *its own* commodity: the battery against electricity, the boiler against gas. + +.. note:: All commodities in one scheduling problem must share the same currency (here, EUR). The prices themselves can of course differ, and may be time series or sensors just like any other price in FlexMeasures. + + +Triggering the schedule +======================= + +We schedule on the **site asset**, so that FlexMeasures considers both devices together in a single optimisation. + +.. tabs:: + + .. tab:: CLI + + .. code-block:: bash + + $ flexmeasures add schedule \ + --asset 1 \ + --start 2024-01-01T00:00+01:00 \ + --duration PT24H \ + --flex-model flex-model-multi-commodity.json \ + --flex-context flex-context-multi-commodity.json + New schedule is stored. + + .. tab:: API + + Example call: `[POST] http://localhost:5000/api/v3_0/assets/1/schedules/trigger <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_: + + .. code-block:: json + + { + "start": "2024-01-01T00:00:00+01:00", + "duration": "PT24H", + "flex-model": [ + { + "sensor": 1, + "commodity": "electricity", + "state-of-charge": {"sensor": 3}, + "soc-at-start": 20.0, + "soc-min": 0.0, + "soc-max": 100.0, + "soc-targets": [ + {"datetime": "2024-01-01T23:00:00+01:00", "value": 80.0} + ], + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95 + }, + { + "sensor": 2, + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0 + } + ], + "flex-context": [ + { + "commodity": "electricity", + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh" + }, + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh" + } + ] + } + + .. tab:: FlexMeasures Client + + Using the `FlexMeasures Client `_: + + .. code-block:: python + + schedule = await client.trigger_and_get_schedule( + asset_id=1, # the site asset + start="2024-01-01T00:00:00+01:00", + duration="PT24H", + flex_model=[ + { + "sensor": 1, # battery power sensor + "commodity": "electricity", + "state-of-charge": {"sensor": 3}, # battery SoC sensor + "soc-at-start": 20.0, + "soc-min": 0.0, + "soc-max": 100.0, + "soc-targets": [ + {"datetime": "2024-01-01T23:00:00+01:00", "value": 80.0} + ], + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95, + }, + { + "sensor": 2, # boiler power sensor + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0, + }, + ], + flex_context=[ + { + "commodity": "electricity", + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh", + }, + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh", + }, + ], + ) + +The scheduler returns one schedule per device (stored on sensors ``1`` and ``2``) and a single commitment-cost result that breaks the cost down per commodity. + + +What to expect +============== + +The asset chart shows both commodities together, with the battery's stock level in between: + +.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/multi-commodity.png + :align: center + :alt: Asset-level chart of the hybrid site, showing battery power, battery state of charge, and the gas boiler. +| + +Reading the chart top to bottom: + +- **Battery power (electricity)** charges at its full 20 kW for the first three hours, then makes one partial-power step, which compensates for its charging efficiency losses to land exactly on the 80 kWh target, and then sits idle for the rest of the day. In the final hour it discharges at −20 kW. Because the electricity price is flat, there is no cheaper window to wait for, so it simply charges as early as possible (``prefer-charging-sooner`` is on by default). +- **Battery state of charge** makes the effect of that power schedule explicit: the stock rises from the 20 kWh ``soc-at-start``, reaches the 80 kWh target during the morning, holds there through the idle hours, and drops in the final hour as the battery discharges. This is the charge level you would otherwise have to infer from the power curve. +- **Gas boiler (gas)** runs at exactly 1 kW every single hour. The ``soc-usage`` field makes this a fixed load that the optimiser cannot shift — its only effect on the result is the gas cost it incurs. + +The schedules match the cost figures reported by the scheduler: + +.. code-block:: text + + Electricity (battery) + Net charge needed : 80 kWh − 20 kWh = 60 kWh stored + Grid draw : 60 kWh ÷ 0.95 = 63.16 kWh + Charge cost : 63.16 kWh × 100 EUR/MWh ≈ 6.32 EUR + Discharge credit : 20 kWh × 100 EUR/MWh = −2.00 EUR + Net electricity ≈ 4.32 EUR + + Gas (boiler) + Consumption : 1 kW × 24 h = 24 kWh + Gas cost : 0.024 MWh × 50 EUR/MWh = 1.20 EUR + + Total = 5.52 EUR + +The commitment-cost result keeps these as separate entries — ``electricity net energy`` (≈ 4.32 EUR) and ``gas net energy`` (1.20 EUR) — so you can always see how much each commodity contributed. + +.. note:: This same pattern extends to more devices and more commodities. Add further entries to the ``flex-model`` list (each with its ``commodity``) and a matching entry in the ``flex-context`` ``commodities`` list. As long as all commodities share one currency, FlexMeasures optimises them together and reports each commodity's cost on its own. + +We hope this demonstration helped to illustrate multi-commodity scheduling. +To revisit scheduling several devices that share a single commodity and stock, head back to :ref:`tut_multi_feed_storage`. +Next, in :ref:`tut_toy_schedule_process`, we'll turn to something different: the optimal timing of processes with fixed energy work and duration. diff --git a/documentation/tut/multi-feed-storage.rst b/documentation/tut/multi-feed-storage.rst new file mode 100644 index 0000000000..798b5fc830 --- /dev/null +++ b/documentation/tut/multi-feed-storage.rst @@ -0,0 +1,234 @@ +.. _tut_multi_feed_storage: + +A flex-modeling tutorial for storage: Multiple feeds into shared storage +------------------------------------------------------------------------ + +So far, our storage tutorials have considered a single power port charging and discharging a single battery. +But what if a buffer is fed by *more than one* device, each with its own power rating and efficiency? + +This is a common situation in practice: a heat buffer (a hot water tank, say) is often fed by more than one heat source, and they all charge and discharge the *same* pool of stored heat. +FlexMeasures supports this through what we call **multiple feeds into a shared storage**: several flexible devices are scheduled together, while they all point at one shared ``state-of-charge`` sensor. + +In this tutorial we will model exactly such a system: a heat buffer fed by a heat pump and a resistive heater, and let the scheduler decide which feeder to use, and when, taking each feeder's power rating into account while continuous heat demand drains the buffer. +(For a more general introduction to flex modeling, see :ref:`describing_flexibility`. For a single-device storage walk-through, see :ref:`tut_v2g`.) + + +The use case +============ + +Consider a single heat buffer with two feeders, and a single state-of-charge sensor for the buffer: + +- Both feeders can charge the buffer, but with **different power ratings**. +- The buffer has a **single state of charge** that both feeders affect. +- The buffer continuously **loses heat to demand**, so the scheduler has to keep feeding it, not just reach a one-off target. +- The scheduler should recognise the shared storage and optimise accordingly, without duplicating baselines or costs. + +Concretely, we model: + +- A ``heat buffer`` asset, with an instantaneous ``state-of-charge`` sensor (in kWh, representing the buffer's thermal energy content). +- A ``heat pump`` asset, with its own ``power`` sensor, rated at 5 kW electrical, and a coefficient of performance (COP) of 3 — slow, but efficient. +- A ``resistive heater`` asset, with its own ``power`` sensor, rated at 15 kW electrical, and a COP of 1 — fast, but inefficient. +- Both feeders only ever add heat to the buffer; neither can extract heat back out of it. + +The buffer starts at 20 kWh, may not drop below 20 kWh or exceed 22 kWh, and continuously loses heat to demand: 4 kW for most of the day, stepping up to 18 kW during a two-hour morning peak (e.g. showers and space heating first thing). +Its narrow operating band means the buffer has little room to pre-heat ahead of the peak, so the scheduler must actively respond to demand rather than simply filling up early. + + +Building the flex model +======================= + +The key idea is that the ``flex-model`` is a **list**, with one entry per flexible device, plus one entry that describes the shared storage. +Each feeder entry references its own power sensor *and* the same ``state-of-charge`` sensor. +The final entry (without a power ``sensor``) carries the constraints that apply to the shared storage itself: the start, the bounds, and the ongoing usage. + +.. code-block:: json + + { + "flex-model": [ + { + "sensor": 1, + "state-of-charge": {"sensor": 3}, + "power-capacity": "5 kW", + "production-capacity": "0 kW", + "charging-efficiency": "300%" + }, + { + "sensor": 2, + "state-of-charge": {"sensor": 3}, + "power-capacity": "15 kW", + "production-capacity": "0 kW", + "charging-efficiency": "100%" + }, + { + "state-of-charge": {"sensor": 3}, + "soc-at-start": 20.0, + "soc-min": 20.0, + "soc-max": 22.0, + "soc-usage": [ + {"start": "2024-01-01T00:00:00+01:00", "duration": "PT7H", "value": "4 kW"}, + {"start": "2024-01-01T07:00:00+01:00", "duration": "PT2H", "value": "18 kW"}, + {"start": "2024-01-01T09:00:00+01:00", "duration": "PT15H", "value": "4 kW"} + ] + } + ] + } + +Here, sensors ``1`` and ``2`` are the power sensors of the heat pump and the resistive heater, respectively, and sensor ``3`` is the shared ``state-of-charge`` sensor on the heat buffer. + +A few things to note: + +- **Each device points at the same ``state-of-charge`` sensor.** This is what tells FlexMeasures that the devices share one storage. The scheduler links the energy balance of all feeds to that single state of charge, rather than tracking a separate stock per device. +- **The shared-storage entry has no power ``sensor``.** It only carries the storage-level fields (``soc-at-start``, ``soc-min``, ``soc-max``, ``soc-usage``), which describe the buffer as a whole and must therefore not be repeated per feeder. +- **Per-device efficiencies live in the device entries.** The heat pump's ``charging-efficiency`` of ``300%`` reflects its COP of 3 (1 kWh in yields 3 kWh of heat), while the resistive heater converts electricity to heat one-to-one at ``100%``. ``production-capacity`` of ``0 kW`` on both feeders means neither can extract heat back out of the buffer — they only ever charge it. +- **``soc-usage`` models the continuous heat demand.** Unlike a one-off ``soc-targets`` entry, it drains the buffer throughout the horizon, so the scheduler must keep feeding it rather than just reach a target once. Here it steps from a 4 kW baseline up to 18 kW for a two-hour morning peak. +- **The narrow gap between ``soc-min`` and ``soc-max``** leaves the buffer only 2 kWh of slack, so it cannot simply pre-heat well ahead of the morning peak. It also means that once the heat pump's power capacity is exhausted during the peak, the buffer has almost nowhere else to draw from — which is what forces the resistive heater into action. + +.. note:: The ``state-of-charge`` sensor should have an instantaneous resolution (``PT0M``), since it records a stock value at a point in time rather than a quantity accumulated over an interval. See the ``state-of-charge`` field in :ref:`flex_models_and_schedulers`. + +For the costs, we use a flat tariff in this example, identical at every hour, so price differences over time play no role in the schedule — only the buffer's physical constraints (capacity and ongoing usage) determine *when* each feeder runs. +The flat tariff still lets the scheduler tell the feeders apart: since the heat pump needs less electricity for the same amount of heat, it is the cheaper choice whenever either feeder could do the job. + +.. code-block:: json + + { + "flex-context": { + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh" + } + } + + +Triggering the schedule +======================= + +We schedule on the **heat buffer asset**, so that FlexMeasures considers both feeders together as feeds into the buffer's shared stock. + +.. tabs:: + + .. tab:: CLI + + .. code-block:: bash + + $ flexmeasures add schedule \ + --asset 1 \ + --start 2024-01-01T00:00+01:00 \ + --duration PT24H \ + --flex-model flex-model-multi-feed.json \ + --flex-context flex-context-flat-price.json + New schedule is stored. + + .. tab:: API + + Example call: `[POST] http://localhost:5000/api/v3_0/assets/1/schedules/trigger <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_: + + .. code-block:: json + + { + "start": "2024-01-01T00:00:00+01:00", + "duration": "PT24H", + "flex-model": [ + { + "sensor": 1, + "state-of-charge": {"sensor": 3}, + "power-capacity": "5 kW", + "production-capacity": "0 kW", + "charging-efficiency": "300%" + }, + { + "sensor": 2, + "state-of-charge": {"sensor": 3}, + "power-capacity": "15 kW", + "production-capacity": "0 kW", + "charging-efficiency": "100%" + }, + { + "state-of-charge": {"sensor": 3}, + "soc-at-start": 20.0, + "soc-min": 20.0, + "soc-max": 22.0, + "soc-usage": [ + {"start": "2024-01-01T00:00:00+01:00", "duration": "PT7H", "value": "4 kW"}, + {"start": "2024-01-01T07:00:00+01:00", "duration": "PT2H", "value": "18 kW"}, + {"start": "2024-01-01T09:00:00+01:00", "duration": "PT15H", "value": "4 kW"} + ] + } + ], + "flex-context": { + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh" + } + } + + .. tab:: FlexMeasures Client + + Using the `FlexMeasures Client `_: + + .. code-block:: python + + schedule = await client.trigger_and_get_schedule( + asset_id=1, # the heat buffer asset + start="2024-01-01T00:00:00+01:00", + duration="PT24H", + flex_model=[ + { + "sensor": 1, # heat pump power sensor + "state-of-charge": {"sensor": 3}, + "power-capacity": "5 kW", + "production-capacity": "0 kW", + "charging-efficiency": "300%", + }, + { + "sensor": 2, # resistive heater power sensor + "state-of-charge": {"sensor": 3}, + "power-capacity": "15 kW", + "production-capacity": "0 kW", + "charging-efficiency": "100%", + }, + { + "state-of-charge": {"sensor": 3}, # shared stock + "soc-at-start": 20.0, + "soc-min": 20.0, + "soc-max": 22.0, + "soc-usage": [ + {"start": "2024-01-01T00:00:00+01:00", "duration": "PT7H", "value": "4 kW"}, + {"start": "2024-01-01T07:00:00+01:00", "duration": "PT2H", "value": "18 kW"}, + {"start": "2024-01-01T09:00:00+01:00", "duration": "PT15H", "value": "4 kW"}, + ], + }, + ], + flex_context={ + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh", + }, + ) + + +The scheduler returns one schedule per feeder (stored on sensors ``1`` and ``2``), and the resulting state of charge (stored on the shared ``state-of-charge`` sensor ``3``). +Note that the costs are *not* duplicated per device: because the feeders feed one shared storage, FlexMeasures computes a single energy balance for the buffer. + + +What to expect +============== + +With a flat tariff, the schedule is driven by the buffer's physical constraints (its narrow ``soc-min``/``soc-max`` band and the ongoing ``soc-usage``) *together with* each feeder's efficiency. +The scheduler specialises each feeder for the situation it is needed in: + +- **The heat pump** covers the baseline heat demand (4 kW) throughout most of the day, drawing just over 1 kW of electricity thanks to its COP of 3. It is cheaper than the resistive heater for the same amount of heat, so the optimiser always prefers it when its power capacity allows. +- **The resistive heater** stays idle until the two-hour morning peak (18 kW), when demand exceeds what the heat pump can supply even at its own power capacity (15 kW of heat). It switches on for the first part of the peak, covering the shortfall the heat pump cannot. Towards the end of the peak, the optimiser instead lets the buffer's thin 2 kWh reserve (between ``soc-min`` and ``soc-max``) drain the rest of the way — that stored heat is "free" to use, so it is spent before calling on the resistive heater any longer than necessary. + +So, even though both feeders *could* run at any time, the optimiser only calls on the resistive heater when neither the heat pump's power capacity nor the buffer's thin margin is enough to keep up with demand. The rest of the time, the cheaper, more efficient heat pump handles everything on its own. + +.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/multi-feed-heat-buffer.png + :align: center +| + +Reading the chart top to bottom: + +- **Feeders** (top panel) shows the power schedule of both feeds together. The heat pump runs at a modest, steady power level for most of the horizon to match the baseline demand, then ramps up to its own power capacity during the morning peak. The resistive heater stays idle until the peak, switches on alongside the heat pump to cover the shortfall, then drops back to idle again before the peak ends, once it's cheaper to draw on the buffer's remaining reserve instead. +- **Shared storage** (bottom panel) shows the *single* ``state-of-charge`` sensor that both feeders feed. It sits at the 22 kWh ``soc-max`` for most of the day, dips to the 20 kWh ``soc-min`` by the end of the morning peak — as its thin 2 kWh reserve is spent covering the tail end of the shortfall — and recovers immediately afterwards. This one curve is the combined effect of both feeds and the ongoing usage, which is exactly what "shared stock" means. + +.. note:: This same pattern generalises beyond two feeders and beyond heat buffers. Any number of devices can feed a shared storage — as long as each device entry references the same ``state-of-charge`` sensor and a single entry carries the shared-stock constraints. + +We hope this demonstration helped to illustrate how FlexMeasures schedules multiple feeds into a shared storage. +For modelling a single storage device in more depth, head back to :ref:`tut_v2g`. +To see how devices on *different* commodities (e.g. electricity and gas) are scheduled together, continue to :ref:`tut_multi_commodity`. diff --git a/flexmeasures/api/common/schemas/scheduling.py b/flexmeasures/api/common/schemas/scheduling.py index f33c5ee555..92c02329e5 100644 --- a/flexmeasures/api/common/schemas/scheduling.py +++ b/flexmeasures/api/common/schemas/scheduling.py @@ -1,6 +1,6 @@ from flexmeasures.api.common.schemas.utils import make_openapi_compatible from flexmeasures.data.schemas.scheduling.storage import StorageFlexModelSchema -from flexmeasures.data.schemas.scheduling import FlexContextSchema +from flexmeasures.data.schemas.scheduling import CommodityFlexContextSchema from flexmeasures.data.schemas.sensors import SensorIdField @@ -18,4 +18,4 @@ } ], ) -flex_context_schema_openAPI = make_openapi_compatible(FlexContextSchema) +flex_context_schema_openAPI = make_openapi_compatible(CommodityFlexContextSchema) diff --git a/flexmeasures/api/common/schemas/utils.py b/flexmeasures/api/common/schemas/utils.py index f457d2bed2..3c7174cee2 100644 --- a/flexmeasures/api/common/schemas/utils.py +++ b/flexmeasures/api/common/schemas/utils.py @@ -32,7 +32,11 @@ def make_openapi_compatible( # noqa: C901 sensor_only_validators.append(validator[-1]) new_fields = {} - tobeadded_fields = schema_cls._declared_fields + try: + # in case `schema_cls.__init__` reordered the fields, preserve their order + tobeadded_fields = schema_cls().fields + except TypeError: + tobeadded_fields = schema_cls._declared_fields if include: for item in include: tobeadded_fields.update(item) diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py index 409abce35a..5e645b2216 100644 --- a/flexmeasures/api/v3_0/assets.py +++ b/flexmeasures/api/v3_0/assets.py @@ -96,22 +96,24 @@ def __init__(self, *args, **kwargs): kwargs["exclude"] = ["asset"] super().__init__(*args, **kwargs) - flex_context = fields.Nested( - flex_context_schema_openAPI, - required=True, + flex_context = fields.List( + fields.Nested( + flex_context_schema_openAPI(), + ), + load_default=[], data_key="flex-context", metadata=dict( - description="The flex-context is validated according to the scheduler's `FlexContextSchema`.", + description="Flex-context per commodity. The flex-context is validated according to the scheduler's `FlexContextSchema`.", ), ) flex_model = fields.List( fields.Nested( storage_flex_model_schema_openAPI(exclude=["asset"]), - required=True, - data_key="flex-model", - metadata=dict( - description="Flex-model per device (identified by `sensor`). The flex-model validation is handled by the scheduler. What follows is the schema used by the `StorageScheduler`.", - ), + ), + load_default=[], + data_key="flex-model", + metadata=dict( + description="Flex-model per device (identified by `sensor`). The flex-model validation is handled by the scheduler. What follows is the schema used by the `StorageScheduler`.", ), ) diff --git a/flexmeasures/api/v3_0/tests/test_asset_schedules_fresh_db.py b/flexmeasures/api/v3_0/tests/test_asset_schedules_fresh_db.py index aac1b80ffa..616a83b5b6 100644 --- a/flexmeasures/api/v3_0/tests/test_asset_schedules_fresh_db.py +++ b/flexmeasures/api/v3_0/tests/test_asset_schedules_fresh_db.py @@ -11,6 +11,8 @@ from flexmeasures import Sensor from flexmeasures.api.v3_0.tests.utils import message_for_trigger_schedule from flexmeasures.data.models.planning.tests.utils import check_constraints +from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.data.models.time_series import TimedBelief from flexmeasures.utils.job_utils import work_on_rq from flexmeasures.data.services.scheduling import ( handle_scheduling_exception, @@ -290,3 +292,488 @@ def compute_expected_length( sum(power_schedule[cheapest_hour * 4 : (cheapest_hour + 1) * 4]) > 0 ), "we expect to charge in the cheapest hour" assert_almost_equal(power_schedule, expected_uni_schedule) + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_asset_trigger_and_get_aggregate_schedule( + app, + fresh_db, + add_market_prices_fresh_db, + setup_roles_users_fresh_db, + add_charging_station_assets_fresh_db, + keep_scheduling_queue_empty, + requesting_user, +): + """Test that aggregate-consumption and aggregate-production flex-context fields get filled with data. + + This test verifies: + 1. Aggregate-consumption sensor receives the total consumption schedule with correct sign + 2. Aggregate-production sensor receives the total production schedule with correct sign + 3. The data source is correctly set to the scheduler + 4. The sign convention matches the scheduler's output (consumption positive, production negative) + """ + # Set up charging hub with aggregate sensors + bidirectional_charging_station = add_charging_station_assets_fresh_db[ + "Test charging station (bidirectional)" + ] + charging_hub = bidirectional_charging_station.parent_asset + + # Create aggregate consumption and production sensors on the hub + aggregate_consumption_sensor = Sensor( + name="aggregate-consumption", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + aggregate_production_sensor = Sensor( + name="aggregate-production", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + fresh_db.session.add(aggregate_consumption_sensor) + fresh_db.session.add(aggregate_production_sensor) + fresh_db.session.flush() + + # Set up price sensor + price_sensor_id = add_market_prices_fresh_db["epex_da"].id + + # Create flex-model with both charging stations + bidirectional_charging_station = add_charging_station_assets_fresh_db[ + "Test charging station (bidirectional)" + ] + charging_station = add_charging_station_assets_fresh_db["Test charging station"] + + sensor_1 = bidirectional_charging_station.sensors[0] + sensor_2 = charging_station.sensors[0] + bi_soc_sensor = add_charging_station_assets_fresh_db["bi-soc"] + uni_soc_sensor = add_charging_station_assets_fresh_db["uni-soc"] + + # Build the message with aggregate sensors in flex-context + message = message_for_trigger_schedule(resolution="PT30M") + message["flex-context"] = { + "consumption-price": {"sensor": price_sensor_id}, + "production-price": {"sensor": price_sensor_id}, + "site-power-capacity": "1 TW", + "aggregate-consumption": {"sensor": aggregate_consumption_sensor.id}, + "aggregate-production": {"sensor": aggregate_production_sensor.id}, + } + + # Set up flex-models for both charging stations + CP_1_flex_model = message["flex-model"].copy() + CP_1_flex_model["state-of-charge"] = {"sensor": bi_soc_sensor.id} + CP_1_flex_model["sensor"] = sensor_1.id + + CP_2_flex_model = message["flex-model"].copy() + CP_2_flex_model["state-of-charge"] = {"sensor": uni_soc_sensor.id} + CP_2_flex_model["sensor"] = sensor_2.id + + message["flex-model"] = [CP_1_flex_model, CP_2_flex_model] + + # Trigger the schedule + assert len(app.queues["scheduling"]) == 0 + with app.test_client() as client: + trigger_response = client.post( + url_for("AssetAPI:trigger_schedule", id=charging_hub.id), + json=message, + ) + assert trigger_response.status_code == 200 + job_id = trigger_response.json["schedule"] + + # Process the scheduling queue + scheduled_jobs = app.queues["scheduling"].jobs + scheduling_job = scheduled_jobs[0] + work_on_rq(app.queues["scheduling"], exc_handler=handle_scheduling_exception) + assert ( + Job.fetch(job_id, connection=app.queues["scheduling"].connection).is_finished + is True + ) + + # Verify scheduler data source was created + scheduling_job.refresh() + scheduler_source = get_data_source_for_job(scheduling_job) + assert scheduler_source is not None + + # Verify aggregate-consumption sensor got filled with data + consumption_beliefs = ( + TimedBelief.query.filter( + TimedBelief.sensor_id == aggregate_consumption_sensor.id + ) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert len(consumption_beliefs) > 0, "aggregate-consumption sensor should have data" + + # Extract consumption schedule (consumption is positive in the scheduler) + consumption_schedule = pd.Series( + [ + -v.event_value for v in consumption_beliefs + ], # Negate because DB stores consumption as negative + index=pd.DatetimeIndex([v.event_start for v in consumption_beliefs]), + ) + + # Verify aggregate-production sensor got filled with data + production_beliefs = ( + TimedBelief.query.filter( + TimedBelief.sensor_id == aggregate_production_sensor.id + ) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert len(production_beliefs) > 0, "aggregate-production sensor should have data" + + # Extract production schedule (production is negative in the scheduler, but stored as positive in DB for production sensors) + production_schedule = pd.Series( + [v.event_value for v in production_beliefs], + index=pd.DatetimeIndex([v.event_start for v in production_beliefs]), + ) + + # Verify sign conventions: some values should be positive (consumption), some negative (production) + # At least one consumption value should be positive + assert ( + consumption_schedule > 0 + ).any(), "consumption schedule should have some positive values" + + # For a test with charging, we might not have discharge, so production could be all zeros + # But we still verify the schedule structure is correct + assert ( + production_schedule >= 0 + ).all(), "production schedule should have non-negative values (production flows are positive)" + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_asset_trigger_with_multi_commodity_flex_context( + app, + fresh_db, + add_market_prices_fresh_db, + setup_roles_users_fresh_db, + add_charging_station_assets_fresh_db, + keep_scheduling_queue_empty, + requesting_user, +): + """Test aggregate sensors with multi-commodity flex-context (electricity and heat). + + This test verifies that: + 1. Multi-commodity flex-context (list format) works correctly + 2. Each commodity has its own aggregate sensors + 3. Devices with different commodities are scheduled together + 4. Aggregate sensors correctly sum their respective commodity's power flows + """ + from flexmeasures.data.models.generic_assets import GenericAssetType + + # Set up charging hub + bidirectional_cs = add_charging_station_assets_fresh_db[ + "Test charging station (bidirectional)" + ] + charging_hub = bidirectional_cs.parent_asset + + # Create a heat device (boiler) as a sibling to the charging stations + boiler_asset_type = GenericAssetType(name="boiler") + fresh_db.session.add(boiler_asset_type) + fresh_db.session.flush() + + boiler = GenericAsset( + name="Test boiler", + owner=charging_hub.owner, + generic_asset_type=boiler_asset_type, + parent_asset=charging_hub, + latitude=10, + longitude=100, + attributes=dict( + is_consumer=True, + is_producer=False, + can_shift=True, + ), + ) + boiler_power_sensor = Sensor( + name="power", + generic_asset=boiler, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + boiler_soc_sensor = Sensor( + name="heat-soc", + generic_asset=boiler, + unit="MWh", + event_resolution=pd.Timedelta(minutes=0), + ) + fresh_db.session.add(boiler) + fresh_db.session.add(boiler_power_sensor) + fresh_db.session.add(boiler_soc_sensor) + fresh_db.session.flush() + + # Create aggregate sensors for each commodity + agg_consumption_electricity = Sensor( + name="aggregate-consumption-electricity", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + agg_production_electricity = Sensor( + name="aggregate-production-electricity", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + agg_consumption_heat = Sensor( + name="aggregate-consumption-heat", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + fresh_db.session.add(agg_consumption_electricity) + fresh_db.session.add(agg_production_electricity) + fresh_db.session.add(agg_consumption_heat) + fresh_db.session.flush() + + # Set up price sensors + price_sensor_id = add_market_prices_fresh_db["epex_da"].id + + # Get the charging station + charging_station_uni = add_charging_station_assets_fresh_db["Test charging station"] + sensor_uni = charging_station_uni.sensors[0] + soc_uni_sensor = add_charging_station_assets_fresh_db["uni-soc"] + + # Build the message with multi-commodity flex-context as a LIST + message = message_for_trigger_schedule(resolution="PT30M") + + # Multi-commodity flex-context as a LIST of commodity contexts + message["flex-context"] = [ + { + "commodity": "electricity", + "consumption-price": {"sensor": price_sensor_id}, + "production-price": {"sensor": price_sensor_id}, + "site-power-capacity": "1 TW", + "aggregate-consumption": {"sensor": agg_consumption_electricity.id}, + "aggregate-production": {"sensor": agg_production_electricity.id}, + }, + { + "commodity": "heat", + "consumption-price": {"sensor": price_sensor_id}, + "site-consumption-capacity": "100 kW", + "site-production-capacity": "0 kW", + "aggregate-consumption": {"sensor": agg_consumption_heat.id}, + }, + ] + + # Set up flex-models for electricity (charging station) and heat (boiler) + flex_model_electricity = message["flex-model"].copy() + flex_model_electricity["state-of-charge"] = {"sensor": soc_uni_sensor.id} + flex_model_electricity["sensor"] = sensor_uni.id + flex_model_electricity["commodity"] = "electricity" + + flex_model_heat = { + "sensor": boiler_power_sensor.id, + "commodity": "heat", + "state-of-charge": {"sensor": boiler_soc_sensor.id}, + "soc-at-start": 10.0, + "soc-min": 0, + "soc-max": 20.0, + "soc-unit": "MWh", + "power-capacity": "1 MW", + "roundtrip-efficiency": "98%", + "storage-efficiency": "99.99%", + } + + message["flex-model"] = [flex_model_electricity, flex_model_heat] + + # Trigger the schedule + assert len(app.queues["scheduling"]) == 0 + with app.test_client() as client: + trigger_response = client.post( + url_for("AssetAPI:trigger_schedule", id=charging_hub.id), + json=message, + ) + if trigger_response.status_code != 200: + print(f"Error response: {trigger_response.json}") + assert trigger_response.status_code == 200 + job_id = trigger_response.json["schedule"] + + # Process the scheduling queue + scheduled_jobs = app.queues["scheduling"].jobs + scheduling_job = scheduled_jobs[0] + work_on_rq(app.queues["scheduling"], exc_handler=handle_scheduling_exception) + assert ( + Job.fetch(job_id, connection=app.queues["scheduling"].connection).is_finished + is True + ) + + # Verify scheduler data source + scheduling_job.refresh() + scheduler_source = get_data_source_for_job(scheduling_job) + assert scheduler_source is not None + + # Verify electricity aggregate-consumption sensor got filled + consumption_beliefs_elec = ( + TimedBelief.query.filter( + TimedBelief.sensor_id == agg_consumption_electricity.id + ) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert ( + len(consumption_beliefs_elec) > 0 + ), "electricity aggregate-consumption should have data" + + # Verify electricity aggregate-production sensor got filled + production_beliefs_elec = ( + TimedBelief.query.filter(TimedBelief.sensor_id == agg_production_electricity.id) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert ( + len(production_beliefs_elec) > 0 + ), "electricity aggregate-production should have data" + + # Verify heat aggregate-consumption sensor got filled + consumption_beliefs_heat = ( + TimedBelief.query.filter(TimedBelief.sensor_id == agg_consumption_heat.id) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert ( + len(consumption_beliefs_heat) > 0 + ), "heat aggregate-consumption should have data" + + # Verify data types are correct + assert all( + isinstance(v.event_value, (int, float)) or v.event_value is None + for v in consumption_beliefs_elec + ), "electricity consumption values should be numeric" + assert all( + isinstance(v.event_value, (int, float)) or v.event_value is None + for v in consumption_beliefs_heat + ), "heat consumption values should be numeric" + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_asset_trigger_with_flex_context_commodity_not_used( + app, + fresh_db, + add_market_prices_fresh_db, + setup_roles_users_fresh_db, + add_charging_station_assets_fresh_db, + keep_scheduling_queue_empty, + requesting_user, +): + """Test multi-commodity flex-context where one commodity is not used by any device. + + This test verifies that: + 1. Commodities in flex-context but not used in flex-model don't cause errors + 2. Aggregate sensors for unused commodities receive no data (which is expected) + 3. Devices for other commodities are scheduled normally + """ + # Set up charging hub + bidirectional_cs = add_charging_station_assets_fresh_db[ + "Test charging station (bidirectional)" + ] + charging_hub = bidirectional_cs.parent_asset + + # Create aggregate sensors for electricity and heat + agg_consumption_electricity = Sensor( + name="aggregate-consumption-elec-unused", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + agg_consumption_heat = Sensor( + name="aggregate-consumption-heat-unused", + generic_asset=charging_hub, + unit="MW", + event_resolution=pd.Timedelta(minutes=15), + ) + fresh_db.session.add(agg_consumption_electricity) + fresh_db.session.add(agg_consumption_heat) + fresh_db.session.flush() + + # Set up price sensors + price_sensor_id = add_market_prices_fresh_db["epex_da"].id + + # Get the charging station + charging_station_uni = add_charging_station_assets_fresh_db["Test charging station"] + sensor_uni = charging_station_uni.sensors[0] + soc_uni_sensor = add_charging_station_assets_fresh_db["uni-soc"] + + # Build the message with multi-commodity flex-context + message = message_for_trigger_schedule(resolution="PT30M") + + # Multi-commodity flex-context with both electricity and heat commodities + # But only electricity devices in flex-model + message["flex-context"] = [ + { + "commodity": "electricity", + "consumption-price": {"sensor": price_sensor_id}, + "production-price": {"sensor": price_sensor_id}, + "site-power-capacity": "1 TW", + "aggregate-consumption": {"sensor": agg_consumption_electricity.id}, + }, + { + "commodity": "heat", + "consumption-price": {"sensor": price_sensor_id}, + "site-consumption-capacity": "100 kW", + "site-production-capacity": "0 kW", + "aggregate-consumption": {"sensor": agg_consumption_heat.id}, + }, + ] + + # Only electricity flex-model (no heat device) + flex_model_electricity = message["flex-model"].copy() + flex_model_electricity["state-of-charge"] = {"sensor": soc_uni_sensor.id} + flex_model_electricity["sensor"] = sensor_uni.id + flex_model_electricity["commodity"] = "electricity" + + message["flex-model"] = [flex_model_electricity] + + # Trigger the schedule + assert len(app.queues["scheduling"]) == 0 + with app.test_client() as client: + trigger_response = client.post( + url_for("AssetAPI:trigger_schedule", id=charging_hub.id), + json=message, + ) + if trigger_response.status_code != 200: + print(f"Error response: {trigger_response.json}") + assert trigger_response.status_code == 200 + job_id = trigger_response.json["schedule"] + + # Process the scheduling queue + scheduled_jobs = app.queues["scheduling"].jobs + scheduling_job = scheduled_jobs[0] + work_on_rq(app.queues["scheduling"], exc_handler=handle_scheduling_exception) + assert ( + Job.fetch(job_id, connection=app.queues["scheduling"].connection).is_finished + is True + ) + + # Verify scheduler data source + scheduling_job.refresh() + scheduler_source = get_data_source_for_job(scheduling_job) + assert scheduler_source is not None + + # Verify electricity aggregate-consumption sensor got filled + consumption_beliefs_elec = ( + TimedBelief.query.filter( + TimedBelief.sensor_id == agg_consumption_electricity.id + ) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert ( + len(consumption_beliefs_elec) > 0 + ), "electricity aggregate-consumption should have data" + + # Verify heat aggregate-consumption sensor is empty (no heat device) + consumption_beliefs_heat = ( + TimedBelief.query.filter(TimedBelief.sensor_id == agg_consumption_heat.id) + .filter(TimedBelief.source_id == scheduler_source.id) + .all() + ) + assert ( + len(consumption_beliefs_heat) == 0 + ), "heat aggregate-consumption should be empty since no heat device was scheduled" diff --git a/flexmeasures/data/models/planning/__init__.py b/flexmeasures/data/models/planning/__init__.py index 2eab59bd9d..0f8c606d2f 100644 --- a/flexmeasures/data/models/planning/__init__.py +++ b/flexmeasures/data/models/planning/__init__.py @@ -1,5 +1,6 @@ from __future__ import annotations +from collections import defaultdict from collections.abc import Iterable from dataclasses import dataclass, field from datetime import datetime, timedelta @@ -13,7 +14,7 @@ from flexmeasures.data import db from flexmeasures.data.models.time_series import Sensor from flexmeasures.data.models.generic_assets import GenericAsset as Asset -from flexmeasures.utils.coding_utils import deprecated +from flexmeasures.utils.coding_utils import deprecated, merge_or_append from .exceptions import WrongEntityException @@ -53,6 +54,7 @@ class Scheduler: flex_model: list[dict] | dict | None = None flex_context: dict | None = None + stock_groups: dict | None = None fallback_scheduler_class: "Type[Scheduler] | None" = None info: dict | None = None @@ -65,6 +67,39 @@ class Scheduler: return_multiple: bool = False + @staticmethod + def _build_stock_groups(flex_model: list[dict]) -> dict: + """ + Build stock groups where devices sharing the same state-of-charge sensor are grouped together. + """ + groups = defaultdict(list) + soc_usage = defaultdict(list) + + for d, fm in enumerate(flex_model): + soc = fm.get("state_of_charge") + if soc is not None: + if hasattr(soc, "id"): + soc_id = soc.id + elif isinstance(soc, dict) and "sensor" in soc: + sensor = soc["sensor"] + soc_id = sensor.id if hasattr(sensor, "id") else sensor + else: + soc_id = soc + + soc_usage[soc_id].append(d) + + for soc_id, device_list in soc_usage.items(): + groups[soc_id] = device_list + + already_grouped = {dev for group in groups.values() for dev in group} + missing_soc_sensor_i = -len(flex_model) + for d in range(len(flex_model)): + if d not in already_grouped: + groups[missing_soc_sensor_i].append(d) + missing_soc_sensor_i += 1 + + return dict(groups) + def __init__( self, sensor: Sensor | None = None, # deprecated @@ -185,8 +220,19 @@ def collect_flex_config(self): asset = self.asset else: asset = self.sensor.generic_asset + + # Merge the passed flex_context with the db_flex_context by matching commodities db_flex_context = asset.get_flex_context() - self.flex_context = {**db_flex_context, **self.flex_context} + if isinstance(self.flex_context, dict): + self.flex_context = {**db_flex_context, **self.flex_context} + elif isinstance(self.flex_context, list): + # Currently, db_flex_context is always a dict describing only electricity + merge_or_append( + db_flex_context, + self.flex_context, + match_key="commodity", + match_value="electricity", + ) # Merge the passed flex_model with the db_flex_model by matching asset IDs db_flex_model = asset.get_flex_model() @@ -203,12 +249,19 @@ def collect_flex_config(self): # Listify the flex-model for the next code block, which actually does the merging with the db_flex_model flex_model = [flex_model] + # Find which asset is relevant for a given device model in the flex-model from the trigger message for flex_model_d in flex_model: asset_id = flex_model_d.get("asset") if asset_id is None: - sensor_id = flex_model_d["sensor"] - sensor = db.session.get(Sensor, sensor_id) - asset_id = sensor.asset_id + sensor_id = flex_model_d.get("sensor") + if sensor_id is not None: + sensor = db.session.get(Sensor, sensor_id) + asset_id = sensor.asset_id + else: + soc_sensor_ref = flex_model_d.get("state-of-charge") + if soc_sensor_ref is not None: + soc_sensor = db.session.get(Sensor, soc_sensor_ref["sensor"]) + asset_id = soc_sensor.asset_id if asset_id in db_flex_model: flex_model_d = {**db_flex_model[asset_id], **flex_model_d} amended_flex_model.append(flex_model_d) diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index b9b6c5430e..12b33fd70c 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -35,12 +35,14 @@ def device_scheduler( # noqa C901 device_constraints: list[pd.DataFrame], - ems_constraints: pd.DataFrame, + ems_constraints: pd.DataFrame | list[pd.DataFrame], commitment_quantities: list[pd.Series] | None = None, commitment_downwards_deviation_price: list[pd.Series] | list[float] | None = None, commitment_upwards_deviation_price: list[pd.Series] | list[float] | None = None, commitments: list[pd.DataFrame] | list[Commitment] | None = None, initial_stock: float | list[float] = 0, + stock_groups: dict[int, list[int]] | None = None, + ems_constraint_groups: list[list[int]] | None = None, ) -> tuple[list[pd.Series], float, SolverResults, ConcreteModel]: """This generic device scheduler is able to handle an EMS with multiple devices, with various types of constraints on the EMS level and on the device level, @@ -63,6 +65,13 @@ def device_scheduler( # noqa C901 :param ems_constraints: EMS constraints are on an EMS level. Handled constraints (listed by column name): derivative max: maximum flow derivative min: minimum flow + May be a single DataFrame (the constraint is applied to the summed flow of all devices), + or a list of DataFrames (one per device group). In the latter case, ``ems_constraint_groups`` + lists the device indices each DataFrame applies to. The StorageScheduler uses one device + group per commodity, so each commodity gets its own EMS-level capacity constraint. + :param ems_constraint_groups: For each EMS constraint DataFrame, the list of device indices it applies to. When omitted, + each EMS constraint is applied to the summed flow of all devices (legacy behaviour). A device + may appear in more than one group. :param commitments: Commitments are on an EMS level by default. Handled parameters (listed by column name): quantity: for example, 5.5 downwards deviation price: 10.1 @@ -100,6 +109,42 @@ def device_scheduler( # noqa C901 resolution = pd.to_timedelta(device_constraints[0].index.freq).to_pytimedelta() end = device_constraints[0].index.to_pydatetime()[-1] + resolution + # Normalise EMS constraints to a list of (DataFrame, device-group) pairs. + # A single DataFrame (legacy behaviour) applies to the summed flow of all devices; + # a list of DataFrames applies one EMS-level constraint per device group, as set up + # per commodity by the StorageScheduler. + all_devices = list(range(len(device_constraints))) + if isinstance(ems_constraints, pd.DataFrame): + ems_constraints_list = [ems_constraints] + ems_constraint_device_groups = [all_devices] + else: + ems_constraints_list = ems_constraints + if ems_constraint_groups is None: + if len(ems_constraints_list) > 1: + raise ValueError( + "When passing multiple EMS constraint DataFrames, you must also specify ems_constraint_groups." + ) + ems_constraint_device_groups = [all_devices for _ in ems_constraints_list] + else: + ems_constraint_device_groups = ems_constraint_groups + + # map device -> primary stock group (used for per-device stock bounds) + # and map stock group -> all member devices (used for stock accumulation). + device_to_group = {} + + if stock_groups: + for g, devices in stock_groups.items(): + for d in devices: + device_to_group[d] = g + # For devices not in any stock group (e.g., inflexible devices), + # map them to themselves so they're treated as individual groups + for d in range(len(device_constraints)): + if d not in device_to_group: + device_to_group[d] = d + else: + for d in range(len(device_constraints)): + device_to_group[d] = d + # Move commitments from old structure to new if commitments is None: commitments = [] @@ -372,15 +417,15 @@ def device_derivative_min_select(m, d, j): else: return np.nanmax([min_v, equal_v]) - def ems_derivative_max_select(m, j): - v = ems_constraints["derivative max"].iloc[j] + def ems_derivative_max_select(m, g, j): + v = ems_constraints_list[g]["derivative max"].iloc[j] if np.isnan(v): return infinity else: return v - def ems_derivative_min_select(m, j): - v = ems_constraints["derivative min"].iloc[j] + def ems_derivative_min_select(m, g, j): + v = ems_constraints_list[g]["derivative min"].iloc[j] if np.isnan(v): return -infinity else: @@ -466,8 +511,15 @@ def grouped_commitment_equalities(m, c, j, g): model.device_derivative_min = Param( model.d, model.j, initialize=device_derivative_min_select ) - model.ems_derivative_max = Param(model.j, initialize=ems_derivative_max_select) - model.ems_derivative_min = Param(model.j, initialize=ems_derivative_min_select) + model.eg = RangeSet( + 0, len(ems_constraints_list) - 1, doc="Set of EMS constraint (device) groups" + ) + model.ems_derivative_max = Param( + model.eg, model.j, initialize=ems_derivative_max_select + ) + model.ems_derivative_min = Param( + model.eg, model.j, initialize=ems_derivative_min_select + ) model.device_efficiency = Param(model.d, model.j, initialize=device_efficiency) model.device_derivative_down_efficiency = Param( model.d, model.j, initialize=device_derivative_down_efficiency @@ -499,32 +551,74 @@ def grouped_commitment_equalities(m, c, j, g): model.commitment_sign = Var(model.c, domain=Binary, initialize=0) def _get_stock_change(m, d, j): - """Determine final stock change of device d until time j. + """Determine final stock change of the stock group of device d until time j. Apply conversion efficiencies to conversion from flow to stock change and vice versa, and apply storage efficiencies to stock levels from one datetime to the next. """ + # if isinstance(initial_stock, list): + # # No initial stock defined for inflexible device + # initial_stock_d = initial_stock[d] if d < len(initial_stock) else 0 + # else: + # initial_stock_d = initial_stock + # + # stock_changes = [ + # ( + # m.device_power_down[d, k] / m.device_derivative_down_efficiency[d, k] + # + m.device_power_up[d, k] * m.device_derivative_up_efficiency[d, k] + # + m.stock_delta[d, k] + # ) + # for k in range(0, j + 1) + # ] + # efficiencies = [m.device_efficiency[d, k] for k in range(0, j + 1)] + # final_stock_change = [ + # stock - initial_stock_d + # for stock in apply_stock_changes_and_losses( + # initial_stock_d, stock_changes, efficiencies + # ) + # ][-1] + # return final_stock_change + + # determine the stock group of this device + group = device_to_group[d] + + # all devices belonging to this stock + devices = [dev for dev, g in device_to_group.items() if g == group] + + # initial stock if isinstance(initial_stock, list): - # No initial stock defined for inflexible device - initial_stock_d = initial_stock[d] if d < len(initial_stock) else 0 + initial_stock_g = initial_stock[d] if d < len(initial_stock) else 0 else: - initial_stock_d = initial_stock + initial_stock_g = initial_stock + + stock_changes = [] + + for k in range(0, j + 1): + + change = 0 + + for dev in devices: + change += ( + m.device_power_down[dev, k] + / m.device_derivative_down_efficiency[dev, k] + + m.device_power_up[dev, k] + * m.device_derivative_up_efficiency[dev, k] + + m.stock_delta[dev, k] + ) + + stock_changes.append(change) - stock_changes = [ - ( - m.device_power_down[d, k] / m.device_derivative_down_efficiency[d, k] - + m.device_power_up[d, k] * m.device_derivative_up_efficiency[d, k] - + m.stock_delta[d, k] - ) - for k in range(0, j + 1) - ] efficiencies = [m.device_efficiency[d, k] for k in range(0, j + 1)] + final_stock_change = [ - stock - initial_stock_d + stock - initial_stock_g for stock in apply_stock_changes_and_losses( - initial_stock_d, stock_changes, efficiencies + initial_stock_g, + stock_changes, + efficiencies, ) ][-1] + return final_stock_change # Add constraints as a tuple of (lower bound, value, upper bound) @@ -567,8 +661,15 @@ def device_down_derivative_sign(m, d, j): """Derivative down if sign points down, derivative not down if sign points up.""" return -m.device_power_down[d, j] <= Md * (1 - m.device_power_sign[d, j]) - def ems_derivative_bounds(m, j): - return m.ems_derivative_min[j], sum(m.ems_power[:, j]), m.ems_derivative_max[j] + def ems_derivative_bounds(m, g, j): + devices = ems_constraint_device_groups[g] + if not devices: + return Constraint.Skip + return ( + m.ems_derivative_min[g, j], + sum(m.ems_power[d, j] for d in devices), + m.ems_derivative_max[g, j], + ) def commitment_up_derivative_sign(m, c): """Up deviation active only if sign points up.""" @@ -661,7 +762,7 @@ def device_derivative_equalities(m, d, j): model.device_power_down_sign = Constraint( model.d, model.j, rule=device_down_derivative_sign ) - model.ems_power_bounds = Constraint(model.j, rule=ems_derivative_bounds) + model.ems_power_bounds = Constraint(model.eg, model.j, rule=ems_derivative_bounds) if not convex_cost_curve: model.commitment_up_derivative_sign_con = Constraint( model.c, rule=commitment_up_derivative_sign diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 04b103bcd3..4c0cd1c57e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -8,7 +8,7 @@ import pandas as pd import numpy as np from flask import current_app - +from marshmallow import ValidationError from flexmeasures import Asset, Sensor from flexmeasures.data import db @@ -32,6 +32,7 @@ from flexmeasures.data.models.planning.exceptions import InfeasibleProblemException from flexmeasures.data.schemas.scheduling.storage import StorageFlexModelSchema from flexmeasures.data.schemas.scheduling import ( + CommodityFlexContextSchema, FlexContextSchema, MultiSensorFlexModelSchema, ) @@ -78,6 +79,31 @@ def compute_schedule(self) -> pd.Series | None: return self.compute() + def _get_commodity_contexts(self) -> dict[str, dict]: + """Return commodity-specific flex-contexts. + + Supports the new format: + + "commodities": [ + {"commodity": "electricity", ...}, + {"commodity": "gas", ...}, + ] + + and keeps backwards compatibility with old top-level fields. + """ + + commodity_contexts = {} + + for commodity_context in self.flex_context.get("commodity_contexts", []): + commodity = commodity_context["commodity"] + commodity_contexts[commodity] = commodity_context + + # Backwards-compatible electricity defaults from old top-level fields. + if "electricity" not in commodity_contexts: + commodity_contexts["electricity"] = self.flex_context + + return commodity_contexts + def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 """This function prepares the required data to compute the schedule: - price data @@ -96,13 +122,100 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 resolution = self.resolution belief_time = self.belief_time + # For backwards compatibility with the single asset scheduler + # Track whether we started with a single dict (single-sensor mode) or a list + is_single_sensor_mode = not isinstance(self.flex_model, list) + flex_model = self.flex_model.copy() + if not isinstance(flex_model, list): + flex_model = [flex_model] + + # Identify stock models: entries not defining a power sensor, but only a (state-of-charge) sensor + self.stock_models = {} + + device_models = [] # everything except stock models + stock_models = {} # stock models only + + missing_soc_sensor_i = -len(flex_model) + for fm in flex_model: + + # stock model: entry in the flex-model list where the sensor key is the state-of-charge sensor of the device (e.g. a stock) + # Only apply this detection in multi-device mode; in single-sensor mode the power sensor is self.sensor (not in the fm dict) + if ( + not is_single_sensor_mode + and fm.get("sensor") is None + and (soc_sensor := fm.get("state_of_charge")) + ): + stock_models[ + soc_sensor.id if isinstance(soc_sensor, Sensor) else soc_sensor + ] = fm + continue + + """ + [ + { + "sensor": 1, + "charging-efficiency": 0.9, + "state-of-charge": {"sensor": 2}, + }, + { + "sensor": 3, + "charging-efficiency": 0.9, + "state-of-charge": {"sensor": 2}, + }, + { + "state-of-charge": {"sensor": 2}, + "storage-efficiency": 0.99, + }, + ] + """ + + # Check if this is a stock-only model (no power sensor) + # Stock-only entries have SOC parameters but no power sensor + # Only apply in multi-device mode; single-sensor mode devices have no "sensor" key by design + soc_sensor = fm.get("state_of_charge") + if ( + not is_single_sensor_mode + and fm.get("sensor") is None + and soc_sensor is not None + ): + # This is a stock-only entry, add to stock_models only + soc_id = soc_sensor.id if isinstance(soc_sensor, Sensor) else soc_sensor + stock_models[soc_id] = fm + continue + + # device model: entry in the flex-model list where the sensor key is the power sensor of the device (e.g. a feeder) + device_models.append(fm) + + # If this device has state-of-charge parameters (soc-at-start, soc-min, etc.), + # also create a stock model entry so those parameters are properly captured + if soc_sensor is not None: + soc_id = soc_sensor.id if isinstance(soc_sensor, Sensor) else soc_sensor + # Check if there are SOC parameters in this device entry + has_soc_params = any( + param in fm + for param in ["soc_at_start", "soc_min", "soc_max", "soc_targets"] + ) + if has_soc_params: + stock_models[soc_id] = fm + elif fm.get("state_of_charge") is None: + stock_models[missing_soc_sensor_i] = fm + missing_soc_sensor_i += 1 + + flex_model = device_models + self.stock_models = stock_models + self._device_models = ( + device_models # Store filtered model for later use in _build_soc_schedule + ) + + # Rebuild stock_groups using only device_models (which have sensors) + # This ensures the mapping aligns with the device indices + self.stock_groups = self._build_stock_groups(device_models) + # List the asset(s) and sensor(s) being scheduled if self.asset is not None: if not isinstance(self.flex_model, list): self.flex_model = [self.flex_model] - sensors: list[Sensor | None] = [ - flex_model_d.get("sensor") for flex_model_d in self.flex_model - ] + sensors: list[Sensor | None] = [fm.get("sensor") for fm in device_models] assets: list[Asset | None] = [ # noqa: F841 s.asset if s is not None else flex_model_d.get("asset") for s, flex_model_d in zip(sensors, self.flex_model) @@ -120,31 +233,44 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 asset = self.sensor.generic_asset assets = [asset] # noqa: F841 - # For backwards compatibility with the single asset scheduler - flex_model = self.flex_model.copy() - if not isinstance(flex_model, list): - flex_model = [flex_model] + num_flexible_devices = len(device_models) - # total number of flexible devices D described in the flex-model - num_flexible_devices = len(flex_model) + soc_at_start = [None] * num_flexible_devices + soc_targets = [None] * num_flexible_devices + soc_min = [None] * num_flexible_devices + soc_max = [None] * num_flexible_devices + soc_minima = [None] * num_flexible_devices + soc_maxima = [None] * num_flexible_devices + soc_gain = [None] * num_flexible_devices + soc_usage = [None] * num_flexible_devices + prefer_charging_sooner = [None] * num_flexible_devices + prefer_curtailing_later = [None] * num_flexible_devices + + # Assign SOC constraints from stock model to the first device in each group + for stock_id, devices in self.stock_groups.items(): - soc_at_start = [flex_model_d.get("soc_at_start") for flex_model_d in flex_model] - soc_targets = [flex_model_d.get("soc_targets") for flex_model_d in flex_model] - soc_min = [flex_model_d.get("soc_min") for flex_model_d in flex_model] - soc_max = [flex_model_d.get("soc_max") for flex_model_d in flex_model] - soc_minima = [flex_model_d.get("soc_minima") for flex_model_d in flex_model] - soc_maxima = [flex_model_d.get("soc_maxima") for flex_model_d in flex_model] + stock_model = self.stock_models.get(stock_id) + + if stock_model is None: + continue + + d0 = devices[0] + + soc_at_start[d0] = stock_model.get("soc_at_start") + soc_targets[d0] = stock_model.get("soc_targets") + soc_min[d0] = stock_model.get("soc_min") + soc_max[d0] = stock_model.get("soc_max") + soc_minima[d0] = stock_model.get("soc_minima") + soc_maxima[d0] = stock_model.get("soc_maxima") + soc_gain[d0] = stock_model.get("soc_gain") + soc_usage[d0] = stock_model.get("soc_usage") + prefer_charging_sooner[d0] = stock_model.get("prefer_charging_sooner") + prefer_curtailing_later[d0] = stock_model.get("prefer_curtailing_later") + + # todo: move storage-efficiency into a shared parameter for the first device belonging to a shared storage storage_efficiency = [ flex_model_d.get("storage_efficiency") for flex_model_d in flex_model ] - prefer_charging_sooner = [ - flex_model_d.get("prefer_charging_sooner") for flex_model_d in flex_model - ] - prefer_curtailing_later = [ - flex_model_d.get("prefer_curtailing_later") for flex_model_d in flex_model - ] - soc_gain = [flex_model_d.get("soc_gain") for flex_model_d in flex_model] - soc_usage = [flex_model_d.get("soc_usage") for flex_model_d in flex_model] consumption = [flex_model_d.get("consumption") for flex_model_d in flex_model] production = [flex_model_d.get("production") for flex_model_d in flex_model] consumption_capacity = [ @@ -161,96 +287,18 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ] # Get info from flex-context - consumption_price_sensor = self.flex_context.get("consumption_price_sensor") - production_price_sensor = self.flex_context.get("production_price_sensor") - gas_price_sensor = self.flex_context.get("gas_price_sensor") - - consumption_price = self.flex_context.get( - "consumption_price", consumption_price_sensor - ) - production_price = self.flex_context.get( - "production_price", production_price_sensor - ) - gas_price = self.flex_context.get("gas_price", gas_price_sensor) - # fallback to using the consumption price, for backwards compatibility - if production_price is None: - production_price = consumption_price inflexible_device_sensors = self.flex_context.get( "inflexible_device_sensors", [] ) - # Fetch the device's power capacity (required Sensor attribute) + # Fetch the device's power capacity required by the device constraints. power_capacity_in_mw = self._get_device_power_capacity(flex_model, assets) - gas_deviation_prices = None - if gas_price is not None: - gas_deviation_prices = get_continuous_series_sensor_or_quantity( - variable_quantity=gas_price, - unit=self.flex_context["shared_currency_unit"] + "/MWh", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ).to_frame(name="event_value") - ensure_prices_are_not_empty(gas_deviation_prices, gas_price) - gas_deviation_prices = ( - gas_deviation_prices.loc[start : end - resolution]["event_value"] - * resolution - / pd.Timedelta("1h") - ) - - # Check for known prices or price forecasts - up_deviation_prices = get_continuous_series_sensor_or_quantity( - variable_quantity=consumption_price, - unit=self.flex_context["shared_currency_unit"] + "/MWh", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ).to_frame(name="event_value") - ensure_prices_are_not_empty(up_deviation_prices, consumption_price) - down_deviation_prices = get_continuous_series_sensor_or_quantity( - variable_quantity=production_price, - unit=self.flex_context["shared_currency_unit"] + "/MWh", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fill_sides=True, - ).to_frame(name="event_value") - ensure_prices_are_not_empty(down_deviation_prices, production_price) - + # Convert to UTC before fetching time series. start = pd.Timestamp(start).tz_convert("UTC") end = pd.Timestamp(end).tz_convert("UTC") - # Create Series with EMS capacities - ems_power_capacity_in_mw = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get("ems_power_capacity_in_mw"), - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - resolve_overlaps="min", - ) - ems_consumption_capacity = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get("ems_consumption_capacity_in_mw"), - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - max_value=ems_power_capacity_in_mw, - resolve_overlaps="min", - ) - ems_production_capacity = -1 * get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get("ems_production_capacity_in_mw"), - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - max_value=ems_power_capacity_in_mw, - resolve_overlaps="min", - ) - - # Set up commitments to optimise for + # Set up commitments to optimise for. commitments = self.convert_to_commitments( query_window=(start, end), resolution=resolution, @@ -261,47 +309,103 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 index = initialize_index(start, end, resolution) commitment_quantities = initialize_series(0, start, end, resolution) - # Convert energy prices to EUR/(deviation of commitment, which is in MW) - commitment_upwards_deviation_price = ( - up_deviation_prices.loc[start : end - resolution]["event_value"] - * resolution - / pd.Timedelta("1h") - ) - commitment_downwards_deviation_price = ( - down_deviation_prices.loc[start : end - resolution]["event_value"] - * resolution - / pd.Timedelta("1h") - ) - - def _device_list_series( + # EMS constraints are kept per commodity (one device group per commodity). + # + # The site-power / site-consumption / site-production capacities + # are enforced as hard EMS-level constraints (derivative max/min). Because each + # commodity has its own set of devices, ``ems_constraints`` is a list of + # DataFrames and ``ems_constraint_groups`` lists the device indices each + # DataFrame applies to. The device_scheduler then bounds the summed flow of each + # commodity's devices separately (instead of summing across all commodities). + # + # The commodity-specific breach/peak penalties below remain modelled as + # FlowCommitments on top of these hard constraints. + ems_constraints: list[pd.DataFrame] = [] + ems_constraint_groups: list[list[int]] = [] + + def device_list_series( devices: list[int], index: pd.DatetimeIndex ) -> pd.Series: return pd.Series([tuple(devices)] * len(index), index=index, name="device") commodity_to_devices = {} - - # Set up commitments DataFrame for d, flex_model_d in enumerate(flex_model): commodity = flex_model_d.get("commodity", "electricity") commodity_to_devices.setdefault(commodity, []).append(d) + # inflexible devices are electricity by default + number_flexible_devices = len(flex_model) + number_inflexible_devices = len( + self.flex_context.get("inflexible_device_sensors", []) + ) + num_flexible_devices = len(flex_model) + commodity_to_devices["electricity"] += list( + range( + number_flexible_devices, + number_flexible_devices + number_inflexible_devices, + ) + ) + + commodity_contexts = self._get_commodity_contexts() + price_frames_by_commodity = {} + for commodity, devices in commodity_to_devices.items(): - commodity_devices = _device_list_series(devices, index) - if commodity == "electricity": - up_price = commitment_upwards_deviation_price - down_price = commitment_downwards_deviation_price - elif commodity == "gas": - if gas_deviation_prices is None: - raise ValueError( - "Gas prices are required in the flex-context to set up gas flow commitments." - ) - up_price = gas_deviation_prices - down_price = gas_deviation_prices - else: + commodity_devices = device_list_series(devices, index) + commodity_context = commodity_contexts.get(commodity, {}) + + # Get info from commodity_context + consumption_price_sensor = commodity_context.get("consumption_price_sensor") + production_price_sensor = commodity_context.get("production_price_sensor") + consumption_price = commodity_context.get( + "consumption_price", consumption_price_sensor + ) + production_price = commodity_context.get( + "production_price", production_price_sensor + ) + + if production_price is None: + production_price = consumption_price + + if consumption_price is None: raise ValueError( - f"Unsupported commodity {commodity} in flex-model. Only 'electricity' and 'gas' are supported." + f"Missing consumption price for commodity '{commodity}'." ) + # Energy prices for this commodity. + up_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=consumption_price, + unit=self.flex_context["shared_currency_unit"] + "/MWh", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ).to_frame(name="event_value") + ensure_prices_are_not_empty(up_deviation_prices, consumption_price) + + down_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=production_price, + unit=self.flex_context["shared_currency_unit"] + "/MWh", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fill_sides=True, + ).to_frame(name="event_value") + ensure_prices_are_not_empty(down_deviation_prices, production_price) + + price_frames_by_commodity[commodity] = up_deviation_prices + + # Convert energy prices to price per MW deviation for one resolution step. + up_price = ( + up_deviation_prices.loc[start : end - resolution]["event_value"] + * resolution + / pd.Timedelta("1h") + ) + down_price = ( + down_deviation_prices.loc[start : end - resolution]["event_value"] + * resolution + / pd.Timedelta("1h") + ) + commitments.append( FlowCommitment( name=f"{commodity} net energy", @@ -315,10 +419,46 @@ def _device_list_series( ) ) - # Set up peak commitments - if self.flex_context.get("ems_peak_consumption_price") is not None: + # Commodity-specific site capacities. + # These are not written into ems_constraints. Instead, they are added as + # FlowCommitments that only aggregate the devices of this commodity. + ems_power_capacity = get_continuous_series_sensor_or_quantity( + variable_quantity=commodity_context.get("ems_power_capacity_in_mw"), + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + resolve_overlaps="min", + ) + + ems_consumption_capacity = get_continuous_series_sensor_or_quantity( + variable_quantity=commodity_context.get( + "ems_consumption_capacity_in_mw" + ), + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + max_value=ems_power_capacity, + resolve_overlaps="min", + ) + + ems_production_capacity = -1 * get_continuous_series_sensor_or_quantity( + variable_quantity=commodity_context.get( + "ems_production_capacity_in_mw" + ), + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + max_value=ems_power_capacity, + resolve_overlaps="min", + ) + + # Commodity-specific peak consumption commitment. + if commodity_context.get("ems_peak_consumption_price") is not None: ems_peak_consumption = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get( + variable_quantity=commodity_context.get( "ems_peak_consumption_in_mw" ), unit="MW", @@ -328,11 +468,10 @@ def _device_list_series( max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given fill_sides=True, ) - ems_peak_consumption_price = self.flex_context.get( - "ems_peak_consumption_price" - ) ems_peak_consumption_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_peak_consumption_price, + variable_quantity=commodity_context.get( + "ems_peak_consumption_price" + ), unit=self.flex_context["shared_currency_unit"] + "/MW", query_window=(start, end), resolution=resolution, @@ -352,9 +491,11 @@ def _device_list_series( commodity=commodity, ) ) - if self.flex_context.get("ems_peak_production_price") is not None: + + # Commodity-specific peak production commitment. + if commodity_context.get("ems_peak_production_price") is not None: ems_peak_production = get_continuous_series_sensor_or_quantity( - variable_quantity=self.flex_context.get( + variable_quantity=commodity_context.get( "ems_peak_production_in_mw" ), unit="MW", @@ -364,11 +505,10 @@ def _device_list_series( max_value=np.inf, # np.nan -> np.inf to ignore commitment if no quantity is given fill_sides=True, ) - ems_peak_production_price = self.flex_context.get( - "ems_peak_production_price" - ) ems_peak_production_price = get_continuous_series_sensor_or_quantity( - variable_quantity=ems_peak_production_price, + variable_quantity=commodity_context.get( + "ems_peak_production_price" + ), unit=self.flex_context["shared_currency_unit"] + "/MW", query_window=(start, end), resolution=resolution, @@ -391,17 +531,14 @@ def _device_list_series( ) # Set up capacity breach commitments and EMS capacity constraints - ems_consumption_breach_price = self.flex_context.get( + ems_consumption_breach_price = commodity_context.get( "ems_consumption_breach_price" ) - - ems_production_breach_price = self.flex_context.get( + ems_production_breach_price = commodity_context.get( "ems_production_breach_price" ) - ems_constraints = initialize_df( - StorageScheduler.COLUMNS, start, end, resolution - ) + # Commodity-specific site consumption breach. if ems_consumption_breach_price is not None: # Convert to Series @@ -427,7 +564,6 @@ def _device_list_series( ) ) - # Set up commitments DataFrame to penalize any breach commitments.append( FlowCommitment( name=f"{commodity} any consumption breach", @@ -442,7 +578,6 @@ def _device_list_series( ) ) - # Set up commitments DataFrame to penalize each breach commitments.append( FlowCommitment( name=f"{commodity} all consumption breaches", @@ -456,12 +591,7 @@ def _device_list_series( ) ) - # Take the physical capacity as a hard constraint - ems_constraints["derivative max"] = ems_power_capacity_in_mw - else: - # Take the contracted capacity as a hard constraint - ems_constraints["derivative max"] = ems_consumption_capacity - + # Commodity-specific site production breach. if ems_production_breach_price is not None: # Convert to Series @@ -514,12 +644,35 @@ def _device_list_series( commodity=commodity, ) ) - # Take the physical capacity as a hard constraint - ems_constraints["derivative min"] = -ems_power_capacity_in_mw - else: - # Take the contracted capacity as a hard constraint - ems_constraints["derivative min"] = ems_production_capacity + # Hard EMS-level capacity constraint for this commodity's device group. + # If a breach price is set, the physical power capacity is the + # hard limit (the contracted capacity is then only softly penalised via the + # breach commitments above); otherwise the contracted capacity itself is the + # hard limit. + commodity_ems_constraints = initialize_df( + StorageScheduler.COLUMNS, start, end, resolution + ) + if ems_consumption_breach_price is not None: + commodity_ems_constraints["derivative max"] = ems_power_capacity + else: + commodity_ems_constraints["derivative max"] = ems_consumption_capacity + if ems_production_breach_price is not None: + commodity_ems_constraints["derivative min"] = -ems_power_capacity + else: + commodity_ems_constraints["derivative min"] = ems_production_capacity + ems_constraints.append(commodity_ems_constraints) + ems_constraint_groups.append(list(devices)) + + # Keep one price frame for later preference logic. + # The existing "prefer charging sooner" code uses `up_deviation_prices`. + # Prefer electricity prices if available, otherwise use the first commodity price. + if "electricity" in price_frames_by_commodity: + up_deviation_prices = price_frames_by_commodity["electricity"] + elif price_frames_by_commodity: + up_deviation_prices = next(iter(price_frames_by_commodity.values())) + else: + raise ValueError("No commodity prices were available.") # Commitments per device # StockCommitment per device to prefer a full storage by penalizing not being full @@ -737,7 +890,15 @@ def _device_list_series( # soc-maxima will become a soft constraint (modelled as stock commitments), so remove hard constraint soc_maxima[d] = None - if soc_at_start[d] is not None: + # only apply SOC constraints to the first device of a shared stock + apply_soc_constraints = True + + for stock_id, devices in self.stock_groups.items(): + if d in devices and d != devices[0]: + apply_soc_constraints = False + break + + if soc_at_start[d] is not None and apply_soc_constraints: device_constraints[d] = add_storage_constraints( start, end, @@ -1050,6 +1211,42 @@ def _device_list_series( + message ) + # --- apply shared stock groups + # Store original stock_delta values for use in _build_soc_schedule + original_stock_deltas = [ + device_constraints[d]["stock delta"].copy() + for d in range(len(device_constraints)) + ] + + if hasattr(self, "stock_groups") and self.stock_groups: + for stock_id, devices in self.stock_groups.items(): + + if len(devices) <= 1: + continue + + d0 = devices[0] + + # Combine all stock_deltas on the primary device + # This ensures the optimizer sees a single shared stock + combined_delta = sum( + device_constraints[d]["stock delta"] for d in devices + ) + device_constraints[d0]["stock delta"] = combined_delta + + # Secondary devices: zero out stock_delta (it's now in primary) but keep power contribution + for d in devices[1:]: + # Zero out stock_delta since it's now in primary device's combined_delta + device_constraints[d]["stock delta"] = 0 + + # disable stock bounds for secondary devices + device_constraints[d]["equals"] = np.nan + device_constraints[d]["min"] = np.nan + device_constraints[d]["max"] = np.nan + + # Store original stock_deltas for use in _build_soc_schedule + self.original_stock_deltas = original_stock_deltas + # Device indices each EMS constraint DataFrame applies to (one group per commodity). + self.ems_constraint_groups = ems_constraint_groups return ( sensors, start, @@ -1104,6 +1301,8 @@ def convert_to_commitments( for d, flex_model_d in enumerate(flex_model): commitment = FlowCommitment( device=d, + # todo: is flex_model_d guaranteed to have "commodity? Consider defaulting the device commodity to "electricity" + # todo: should there not be something matching the "commodity" from the commitment_spec (default to "electricity") to the device commodity? device_group=flex_model_d["commodity"], **commitment_spec, ) @@ -1141,8 +1340,48 @@ def deserialize_flex_config(self): self.flex_model = {} self.collect_flex_config() - self.flex_context = FlexContextSchema().load(self.flex_context) + self._deserialize_flex_context() + self._deserialize_flex_model() + + def _deserialize_flex_context(self): + if isinstance(self.flex_context, dict): + # Load the one flex-context for electricity + self.flex_context = FlexContextSchema().load(self.flex_context) + elif isinstance(self.flex_context, list): + # Load each flex-context per commodity + for g, commodity_flex_context in enumerate(self.flex_context): + self.flex_context[g] = CommodityFlexContextSchema().load( + commodity_flex_context + ) + # Ensure all flex-contexts share the same currency unit + # todo: move this into a validator for FlexContextSchema.commodity_contexts? + shared_currency_unit = None + for commodity_flex_context in self.flex_context: + shared_currency_unit = commodity_flex_context["shared_currency_unit"] + if shared_currency_unit is None: + shared_currency_unit = commodity_flex_context[ + "shared_currency_unit" + ] + elif ( + commodity_flex_context["shared_currency_unit"] + != shared_currency_unit + ): + raise ValidationError( + f"All prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." + ) + + # Nest the flex-contexts per commodity under the commodity_contexts field + self.flex_context = dict( + commodity_contexts=self.flex_context, + shared_currency_unit=shared_currency_unit, + ) + else: + raise TypeError( + f"Unsupported type of flex-context: '{type(self.flex_context)}'" + ) + + def _deserialize_flex_model(self): if isinstance(self.flex_model, dict): if self.sensor.generic_asset.asset_type.name in storage_asset_types: self.ensure_soc_at_start() @@ -1162,18 +1401,25 @@ def deserialize_flex_config(self): # Extend schedule period in case a target exceeds its end self.possibly_extend_end(soc_targets=self.flex_model.get("soc_targets")) elif isinstance(self.flex_model, list): - # todo: ensure_soc_min_max in case the device is a storage (see line 847) self.flex_model = MultiSensorFlexModelSchema(many=True).load( self.flex_model ) for d, sensor_flex_model in enumerate(self.flex_model): - sensor_flex_model["sensor_flex_model"] = self.ensure_soc_at_start( - flex_model=sensor_flex_model["sensor_flex_model"], - sensor=sensor_flex_model.get("sensor"), + soc_sensor_id = ( + sensor_flex_model["sensor_flex_model"] + .get("state-of-charge", {}) + .get("sensor", None) ) + soc_sensor = None + if soc_sensor_id is not None: + soc_sensor = Sensor.query.filter_by(id=soc_sensor_id).first() self.flex_model[d] = StorageFlexModelSchema( start=self.start, - sensor=sensor_flex_model.get("sensor"), + sensor=( + sensor_flex_model.get("sensor") + if sensor_flex_model.get("sensor") is not None + else soc_sensor + ), default_soc_unit=sensor_flex_model["sensor_flex_model"].get( "soc-unit" ), @@ -1186,6 +1432,7 @@ def deserialize_flex_config(self): soc_targets=self.flex_model[d].get("soc_targets"), sensor=self.flex_model[d]["sensor"], ) + self.stock_groups = self._build_stock_groups(self.flex_model) else: raise TypeError( @@ -1681,61 +1928,114 @@ class StorageScheduler(MetaStorageScheduler): @staticmethod def _build_soc_schedule( flex_model: list[dict], - ems_schedule: pd.DataFrame, + ems_schedule: list[pd.Series], soc_at_start: list[float], device_constraints: list, resolution: timedelta, + stock_groups: dict[int, list[int]], ) -> dict: - """Build the state-of-charge schedule for each device that has a state-of-charge sensor. + """Build the state-of-charge schedule for each stock group. - Converts the integrated power schedule from MWh to the sensor's unit. - For sensors with a '%' unit, the soc-max flex-model field is used as capacity. - If soc-max is missing or zero for a '%' sensor, the schedule is skipped with a warning. + Supports both: + - original logic: one device per stock group + - local/shared-stock logic: multiple devices contribute to one shared stock - Note: soc-max is a QuantityField (not a VariableQuantityField), so it is always a float - after deserialization and cannot be a sensor reference. The isinstance guard below is - therefore a defensive check for forward-compatibility. + For shared stock groups, each device contribution is integrated separately with + its own efficiencies and stock delta, then summed on top of the shared initial stock. + + Converts the integrated stock schedule from MWh to the state-of-charge sensor unit. + For '%' sensors, the soc-max flex-model field is used as capacity. """ soc_schedule = {} - for d, flex_model_d in enumerate(flex_model): - state_of_charge_sensor = flex_model_d.get("state_of_charge", None) + + for stock_id, devices in stock_groups.items(): + if not devices: + continue + + d0 = devices[0] + flex_model_d0 = flex_model[d0] + + state_of_charge_sensor = flex_model_d0.get("state_of_charge") if isinstance(state_of_charge_sensor, SensorReference): state_of_charge_sensor = state_of_charge_sensor.sensor if not isinstance(state_of_charge_sensor, Sensor): continue + + # Build the SoC series for this stock group + if len(devices) > 1: + soc_contributions = [] + reference_index = None + + for d in devices: + contribution = integrate_time_series( + series=ems_schedule[d], + initial_stock=0, + stock_delta=device_constraints[d]["stock delta"] + * resolution + / timedelta(hours=1), + up_efficiency=device_constraints[d]["derivative up efficiency"], + down_efficiency=device_constraints[d][ + "derivative down efficiency" + ], + storage_efficiency=device_constraints[d]["efficiency"] + .astype(float) + .fillna(1), + ) + soc_contributions.append(contribution) + + if reference_index is None: + reference_index = contribution.index + + initial_stock = soc_at_start[d0] if soc_at_start[d0] is not None else 0 + soc = pd.Series( + [ + initial_stock + + sum(contrib.iloc[i] for contrib in soc_contributions) + for i in range(len(soc_contributions[0])) + ], + index=reference_index, + ) + else: + soc = integrate_time_series( + series=ems_schedule[d0], + initial_stock=soc_at_start[d0], + stock_delta=device_constraints[d0]["stock delta"] + * resolution + / timedelta(hours=1), + up_efficiency=device_constraints[d0]["derivative up efficiency"], + down_efficiency=device_constraints[d0][ + "derivative down efficiency" + ], + storage_efficiency=device_constraints[d0]["efficiency"] + .astype(float) + .fillna(1), + ) + + # Convert to sensor unit soc_unit = state_of_charge_sensor.unit capacity = None if soc_unit == "%": - soc_max = flex_model_d.get("soc_max") + soc_max = flex_model_d0.get("soc_max") if isinstance(soc_max, (Sensor, SensorReference)): raise ValueError( - f"Cannot convert state-of-charge schedule to '%' unit for sensor {state_of_charge_sensor.id}: " - "soc-max as a sensor reference is not supported for '%' unit conversion. " - "Skipping state-of-charge schedule." + f"Cannot convert state-of-charge schedule to '%' unit for sensor " + f"{state_of_charge_sensor.id}: soc-max as a sensor reference is " + "not supported for '%' unit conversion." ) if not soc_max: raise ValueError( - f"Cannot convert state-of-charge schedule to '%' unit for sensor {state_of_charge_sensor.id}: " - "soc-max is missing or zero. Skipping state-of-charge schedule." + f"Cannot convert state-of-charge schedule to '%' unit for sensor " + f"{state_of_charge_sensor.id}: soc-max is missing or zero." ) - capacity = f"{soc_max} MWh" # all flex model fields are in MWh by now + capacity = f"{soc_max} MWh" + soc_schedule[state_of_charge_sensor] = convert_units( - integrate_time_series( - series=ems_schedule[d], - initial_stock=soc_at_start[d], - stock_delta=device_constraints[d]["stock delta"] - * resolution - / timedelta(hours=1), - up_efficiency=device_constraints[d]["derivative up efficiency"], - down_efficiency=device_constraints[d]["derivative down efficiency"], - storage_efficiency=device_constraints[d]["efficiency"] - .astype(float) - .fillna(1), - ), + soc, from_unit="MWh", to_unit=soc_unit, capacity=capacity, ) + return soc_schedule @staticmethod @@ -1822,6 +2122,153 @@ def _build_consumption_production_schedules( ) return schedules + def _compute_commodity_aggregate_schedules( + self, + storage_schedule: dict, + ems_schedule: pd.DataFrame, + # sensors: list[Sensor | None], + ) -> None: + """Compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors. + + This method populates the storage_schedule dict with aggregate schedules for each commodity + that defines aggregate-consumption and/or aggregate-production sensors in its commodity context. + + The sign convention and split logic follows the same pattern as _build_consumption_production_schedules: + - Only aggregate-consumption defined: full aggregate schedule (consumption +, production -) + - Only aggregate-production defined: full aggregate schedule (consumption +, production -) + (sign will be flipped by make_schedule based on consumption_is_positive=False) + - Both defined: consumption sensor gets non-negative part, production sensor gets non-positive part + (sign will be flipped for production by make_schedule) + + For backwards compatibility, when no commodity_contexts are defined, all devices are treated + as electricity devices and use the top-level flex-context fields. + + :param storage_schedule: Dict to populate with aggregate schedules (will be modified in-place) + :param ems_schedule: DataFrame of per-device power schedules in MW (consumption positive) + :param sensors: List of sensors corresponding to device indices + """ + # Get the device models to reconstruct commodity_to_devices mapping + flex_model = getattr(self, "_device_models", None) + if flex_model is None: + # Fallback: reconstruct if not available (shouldn't happen in normal flow) + flex_model = ( + self.flex_model.copy() + if isinstance(self.flex_model, dict) + else [fm for fm in self.flex_model if fm.get("sensor") is not None] + ) + if not isinstance(flex_model, list): + flex_model = [flex_model] + + # Reconstruct commodity_to_devices mapping + commodity_to_devices = {} + for d, flex_model_d in enumerate(flex_model): + commodity = flex_model_d.get("commodity", "electricity") + commodity_to_devices.setdefault(commodity, []).append(d) + + # Add inflexible devices to commodities, mirroring _prepare()'s device + # enumeration so the device indices line up with ems_schedule: + # - top-level inflexible-device-sensors go to electricity (backwards compat), + # - then each commodity context's own inflexible-device-sensors are appended to + # that commodity, in the order the commodity contexts are given. + # Without step 2 below, a commodity's inflexible demand (e.g. a heat load) is left + # out of its aggregate schedule, so an aggregate-consumption sensor only reflects + # the flexible devices of that commodity. + inflexible_device_sensors = self.flex_context.get( + "inflexible_device_sensors", [] + ) + number_flexible_devices = len(flex_model) + commodity_to_devices.setdefault("electricity", []).extend( + range( + number_flexible_devices, + number_flexible_devices + len(inflexible_device_sensors), + ) + ) + + # Per-commodity inflexible devices, enumerated after the top-level ones. + num_devices = number_flexible_devices + len(inflexible_device_sensors) + for commodity_context in self.flex_context.get("commodity_contexts", []): + commodity = commodity_context["commodity"] + commodity_inflexible_device_sensors = commodity_context.get( + "inflexible_device_sensors", [] + ) + commodity_to_devices.setdefault(commodity, []).extend( + range( + num_devices, + num_devices + len(commodity_inflexible_device_sensors), + ) + ) + num_devices += len(commodity_inflexible_device_sensors) + + # Get commodity contexts (handles backwards compatibility) + commodity_contexts = self._get_commodity_contexts() + + # Process each commodity + for commodity, devices in commodity_to_devices.items(): + commodity_context = commodity_contexts.get(commodity, {}) + + # Get aggregate sensors for this commodity + aggregate_consumption_field = commodity_context.get("aggregate_consumption") + aggregate_production_field = commodity_context.get("aggregate_production") + + # Extract sensor objects + aggregate_consumption_sensor = ( + aggregate_consumption_field.get("sensor") + if isinstance(aggregate_consumption_field, dict) + and "sensor" in aggregate_consumption_field + else None + ) + aggregate_production_sensor = ( + aggregate_production_field.get("sensor") + if isinstance(aggregate_production_field, dict) + and "sensor" in aggregate_production_field + else None + ) + + # Skip if no aggregate sensors defined for this commodity + if ( + aggregate_consumption_sensor is None + and aggregate_production_sensor is None + ): + continue + + # Sum the schedules for all devices in this commodity + # ems_schedule is a list of Series, one per device + device_indices = [d for d in devices if d < len(ems_schedule)] + + # If no devices contribute to this commodity's aggregate, skip it + # (e.g., heat commodity with no heat devices) + if not device_indices: + continue + + commodity_aggregate = sum(ems_schedule[d] for d in device_indices) + + # Apply split logic based on which sensors are defined + if ( + aggregate_consumption_sensor is not None + and aggregate_production_sensor is None + ): + # Only consumption sensor: full aggregate schedule + # (consumption positive, production negative) + storage_schedule[aggregate_consumption_sensor] = commodity_aggregate + + elif ( + aggregate_production_sensor is not None + and aggregate_consumption_sensor is None + ): + # Only production sensor: full aggregate schedule in native convention + # make_schedule will flip the sign via consumption_is_positive=False + storage_schedule[aggregate_production_sensor] = commodity_aggregate + + else: + # Both sensors defined: split into consumption (>=0) and production (<=0) parts + # make_schedule will flip the sign for production sensor via consumption_is_positive=False + storage_schedule[aggregate_consumption_sensor] = ( + commodity_aggregate.clip(lower=0) + ) + storage_schedule[aggregate_production_sensor] = ( + commodity_aggregate.clip(upper=0) + ) + def compute(self, skip_validation: bool = False) -> SchedulerOutputType: """Schedule a battery or Charge Point based directly on the latest beliefs regarding market prices within the specified time window. For the resulting consumption schedule, consumption is defined as positive values. @@ -1841,18 +2288,24 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: commitments, ) = self._prepare(skip_validation=skip_validation) + initial_stock = [0] * len(soc_at_start) + + for stock_id, devices in self.stock_groups.items(): + d0 = devices[0] + s = soc_at_start[d0] + + value = s * (timedelta(hours=1) / resolution) if s is not None else 0 + + for d in devices: + initial_stock[d] = value + ems_schedule, expected_costs, scheduler_results, model = device_scheduler( device_constraints=device_constraints, ems_constraints=ems_constraints, + ems_constraint_groups=self.ems_constraint_groups, commitments=commitments, - initial_stock=[ - ( - soc_at_start_d * (timedelta(hours=1) / resolution) - if soc_at_start_d is not None - else 0 - ) - for soc_at_start_d in soc_at_start - ], + initial_stock=initial_stock, + stock_groups=self.stock_groups, ) if "infeasible" in (tc := scheduler_results.solver.termination_condition): raise InfeasibleProblemException(tc) @@ -1870,8 +2323,13 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: aggregate_power_sensor = self.flex_context.get("aggregate_power", None) if isinstance(aggregate_power_sensor, Sensor): storage_schedule[aggregate_power_sensor] = pd.concat( - ems_schedule, axis=1 + ems_schedule, + axis=1, # todo: select only electric devices (flexible and inflexible) ).sum(axis=1) + # Compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors + self._compute_commodity_aggregate_schedules( + storage_schedule, ems_schedule # , sensors + ) # Convert each device schedule to the unit of the device's power sensor storage_schedule = { @@ -1885,18 +2343,31 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: if sensor is not None } - flex_model = self.flex_model.copy() + # Use the filtered device_models (stored during _prepare) not self.flex_model + # because stock_groups was rebuilt with device indices, not original indices + flex_model_for_soc = getattr(self, "_device_models", None) + if flex_model_for_soc is None: + # Fallback: reconstruct if not available (shouldn't happen in normal flow) + flex_model_for_soc = ( + self.flex_model.copy() + if isinstance(self.flex_model, dict) + else [fm for fm in self.flex_model if fm.get("sensor") is not None] + ) - if not isinstance(self.flex_model, list): - flex_model["sensor"] = sensors[0] - flex_model = [flex_model] + if not isinstance(flex_model_for_soc, list): + flex_model_for_soc = [flex_model_for_soc] soc_schedule = self._build_soc_schedule( - flex_model, ems_schedule, soc_at_start, device_constraints, resolution + flex_model=flex_model_for_soc, + ems_schedule=ems_schedule, + soc_at_start=soc_at_start, + device_constraints=device_constraints, + stock_groups=self.stock_groups, + resolution=resolution, ) consumption_production_schedule = self._build_consumption_production_schedules( - flex_model, ems_schedule + flex_model_for_soc, ems_schedule ) # Resample each device schedule to the resolution of the device's power sensor @@ -1968,7 +2439,7 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: # Determine which sensors are consumption vs. production output sensors consumption_output_sensors = { flex_model_d["consumption"]["sensor"] - for flex_model_d in flex_model + for flex_model_d in flex_model_for_soc if isinstance(flex_model_d.get("consumption"), dict) and "sensor" in flex_model_d["consumption"] } diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index ee4204b8f3..d9ffc8a2dc 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1,5 +1,5 @@ -import pytest import pandas as pd +import pytest import numpy as np from flexmeasures.data.services.utils import get_or_create_model @@ -15,6 +15,7 @@ from flexmeasures.data.models.time_series import Sensor from flexmeasures.data.models.planning.linear_optimization import device_scheduler from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType +from flexmeasures.data.utils import save_to_db def test_multi_feed_device_scheduler_shared_buffer(): @@ -762,11 +763,18 @@ def test_mixed_gas_and_electricity_assets(app, db): }, ] - flex_context = { - "consumption-price": "100 EUR/MWh", # electricity price - "production-price": "100 EUR/MWh", - "gas-price": "50 EUR/MWh", # gas price - } + flex_context = [ + { + "commodity": "electricity", + "consumption-price": "100 EUR/MWh", # electricity price + "production-price": "100 EUR/MWh", + }, + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh", # gas price + "production-price": "50 EUR/MWh", + }, + ] scheduler = StorageScheduler( asset_or_sensor=battery, @@ -876,3 +884,669 @@ def test_mixed_gas_and_electricity_assets(app, db): f"Battery preference cost should be positive since it can optimize charging timing, " f"got {battery_total_pref:.2e}" ) + + +def test_two_devices_shared_stock(app, db): + """ + Two feeders charging a single storage. + Consider a single battery with two inverters feeding it, and a single state-of-charge sensor for the battery. + - Both inverters can charge the battery, but with different efficiencies. + - The battery has a single state of charge that both inverters affect. + - The scheduler should recognize the shared stock and optimize accordingly, without duplicating baselines or costs. + """ + # ---- time + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-02T00:00:00+01:00") + power_sensor_resolution = pd.Timedelta("15m") + soc_sensor_resolution = pd.Timedelta(0) + + # ---- assets + battery_type = get_or_create_model(GenericAssetType, name="battery") + inverter_type = get_or_create_model(GenericAssetType, name="inverter") + + battery = GenericAsset(name="battery", generic_asset_type=battery_type) + inverter_1 = GenericAsset(name="inverter 1", generic_asset_type=inverter_type) + inverter_2 = GenericAsset(name="inverter 2", generic_asset_type=inverter_type) + + db.session.add_all([battery, inverter_1, inverter_2]) + db.session.commit() + + power_1 = Sensor( + name="power", + unit="kW", + event_resolution=power_sensor_resolution, + generic_asset=inverter_1, + ) + power_2 = Sensor( + name="power", + unit="kW", + event_resolution=power_sensor_resolution, + generic_asset=inverter_2, + ) + power_3 = Sensor( + name="power", + unit="kW", + event_resolution=power_sensor_resolution, + generic_asset=battery, + ) + + state_of_charge = Sensor( + name="state-of-charge", + unit="kWh", + event_resolution=soc_sensor_resolution, + generic_asset=battery, + ) + + db.session.add_all([power_1, power_2, power_3, state_of_charge]) + db.session.commit() + + # ---- shared stock (both batteries charge from same pool) + flex_model = [ + { + "sensor": power_1.id, + "state-of-charge": {"sensor": state_of_charge.id}, + "power-capacity": "20 kW", + "charging-efficiency": 0.95, + "discharging-efficiency": 0.95, + }, + { + "sensor": power_2.id, + "state-of-charge": {"sensor": state_of_charge.id}, + "power-capacity": "20 kW", + "charging-efficiency": 0.99, + "discharging-efficiency": 0.45, + }, + { + "state-of-charge": {"sensor": state_of_charge.id}, + "soc-at-start": 20.0, + "soc-min": 10, + "soc-max": 200.0, + "soc-targets": [{"datetime": "2024-01-01T12:00:00+01:00", "value": 189.0}], + }, + ] + + flex_context = { + "consumption-price": "100 EUR/MWh", + "production-price": "100 EUR/MWh", + } + + scheduler = StorageScheduler( + asset_or_sensor=battery, + start=start, + end=end, + resolution=power_sensor_resolution, + belief_time=start, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + schedules = scheduler.compute(skip_validation=True) + + # ---- verify scheduler returned expected outputs + assert isinstance(schedules, list), ( + "Scheduler should return a list of result objects " + "(device schedules, commitment costs, SOC)." + ) + + assert len(schedules) == 4, ( + "Expected 4 outputs: two inverter schedules, one commitment_costs " + "object, and one state_of_charge schedule." + ) + + # ---- extract schedules + storage_schedules = [s for s in schedules if s["name"] == "storage_schedule"] + commitment_costs = [s for s in schedules if s["name"] == "commitment_costs"] + soc_schedule = next(s for s in schedules if s["name"] == "state_of_charge") + + assert len(storage_schedules) == 2, ( + "There should be two storage schedules corresponding to the two " + "inverters feeding the shared battery." + ) + + assert ( + len(commitment_costs) == 1 + ), "Commitment costs should be aggregated into a single result." + + power1_schedule = next(s for s in storage_schedules if s["sensor"] == power_1) + power2_schedule = next(s for s in storage_schedules if s["sensor"] == power_2) + + power1_data = power1_schedule["data"] + power2_data = power2_schedule["data"] + soc_data = soc_schedule["data"] + costs_data = commitment_costs[0]["data"] + + # ---- charging behaviour + assert (power2_data > 0).any(), ( + "The more efficient inverter should charge the battery at least " + "during some periods, showing that the optimizer prefers it." + ) + + assert (power1_data == 0).sum() > len(power1_data) * 0.5, ( + "The less efficient inverter should remain idle for most of the " + "charging window, confirming that efficiency differences influence " + "device selection." + ) + + # ---- discharge behaviour + # Both inverters have zero power in the middle of the horizon + # Charging happens through inverter 2 (more efficient) as soon as possible (full SoC is preferred) + # Discharging happens through inverter 1 (more efficient) as late as possible (full SoC is preferred) + assert ( + power1_data.iloc[0 : int(96 / 2 + 13)] == 0 + ).all(), "Inverter 1 should be idle at the beginning of the scheduling period." + + assert ( + power2_data.iloc[int(96 / 2 - 13) : -1] == 0 + ).all(), "Inverter 2 should be idle at the end of the scheduling period." + + # Verify that inverter 1 actually discharges + assert (power1_data < 0).any(), "Inverter 1 should discharge the battery." + # Verify that inverter 1 never charges + assert not (power1_data > 0).any(), "Inverter 1 should not charge the battery." + + # Verify that inverter 2 actually charges + assert (power2_data > 0).any(), "Inverter 2 should charge the battery." + # Verify that inverter 1 never charges + assert not (power2_data < 0).any(), "Inverter 2 should not discharge the battery." + + # ---- SOC behaviour + assert soc_data.iloc[0] == pytest.approx( + 20.0 + ), "Initial state of charge must match the provided soc-at-start value." + + assert soc_data.max() == pytest.approx(189.0, rel=1e-3), ( + "SOC should rise to exactly 189.0 kWh (the target value), " + "confirming that both inverters contribute to the same shared stock." + ) + + assert soc_data.iloc[-1] == pytest.approx( + 10.0, rel=1e-3 + ), "SOC should decrease to soc-min (10.0) after the target is reached." + + assert ( + soc_data.max() > soc_data.iloc[0] + ), "SOC must increase during the charging phase." + + # ---- energy cost checks + electricity_net_energy_cost = costs_data.get("electricity net energy", 0) + assert electricity_net_energy_cost == pytest.approx(0.0657, rel=1e-2), ( + "Inverter 1 (discharge efficiency 0.95) discharges ~340 kWh (20 kW for ~40 periods) " + "from 189 kWh down to 10 kWh (soc-min), incurring discharge losses. " + "Inverter 2 (charge efficiency 0.99) charges continuously at 20 kW from start until " + "reaching the soc-target of 189 kWh at 07:30, incurring minimal charge losses. " + "Net electricity cost of ~0.0657 EUR at 100 EUR/MWh reflects the efficiency difference " + "between the two inverters specializing in their respective operations." + ) + + +def set_up_simulation_assets_and_sensors(app, db): + # ---- asset types and assets + gas_boiler_type = get_or_create_model(GenericAssetType, name="gas-boiler") + buffer_type = get_or_create_model(GenericAssetType, name="heat-buffer") + site_type = get_or_create_model(GenericAssetType, name="site") + + site = GenericAsset( + name="Test Site", + generic_asset_type=site_type, + ) + building = GenericAsset( + name="Building", generic_asset_type=site_type, parent_asset_id=site.id + ) + + gas_boiler = GenericAsset( + name="Gas Boiler", generic_asset_type=gas_boiler_type, parent_asset_id=site.id + ) + heat_buffer = GenericAsset( + name="Heat Buffer", generic_asset_type=buffer_type, parent_asset_id=site.id + ) + electric_heater = GenericAsset( + name="Electric Heater", + generic_asset_type=get_or_create_model( + GenericAssetType, name="electric-heater" + ), + parent_asset_id=site.id, + ) + + db.session.add_all([gas_boiler, heat_buffer, building, electric_heater, site]) + db.session.commit() + + power_resolution = pd.Timedelta("15m") + energy_resolution = pd.Timedelta(0) + + building_raw_power = Sensor( + name="building raw power", + unit="kW", + event_resolution=power_resolution, + generic_asset=building, + ) + + boiler_power = Sensor( + name="boiler power", + unit="kW", + event_resolution=power_resolution, + generic_asset=gas_boiler, + ) + + tank_power = Sensor( + name="heat buffer power", + unit="kW", + event_resolution=power_resolution, + generic_asset=heat_buffer, + ) + + buffer_soc = Sensor( + name="buffer state of charge", + unit="kWh", + event_resolution=energy_resolution, # instantaneous + generic_asset=heat_buffer, + ) + + buffer_soc_usage = Sensor( + name="buffer soc usage", + unit="kW", + event_resolution=power_resolution, + generic_asset=heat_buffer, + ) + + heater_power = Sensor( + name="heater power", + unit="kW", + event_resolution=power_resolution, + generic_asset=electric_heater, + ) + soc_targets = Sensor( + name="buffer soc targets", + unit="kWh", + event_resolution=energy_resolution, # instantaneous + generic_asset=heat_buffer, + ) + consumption_price = Sensor( + name="consumption price", + unit="EUR/MWh", + event_resolution=energy_resolution, + generic_asset=site, + ) + production_price = Sensor( + name="production price", + unit="EUR/MWh", + event_resolution=energy_resolution, + generic_asset=site, + ) + gas_price = Sensor( + name="gas price", + unit="EUR/MWh", + event_resolution=energy_resolution, + generic_asset=site, + ) + dynamic_consumption_capacity = Sensor( + name="dynamic consumption capacity", + unit="kW", + event_resolution=power_resolution, + generic_asset=site, + ) + + db.session.add_all( + [ + boiler_power, + buffer_soc, + tank_power, + buffer_soc_usage, + building_raw_power, + heater_power, + soc_targets, + consumption_price, + production_price, + gas_price, + dynamic_consumption_capacity, + ] + ) + db.session.commit() + return { + "site": site, + "building": building, + "gas_boiler": gas_boiler, + "heat_buffer": heat_buffer, + "electric_heater": electric_heater, + "building_raw_power": building_raw_power, + "boiler_power": boiler_power, + "tank_power": tank_power, + "buffer_soc": buffer_soc, + "buffer_soc_usage": buffer_soc_usage, + "heater_power": heater_power, + "soc_targets": soc_targets, + "power_resolution": power_resolution, + "energy_resolution": energy_resolution, + "consumption_price": consumption_price, + "production_price": production_price, + "gas_price": gas_price, + "dynamic_consumption_capacity": dynamic_consumption_capacity, + } + + +def test_simulation_with_dynamic_consumption_capacity(app, db): + + start = pd.Timestamp("2026-04-07T00:00:00+01:00") + end = pd.Timestamp( + "2026-04-09T06:00:00+01:00" + ) # Extended to allow discharge target on April 8 + belief_time = pd.Timestamp( + "2026-04-05T00:00:00+01:00" + ) # 2 days before start for generous planning horizon + + setup_data = set_up_simulation_assets_and_sensors(app, db) + + site = setup_data["site"] + building_raw_power = setup_data["building_raw_power"] + heater_power = setup_data["heater_power"] + boiler_power = setup_data["boiler_power"] + buffer_soc = setup_data["buffer_soc"] + buffer_soc_usage = setup_data["buffer_soc_usage"] + consumption_price = setup_data["consumption_price"] + gas_price = setup_data["gas_price"] + dynamic_consumption_capacity = setup_data["dynamic_consumption_capacity"] + + import timely_beliefs as tb + from flexmeasures import Source + + # add dummy data to building raw power to ensure site-level constraints are respected + building_data = pd.Series( + 100.0, + index=pd.date_range( + start, end, freq=setup_data["power_resolution"], name="event_start" + ), + name="event_value", + ).reset_index() + + soc_usage = building_data.copy() + + bdf = tb.BeliefsDataFrame( + building_data, + belief_horizon=-pd.Timedelta(seconds=1) * np.array(range(len(building_data))), + sensor=setup_data["building_raw_power"], + source=get_or_create_model(Source, name="Simulation"), + ) + save_to_db(bdf, bulk_save_objects=False, save_changed_beliefs_only=False) + + # Dynamic site consumption capacity: + # - 1200 * 0.6 = 720 kW from 12:00 to 18:00 + # - 1200 kW for the rest of the day + dynamic_capacity_data = pd.DataFrame( + index=pd.date_range( + start, end, freq=setup_data["power_resolution"], name="event_start" + ) + ).reset_index() + + # Dynamic electricity and gas prices: + # - Electricity is cheaper than gas from 12:00 to 16:00 + # - Gas is cheaper for the rest of the day + price_index = pd.date_range( + start, + end, + freq=setup_data["power_resolution"], + name="event_start", + ) + + electricity_price_data = pd.DataFrame(index=price_index).reset_index() + gas_price_data = pd.DataFrame(index=price_index).reset_index() + + # Default prices: gas cheaper than electricity + electricity_price_data["event_value"] = 120.0 + gas_price_data["event_value"] = 90.0 + + # From 12:00 until before 16:00, electricity cheaper than gas + cheap_electricity_mask = electricity_price_data["event_start"].dt.hour.between( + 12, 15 + ) + + electricity_price_data.loc[ + cheap_electricity_mask, + "event_value", + ] = 50.0 + + gas_price_data.loc[ + cheap_electricity_mask, + "event_value", + ] = 150.0 + + bdf = tb.BeliefsDataFrame( + electricity_price_data, + belief_time=belief_time, + sensor=setup_data["consumption_price"], + source=get_or_create_model(Source, name="Simulation"), + ) + save_to_db(bdf, bulk_save_objects=False, save_changed_beliefs_only=False) + + bdf = tb.BeliefsDataFrame( + gas_price_data, + belief_time=belief_time, + sensor=setup_data["gas_price"], + source=get_or_create_model(Source, name="Simulation"), + ) + save_to_db(bdf, bulk_save_objects=False, save_changed_beliefs_only=False) + + dynamic_capacity_data["event_value"] = 100.0 + + dynamic_capacity_data.loc[ + dynamic_capacity_data["event_start"].dt.hour.between(12, 17), + "event_value", + ] = ( + 100.0 * 0.6 + ) + + bdf = tb.BeliefsDataFrame( + dynamic_capacity_data, + belief_time=belief_time, + sensor=setup_data["dynamic_consumption_capacity"], + source=get_or_create_model(Source, name="Simulation"), + ) + + save_to_db(bdf, bulk_save_objects=False, save_changed_beliefs_only=False) + + soc_usage["event_value"] = 100 + bdf = tb.BeliefsDataFrame( + soc_usage, + belief_time=belief_time, + sensor=setup_data["buffer_soc_usage"], + source=get_or_create_model(Source, name="Simulation"), + ) + + save_to_db(bdf, bulk_save_objects=False, save_changed_beliefs_only=False) + + flex_model = [ + { + "sensor": heater_power.id, + "state-of-charge": {"sensor": buffer_soc.id}, + "power-capacity": "100 kW", + "charging-efficiency": 0.9, + "commodity": "electricity", + "production-capacity": "0 kW", + # "storage-efficiency": 0.9, # todo: workaround does not work yet + }, + { + "sensor": boiler_power.id, + "state-of-charge": {"sensor": buffer_soc.id}, + "power-capacity": "100 kW", + "charging-efficiency": 0.9, + "commodity": "gas", + "production-capacity": "0 kW", + # "storage-efficiency": 0.9, # todo: workaround does not work yet + }, + { + # "sensor": tank_power.id, + "soc-min": 200.0, + "soc-max": 1000.0, + "soc-at-start": 200.0, + # "soc-targets": [ + # {"datetime": "2026-04-07T20:00:00+01:00", "value": 700.0}, + # ], + "state-of-charge": {"sensor": buffer_soc.id}, + "soc-usage": [{"sensor": buffer_soc_usage.id}], + "storage-efficiency": 0.9, # todo: does not work yet + # todo: consider assigning this to the heat commodity, maybe we can derive some useful (costs?) KPI from it + }, + ] + + flex_context = { + "commodities": [ + { + "commodity": "electricity", + "consumption-price": { + "sensor": consumption_price.id, + }, + "production-price": { + "sensor": consumption_price.id, + }, + "site-power-capacity": "1900 kW", + "site-consumption-capacity": { + "sensor": dynamic_consumption_capacity.id, + }, + "site-production-capacity": "100 kW", + "site-consumption-breach-price": "100000 EUR/kW", + "site-production-breach-price": "100000 EUR/kW", + "inflexible-device-sensors": [building_raw_power.id], + }, + { + "commodity": "gas", + "consumption-price": { + "sensor": gas_price.id, + }, + "production-price": { + "sensor": gas_price.id, + }, + # No electricity dynamic capacity here. + "site-consumption-capacity": "100000 kW", + "inflexible-device-sensors": [building_raw_power.id], + }, + ], + "relax-constraints": True, + } + + scheduler = StorageScheduler( + asset_or_sensor=site, + start=start, + end=end, + resolution=setup_data["power_resolution"], + belief_time=belief_time, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + schedules = scheduler.compute(skip_validation=True) + + heater_schedule = next( + schedule["data"] + for schedule in schedules + if schedule.get("sensor") == heater_power + ) + + boiler_schedule = next( + schedule["data"] + for schedule in schedules + if schedule.get("sensor") == boiler_power + ) + # The electric heater should only be active in the cheap-electricity window. + # In local time, electricity is cheaper from 12:00 to 16:00. + # During this period, the dynamic electricity site capacity is only 60 kW. + # Therefore, the electric heater is expected to run at 60 kW, not its full + # 100 kW device capacity. + pd.testing.assert_series_equal( + heater_schedule.loc["2026-04-07T11:00:00+00:00":"2026-04-07T14:45:00+00:00"], + pd.Series( + 60.0, + index=pd.date_range( + "2026-04-07T11:00:00+00:00", + "2026-04-07T14:45:00+00:00", + freq="15min", + ), + dtype="float64", + ), + check_names=False, + obj=( + "electric heater dispatch during cheap-electricity window on day 1; " + "expected 60 kW because dynamic electricity capacity limits the heater" + ), + ) + + # When electricity is cheaper than gas, the gas boiler should stay off. + # The heat demand is then supplied by the electric heater instead. + pd.testing.assert_series_equal( + boiler_schedule.loc["2026-04-07T11:00:00+00:00":"2026-04-07T14:45:00+00:00"], + pd.Series( + 0.0, + index=pd.date_range( + "2026-04-07T11:00:00+00:00", + "2026-04-07T14:45:00+00:00", + freq="15min", + ), + dtype="float64", + ), + check_names=False, + obj=( + "gas boiler dispatch during cheap-electricity window on day 1; " + "expected 0 kW because electricity is cheaper than gas" + ), + ) + + pd.testing.assert_series_equal( + heater_schedule.loc["2026-04-08T11:00:00+00:00":"2026-04-08T14:45:00+00:00"], + pd.Series( + 60.0, + index=pd.date_range( + "2026-04-08T11:00:00+00:00", + "2026-04-08T14:45:00+00:00", + freq="15min", + ), + dtype="float64", + ), + check_names=False, + obj=( + "electric heater dispatch during cheap-electricity window on day 2; " + "expected 60 kW because dynamic electricity capacity limits the heater" + ), + ) + + pd.testing.assert_series_equal( + boiler_schedule.loc["2026-04-08T11:00:00+00:00":"2026-04-08T14:45:00+00:00"], + pd.Series( + 0.0, + index=pd.date_range( + "2026-04-08T11:00:00+00:00", + "2026-04-08T14:45:00+00:00", + freq="15min", + ), + dtype="float64", + ), + check_names=False, + obj=( + "gas boiler dispatch during cheap-electricity window on day 2; " + "expected 0 kW because electricity is cheaper than gas" + ), + ) + + # Outside the cheap-electricity window, gas is cheaper than electricity. + # Therefore, the gas boiler should become the preferred heat source and run + # at full 100 kW capacity, while the electric heater should remain off. + assert boiler_schedule.loc["2026-04-07T15:00:00+00:00"] == pytest.approx( + 100.0 + ), "Gas boiler should run at full capacity after the cheap-electricity window on day 1." + + assert heater_schedule.loc["2026-04-07T15:00:00+00:00"] == pytest.approx( + 0.0 + ), "Electric heater should be off after the cheap-electricity window because gas is cheaper." + + assert boiler_schedule.loc["2026-04-08T15:00:00+00:00"] == pytest.approx( + 100.0 + ), "Gas boiler should run at full capacity after the cheap-electricity window on day 2." + + assert heater_schedule.loc["2026-04-08T15:00:00+00:00"] == pytest.approx( + 0.0 + ), "Electric heater should be off after the cheap-electricity window on day 2 because gas is cheaper." + + # Before the first cheap-electricity window, the optimizer uses a partial + # 80 kW electric-heater step to prepare the heat buffer. This is part of the + # expected optimal schedule and protects against accidental dispatch changes. + assert heater_schedule.loc["2026-04-07T08:00:00+00:00"] == pytest.approx( + 80.0 + ), "Electric heater should have one expected partial 80 kW dispatch step before the first cheap-electricity window." diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 59ed27d811..c727d357e2 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -759,21 +759,16 @@ def test_building_solver_day_2( soc_max, ur.Quantity(battery.get_attribute("soc-max")).to("MWh").magnitude ) - if market_scenario == "dynamic contract": - # Result after 8 hours: Sell what you begin with (high prices drive full discharge) - assert soc_schedule.loc[start + timedelta(hours=8)] == soc_min_value - # Result after second 8 hour-interval: Buy as much as possible (low prices) - assert soc_schedule.loc[start + timedelta(hours=16)] == soc_max_value - # Result at end of day: Sold out at end of planning horizon - assert soc_schedule.iloc[-1] == soc_min_value - else: - # fixed contract: inflexible devices are not included in the energy commitment - # coupling under the new multi-commodity scheduler, so price-based scheduling - # drives the result independently of the inflexible load profile. - # The battery partially discharges early and fully discharges near end of day. - assert soc_schedule.loc[start + timedelta(hours=8)] == 2.0 - assert soc_schedule.loc[start + timedelta(hours=16)] == 2.0 - assert soc_schedule.iloc[-1] == soc_min_value + # In both scenarios the battery should fully discharge in the first 8 hours, + # fully charge in the next 8, and fully discharge again in the last 8 (driven by + # 1) the dynamic price profile, or 2) the net-consumption/net-production profile of + # the inflexible devices, which are part of the electricity commodity device group). + # Result after 8 hours: discharged as far as possible. + assert soc_schedule.loc[start + timedelta(hours=8)] == soc_min_value + # Result after second 8 hour-interval: charged as far as possible. + assert soc_schedule.loc[start + timedelta(hours=16)] == soc_max_value + # Result at end of day: discharged as far as possible. + assert soc_schedule.iloc[-1] == soc_min_value def test_soc_bounds_timeseries(db, add_battery_assets): @@ -1388,8 +1383,14 @@ def set_if_not_none(dictionary, key, value): assert all(device_constraints[0]["derivative min"] == -expected_capacity) assert all(device_constraints[0]["derivative max"] == expected_capacity) - assert all(ems_constraints["derivative min"] == expected_site_production_capacity) - assert all(ems_constraints["derivative max"] == expected_site_consumption_capacity) + # EMS constraints are kept per commodity; this single-battery case has only the + # default "electricity" commodity, so its constraints are in ems_constraints[0]. + assert all( + ems_constraints[0]["derivative min"] == expected_site_production_capacity + ) + assert all( + ems_constraints[0]["derivative max"] == expected_site_consumption_capacity + ) @pytest.mark.parametrize( @@ -1669,7 +1670,8 @@ def test_battery_power_capacity_as_sensor( data_to_solver = scheduler._prepare() device_constraints = data_to_solver[5][0] - ems_constraints = data_to_solver[6] + # EMS constraints are kept per commodity; index [0] selects the "electricity" group. + ems_constraints = data_to_solver[6][0] assert all(device_constraints["derivative min"].values == expected_production) assert all(device_constraints["derivative max"].values == expected_consumption) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 66fbe85cf2..5bd8021e6a 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -1,4 +1,6 @@ from __future__ import annotations + +from collections import OrderedDict from datetime import timedelta from typing import Any, Callable, Dict @@ -21,6 +23,7 @@ VariableQuantityField, SensorIdField, SensorReference, + OutputSensorReferenceSchema, ) from flexmeasures.data.schemas.scheduling import metadata from flexmeasures.data.schemas.units import UnitField @@ -140,71 +143,9 @@ class DBCommitmentSchema(CommitmentSchema, NoTimeSeriesSpecs): pass -class FlexContextSchema(Schema): - """This schema defines fields that provide context to the portfolio to be optimized.""" +class SharedSchema(Schema): + """Shared schema for fields common across commodities in flex-context and commodity-context.""" - # Device commitments - consumption_breach_price = VariableQuantityField( - "/MW", - data_key="consumption-breach-price", - required=False, - value_validator=validate.Range(min=0), - metadata=metadata.CONSUMPTION_BREACH_PRICE.to_dict(), - ) - production_breach_price = VariableQuantityField( - "/MW", - data_key="production-breach-price", - required=False, - value_validator=validate.Range(min=0), - metadata=metadata.PRODUCTION_BREACH_PRICE.to_dict(), - ) - soc_minima_breach_price = VariableQuantityField( - "/MWh", - data_key="soc-minima-breach-price", - required=False, - value_validator=validate.Range(min=0), - metadata=metadata.SOC_MINIMA_BREACH_PRICE.to_dict(), - ) - soc_maxima_breach_price = VariableQuantityField( - "/MWh", - data_key="soc-maxima-breach-price", - required=False, - value_validator=validate.Range(min=0), - metadata=metadata.SOC_MAXIMA_BREACH_PRICE.to_dict(), - ) - relax_constraints = fields.Bool( - data_key="relax-constraints", - load_default=False, - metadata=metadata.RELAX_CONSTRAINTS.to_dict(), - ) - # Dev fields - relax_soc_constraints = fields.Bool( - data_key="relax-soc-constraints", - load_default=False, - metadata=metadata.RELAX_SOC_CONSTRAINTS.to_dict(), - ) - relax_capacity_constraints = fields.Bool( - data_key="relax-capacity-constraints", - load_default=False, - metadata=metadata.RELAX_CAPACITY_CONSTRAINTS.to_dict(), - ) - relax_site_capacity_constraints = fields.Bool( - data_key="relax-site-capacity-constraints", - load_default=False, - metadata=metadata.RELAX_SITE_CAPACITY_CONSTRAINTS.to_dict(), - ) - - # Energy commitments - ems_power_capacity_in_mw = VariableQuantityField( - "MW", - required=False, - data_key="site-power-capacity", - value_validator=validate.Range(min=0), - metadata=metadata.SITE_POWER_CAPACITY.to_dict(), - ) - # todo: deprecated since flexmeasures==0.23 - consumption_price_sensor = SensorIdField(data_key="consumption-price-sensor") - production_price_sensor = SensorIdField(data_key="production-price-sensor") consumption_price = VariableQuantityField( "/MWh", required=False, @@ -212,6 +153,7 @@ class FlexContextSchema(Schema): return_magnitude=False, metadata=metadata.CONSUMPTION_PRICE.to_dict(), ) + production_price = VariableQuantityField( "/MWh", required=False, @@ -220,14 +162,14 @@ class FlexContextSchema(Schema): metadata=metadata.PRODUCTION_PRICE.to_dict(), ) - # Capacity breach commitments - ems_production_capacity_in_mw = VariableQuantityField( + ems_power_capacity_in_mw = VariableQuantityField( "MW", required=False, - data_key="site-production-capacity", + data_key="site-power-capacity", value_validator=validate.Range(min=0), - metadata=metadata.SITE_PRODUCTION_CAPACITY.to_dict(), + metadata=metadata.SITE_POWER_CAPACITY.to_dict(), ) + ems_consumption_capacity_in_mw = VariableQuantityField( "MW", required=False, @@ -235,6 +177,15 @@ class FlexContextSchema(Schema): value_validator=validate.Range(min=0), metadata=metadata.SITE_CONSUMPTION_CAPACITY.to_dict(), ) + + ems_production_capacity_in_mw = VariableQuantityField( + "MW", + required=False, + data_key="site-production-capacity", + value_validator=validate.Range(min=0), + metadata=metadata.SITE_PRODUCTION_CAPACITY.to_dict(), + ) + ems_consumption_breach_price = VariableQuantityField( "/MW", data_key="site-consumption-breach-price", @@ -242,6 +193,7 @@ class FlexContextSchema(Schema): value_validator=validate.Range(min=0), metadata=metadata.SITE_CONSUMPTION_BREACH_PRICE.to_dict(), ) + ems_production_breach_price = VariableQuantityField( "/MW", data_key="site-production-breach-price", @@ -250,7 +202,6 @@ class FlexContextSchema(Schema): metadata=metadata.SITE_PRODUCTION_BREACH_PRICE.to_dict(), ) - # Peak consumption commitment ems_peak_consumption_in_mw = VariableQuantityField( "MW", required=False, @@ -259,6 +210,7 @@ class FlexContextSchema(Schema): load_default=ur.Quantity("0 kW"), metadata=metadata.SITE_PEAK_CONSUMPTION.to_dict(), ) + ems_peak_consumption_price = VariableQuantityField( "/MW", data_key="site-peak-consumption-price", @@ -267,7 +219,6 @@ class FlexContextSchema(Schema): metadata=metadata.SITE_PEAK_CONSUMPTION_PRICE.to_dict(), ) - # Peak production commitment ems_peak_production_in_mw = VariableQuantityField( "MW", required=False, @@ -276,6 +227,7 @@ class FlexContextSchema(Schema): load_default=ur.Quantity("0 kW"), metadata=metadata.SITE_PEAK_PRODUCTION.to_dict(), ) + ems_peak_production_price = VariableQuantityField( "/MW", data_key="site-peak-production-price", @@ -283,7 +235,58 @@ class FlexContextSchema(Schema): value_validator=validate.Range(min=0), metadata=metadata.SITE_PEAK_PRODUCTION_PRICE.to_dict(), ) - # todo: group by month start (MS), something like a commitment resolution, or a list of datetimes representing splits of the commitments + + # Breach prices for device capacity constraints + consumption_breach_price = VariableQuantityField( + "/MW", + data_key="consumption-breach-price", + required=False, + value_validator=validate.Range(min=0), + metadata=metadata.CONSUMPTION_BREACH_PRICE.to_dict(), + ) + production_breach_price = VariableQuantityField( + "/MW", + data_key="production-breach-price", + required=False, + value_validator=validate.Range(min=0), + metadata=metadata.PRODUCTION_BREACH_PRICE.to_dict(), + ) + soc_minima_breach_price = VariableQuantityField( + "/MWh", + data_key="soc-minima-breach-price", + required=False, + value_validator=validate.Range(min=0), + metadata=metadata.SOC_MINIMA_BREACH_PRICE.to_dict(), + ) + soc_maxima_breach_price = VariableQuantityField( + "/MWh", + data_key="soc-maxima-breach-price", + required=False, + value_validator=validate.Range(min=0), + metadata=metadata.SOC_MAXIMA_BREACH_PRICE.to_dict(), + ) + + # Relaxation fields + relax_constraints = fields.Bool( + data_key="relax-constraints", + load_default=False, + metadata=metadata.RELAX_CONSTRAINTS.to_dict(), + ) + relax_soc_constraints = fields.Bool( + data_key="relax-soc-constraints", + load_default=False, + metadata=metadata.RELAX_SOC_CONSTRAINTS.to_dict(), + ) + relax_capacity_constraints = fields.Bool( + data_key="relax-capacity-constraints", + load_default=False, + metadata=metadata.RELAX_CAPACITY_CONSTRAINTS.to_dict(), + ) + relax_site_capacity_constraints = fields.Bool( + data_key="relax-site-capacity-constraints", + load_default=False, + metadata=metadata.RELAX_SITE_CAPACITY_CONSTRAINTS.to_dict(), + ) commitments = fields.Nested( CommitmentSchema, @@ -298,18 +301,19 @@ class FlexContextSchema(Schema): data_key="inflexible-device-sensors", metadata=metadata.INFLEXIBLE_DEVICE_SENSORS.to_dict(), ) - aggregate_power = VariableQuantityField( - to_unit="MW", - data_key="aggregate-power", + + # Aggregate output sensors + aggregate_consumption = fields.Nested( + OutputSensorReferenceSchema, required=False, - metadata=metadata.AGGREGATE_POWER.to_dict(), + data_key="aggregate-consumption", + metadata=metadata.AGGREGATE_CONSUMPTION.to_dict(), ) - gas_price = VariableQuantityField( - "/MWh", - data_key="gas-price", + aggregate_production = fields.Nested( + OutputSensorReferenceSchema, required=False, - return_magnitude=False, - metadata=metadata.GAS_PRICE.to_dict(), + data_key="aggregate-production", + metadata=metadata.AGGREGATE_PRODUCTION.to_dict(), ) def set_default_breach_prices( @@ -328,6 +332,109 @@ def set_default_breach_prices( ) return data + @validates_schema(pass_original=True) + def _try_to_convert_price_units(self, data: dict, original_data: dict, **kwargs): + """Convert price units to the same unit and scale if they can (incl. same currency).""" + + shared_currency_unit = None + previous_field_name = None + for field in self.declared_fields: + if field[-5:] == "price" and field in data: + price_field = self.declared_fields[field] + price_unit = price_field._get_unit(data[field]) + currency_unit = str( + ( + ur.Quantity(price_unit) / ur.Quantity(f"1{price_field.to_unit}") + ).units + ) + + if shared_currency_unit is None: + shared_currency_unit = str( + ur.Quantity(currency_unit).to_base_units().units + ) + previous_field_name = price_field.data_key + if not units_are_convertible(currency_unit, shared_currency_unit): + field_name = price_field.data_key + original_price_unit = price_field._get_original_unit( + original_data[field_name], data[field] + ) + error_message = f"Invalid unit. A valid unit would be, for example, '{shared_currency_unit + price_field.to_unit}' (this example uses '{shared_currency_unit}', because '{previous_field_name}' used that currency). However, you passed an incompatible price ('{original_price_unit}') for the '{field_name}' field." + if shared_currency_unit not in price_unit: + error_message += f" Also note that all prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." + raise ValidationError(error_message, field_name=field_name) + if shared_currency_unit is not None: + data["shared_currency_unit"] = shared_currency_unit + elif sensor := data.get("consumption_price_sensor"): + data["shared_currency_unit"] = self._to_currency_per_mwh(sensor.unit) + elif sensor := data.get("production_price_sensor"): + data["shared_currency_unit"] = self._to_currency_per_mwh(sensor.unit) + else: + data["shared_currency_unit"] = "EUR" + return data + + @staticmethod + def _to_currency_per_mwh(price_unit: str) -> str: + """Convert a price unit to a base currency used to express that price per MWh. + + >>> FlexContextSchema()._to_currency_per_mwh("EUR/MWh") + 'EUR' + >>> FlexContextSchema()._to_currency_per_mwh("EUR/kWh") + 'EUR' + """ + currency = str(ur.Quantity(price_unit + " * MWh").to_base_units().units) + return currency + + +class CommodityFlexContextSchema(SharedSchema): + commodity = fields.Str( + required=False, + load_default="electricity", + data_key="commodity", + metadata=metadata.COMMODITY_FLEX_CONTEXT.to_dict(), + ) + + # For flex-context listings (per commodity), default relax_constraints to True + relax_constraints = fields.Bool( + data_key="relax-constraints", + load_default=True, + metadata=metadata.RELAX_CONSTRAINTS.to_dict(), + ) + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + commodity_field = self.fields.pop("commodity") + self.fields = OrderedDict( + [("commodity", commodity_field), *self.fields.items()] + ) + + +class FlexContextSchema(SharedSchema): + """This schema defines fields that provide context to the portfolio to be optimized.""" + + commodity_contexts = fields.Nested( + CommodityFlexContextSchema, + data_key="commodities", + required=False, + many=True, + metadata=dict( + description="For multi-commodity scheduling problems, the above fields can be set here per commodity.", + ), + ) + + # Energy commitments + # todo: deprecated since flexmeasures==0.23 + consumption_price_sensor = SensorIdField(data_key="consumption-price-sensor") + production_price_sensor = SensorIdField(data_key="production-price-sensor") + + # todo: group by month start (MS), something like a commitment resolution, or a list of datetimes representing splits of the commitments + aggregate_power = VariableQuantityField( + to_unit="MW", + data_key="aggregate-power", + required=False, + metadata=metadata.AGGREGATE_POWER.to_dict(), + ) + @validates("aggregate_power") def validate_aggregate_power_is_sensor( self, @@ -341,6 +448,42 @@ def validate_aggregate_power_is_sensor( if not isinstance(aggregate_power, Sensor): raise ValidationError("The `aggregate-power` field can only be a Sensor.") + @validates("commodity_contexts") + def validate_commodity_contexts_shared_currency( + self, commodity_contexts: list[dict], **kwargs + ): + """Validate that all prices across commodity contexts share the same currency.""" + if not commodity_contexts: + return + + shared_currency_unit = None + + for context in commodity_contexts: + # Check all price fields in this context + for field_name, field_value in context.items(): + if field_name.endswith("_price") and field_value is not None: + # Get the price unit + if hasattr(field_value, "units"): + price_unit = str(field_value.units) + elif isinstance(field_value, ur.Quantity): + price_unit = str(field_value.units) + else: + continue + + # Extract currency from the price unit + # Price units are typically like "EUR/MWh" or "USD/MW" + # Split by "/" and take first part as currency + currency_unit = price_unit.split("/")[0].strip() + + if shared_currency_unit is None: + shared_currency_unit = str( + ur.Quantity(currency_unit).to_base_units().units + ) + elif not units_are_convertible(currency_unit, shared_currency_unit): + raise ValidationError( + "all prices in the flex-context must share the same currency unit" + ) + @validates_schema(pass_original=True) def check_prices(self, data: dict, original_data: dict, **kwargs): """Check assumptions about prices. @@ -437,59 +580,9 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): return data - def _try_to_convert_price_units(self, data: dict, original_data: dict): - """Convert price units to the same unit and scale if they can (incl. same currency).""" - - shared_currency_unit = None - previous_field_name = None - for field in self.declared_fields: - if field[-5:] == "price" and field in data: - price_field = self.declared_fields[field] - price_unit = price_field._get_unit(data[field]) - currency_unit = str( - ( - ur.Quantity(price_unit) / ur.Quantity(f"1{price_field.to_unit}") - ).units - ) - - if shared_currency_unit is None: - shared_currency_unit = str( - ur.Quantity(currency_unit).to_base_units().units - ) - previous_field_name = price_field.data_key - if not units_are_convertible(currency_unit, shared_currency_unit): - field_name = price_field.data_key - original_price_unit = price_field._get_original_unit( - original_data[field_name], data[field] - ) - error_message = f"Invalid unit. A valid unit would be, for example, '{shared_currency_unit + price_field.to_unit}' (this example uses '{shared_currency_unit}', because '{previous_field_name}' used that currency). However, you passed an incompatible price ('{original_price_unit}') for the '{field_name}' field." - if shared_currency_unit not in price_unit: - error_message += f" Also note that all prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." - raise ValidationError(error_message, field_name=field_name) - if shared_currency_unit is not None: - data["shared_currency_unit"] = shared_currency_unit - elif sensor := data.get("consumption_price_sensor"): - data["shared_currency_unit"] = self._to_currency_per_mwh(sensor.unit) - elif sensor := data.get("production_price_sensor"): - data["shared_currency_unit"] = self._to_currency_per_mwh(sensor.unit) - else: - data["shared_currency_unit"] = "EUR" - return data - - @staticmethod - def _to_currency_per_mwh(price_unit: str) -> str: - """Convert a price unit to a base currency used to express that price per MWh. - - >>> FlexContextSchema()._to_currency_per_mwh("EUR/MWh") - 'EUR' - >>> FlexContextSchema()._to_currency_per_mwh("EUR/kWh") - 'EUR' - """ - currency = str(ur.Quantity(price_unit + " * MWh").to_base_units().units) - return currency - EXAMPLE_UNIT_TYPES: Dict[str, list[str]] = { + "commodity": ["electricity", "gas"], "energy-price": ["EUR/MWh", "JPY/kWh", "USD/MWh", "and other currencies."], "power-price": ["EUR/kW", "JPY/kW", "USD/kW", "and other currencies."], "power": ["MW", "kW"], @@ -499,6 +592,26 @@ def _to_currency_per_mwh(price_unit: str) -> str: } UI_FLEX_CONTEXT_SCHEMA: Dict[str, Dict[str, Any]] = { + "aggregate-consumption": { + "default": None, + "description": rst_to_openapi(metadata.AGGREGATE_CONSUMPTION.description), + # todo: the field type is defined in asset_context.html in 3 places? + # "types": { + # "backend": "typeTwo", + # "ui": "A sensor which records the scheduled aggregate consumption.", + # }, + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, + "aggregate-production": { + "default": None, + "description": rst_to_openapi(metadata.AGGREGATE_PRODUCTION.description), + # todo: the field type is defined in asset_context.html in 3 places? + # "types": { + # "backend": "typeTwo", + # "ui": "A sensor which records the scheduled aggregate production.", + # }, + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, "consumption-price": { "default": None, # Refers to default value of the field "description": rst_to_openapi(metadata.CONSUMPTION_PRICE.description), @@ -593,11 +706,6 @@ def _to_currency_per_mwh(price_unit: str) -> str: "description": rst_to_openapi(metadata.AGGREGATE_POWER.description), "example-units": EXAMPLE_UNIT_TYPES["power"], }, - "gas-price": { - "default": None, - "description": rst_to_openapi(metadata.GAS_PRICE.description), - "example-units": EXAMPLE_UNIT_TYPES["energy-price"], - }, } UI_FLEX_MODEL_SCHEMA: Dict[str, Dict[str, Any]] = { @@ -774,12 +882,12 @@ def _to_currency_per_mwh(price_unit: str) -> str: }, "commodity": { "default": "electricity", - "description": rst_to_openapi(metadata.COMMODITY.description), + "description": rst_to_openapi(metadata.COMMODITY_FLEX_MODEL.description), "types": { "backend": "typeOne", "ui": "One fixed value only.", }, - "options": ["electricity", "gas"], + "example-units": EXAMPLE_UNIT_TYPES["commodity"], }, } @@ -929,7 +1037,17 @@ def ensure_sensor_or_asset(self, data, **kwargs): and data["sensor"].asset != data["asset"] ): raise ValidationError("Sensor does not belong to asset.") - if "sensor" not in data and "asset" not in data: + # if ( + # "state-of-charge" in data["sensor_flex_model"] + # and "asset" in data + # and data["sensor_flex_model"]["state-of-charge"].asset != data["asset"] + # ): + # raise ValidationError("Sensor does not belong to asset.") + if ( + "sensor" not in data + and "state-of-charge" not in data["sensor_flex_model"] + and "asset" not in data + ): raise ValidationError("Specify either a sensor or an asset.") @pre_load @@ -1010,7 +1128,7 @@ class AssetTriggerSchema(Schema): data_key="flex-model", load_default=[], ) - flex_context = fields.Dict( + flex_context = fields.Raw( required=False, data_key="flex-context", load_default={}, @@ -1029,6 +1147,37 @@ class AssetTriggerSchema(Schema): ), ) + @pre_load + def normalize_flex_context_format(self, data, **kwargs): + """Normalize flex_context to always be a dict. + + Accepts both: + - Single commodity dict: {"commodity": "electricity", ...} + - List of commodity dicts: [{"commodity": "electricity", ...}, {"commodity": "heat", ...}] + - MultiDict with multiple 'flex-context' entries (when JSON list is parsed by webargs) + + If a list is provided, it is wrapped under the 'commodities' field. + If a dict is provided, it is kept as-is. + This ensures downstream code always sees a dict structure. + """ + if "flex-context" in data: + raw_flex_context = data.get("flex-context") + + # Check if data is a MultiDict with multiple 'flex-context' entries + # This happens when JSON contains a list which webargs converts to multiple entries + if hasattr(data, "getlist"): + # MultiDict case - get all values for 'flex-context' + flex_contexts = data.getlist("flex-context") + if len(flex_contexts) > 1: + # Multiple commodities: wrap in a dict with commodity_contexts field + data["flex-context"] = {"commodities": flex_contexts} + # If only 1 entry, leave as-is (it's already a dict) + elif isinstance(raw_flex_context, list): + # Regular list case + data["flex-context"] = {"commodities": raw_flex_context} + # else: already a dict, leave as-is + return data + @validates_schema def check_flex_model_sensors(self, data, **kwargs): """Verify that the flex-model's sensors live under the asset for which a schedule is triggered.""" diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index bb7827e693..88d40b9ef9 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -27,6 +27,12 @@ def to_dict(self): # FLEX-CONTEXT +COMMODITY_FLEX_CONTEXT = MetaData( + description="""Commodity to which this part of the flex-context applies. +Defaults to ``"electricity"``. +""", + examples=["electricity", "gas"], +) INFLEXIBLE_DEVICE_SENSORS = MetaData( description="""Power sensors representing devices that are relevant, but not flexible in the timing of their demand/supply. For example, a sensor recording rooftop solar power that is connected behind the main meter, and whose production falls under the same contract as the flexible device(s) being scheduled. @@ -36,27 +42,49 @@ def to_dict(self): example=[3, 4], ) AGGREGATE_POWER = MetaData( - description="""Sensor used to record the aggregate power schedule of all flexible and inflexible devices involved when scheduling this asset.""", + description="""[Deprecated field] Sensor used to record the aggregate power schedule of all flexible and inflexible devices involved when scheduling this asset. +To avoid using the field, use ``aggregate-consumption`` or ``aggregate-production`` instead, which make clear the sign convention. +""", example={"sensor": 9}, ) +AGGREGATE_CONSUMPTION = MetaData( + description="""Sensor used to record the aggregate consumption schedule of all flexible and inflexible devices involved when scheduling this asset. + +The sign convention is determined by the key name, and is stored on the sensor itself using the ``consumption_is_positive`` attribute. + +Depending on which output sensors are defined: + +- **Only** ``aggregate-consumption`` **defined**: the full aggregate power schedule is stored on this sensor using the + consumption-positive sign convention (consumption positive, production negative). +- **Only** ``aggregate-production`` **defined**: the full aggregate power schedule is stored on the aggregate-production sensor + with the production-positive convention (production positive, consumption negative). +- **Both defined**: only the non-negative part of the aggregate schedule is stored on this sensor (zero for + time steps with net production), and only the non-positive part (sign-flipped) is stored on the + aggregate-production sensor. +""", + example={"sensor": 10}, +) +AGGREGATE_PRODUCTION = MetaData( + description="""Sensor used to record the aggregate production schedule of all flexible and inflexible devices involved when scheduling this asset. + +The sign convention is determined by the key name, and is stored on the sensor itself using the ``consumption_is_positive`` attribute. + +See the ``aggregate-consumption`` field for the full description of the split logic when both sensors are defined. +""", + example={"sensor": 11}, +) COMMITMENTS = MetaData( description="Prior commitments. Support for this field in the UI is still under further development, but you can find more information in :ref:`commitments`.", example=[], ) CONSUMPTION_PRICE = MetaData( - description="The electricity price applied to the site's aggregate consumption. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem. [#old_consumption_price_field]_", - example={"sensor": 5}, - # examples=[{"sensor": 5}, "0.29 EUR/kWh"], # todo: waiting for https://github.com/marshmallow-code/apispec/pull/999 + description="The commodity price (e.g. electricity price) applied to the site's aggregate consumption. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem. [#old_consumption_price_field]_", + examples=[{"sensor": 5}, "0.29 EUR/kWh"], ) PRODUCTION_PRICE = MetaData( - description="The electricity price applied to the site's aggregate production. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem, as long as the unit matches the ``consumption-price`` unit. [#old_production_price_field]_", + description="The commodity price (e.g. electricity price) applied to the site's aggregate production. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem, as long as the unit matches the ``consumption-price`` unit. [#old_production_price_field]_", example="0.12 EUR/kWh", ) -GAS_PRICE = MetaData( - description="The gas price applied to the site's aggregate gas consumption. Can be (a sensor recording) market prices, but also CO₂ intensity—whatever fits your optimization problem", - example={"sensor": 6}, - # example="0.09 EUR/kWh", -) SITE_POWER_CAPACITY = MetaData( description="""Maximum achievable power at the site's grid connection point, in either direction. Becomes a hard constraint in the optimization problem, which is especially suitable for physical limitations. [#asymmetric]_ [#minimum_capacity_overlap]_ @@ -189,12 +217,11 @@ def to_dict(self): # FLEX-MODEL -COMMODITY = MetaData( - description="""Commodity type for this storage flex-model. -Allowed values are ``electricity`` and ``gas``. -Defaults to ``electricity``. +COMMODITY_FLEX_MODEL = MetaData( + description="""Commodity on which this device acts. +Defaults to ``"electricity"``. """, - example="electricity", + examples=["electricity", "gas"], ) CONSUMPTION = MetaData( description="""Sensor used to record the scheduled power as seen from a consumption perspective. @@ -218,7 +245,7 @@ def to_dict(self): The sign convention is determined by the key name, and is stored on the sensor itself using the ``consumption_is_positive`` attribute. -See ``consumption`` for the full description of the split logic when both sensors are defined. +See the ``consumption`` field for the full description of the split logic when both sensors are defined. """, example={"sensor": 15}, ) @@ -310,14 +337,14 @@ def to_dict(self): example="90%", ) CHARGING_EFFICIENCY = MetaData( - description="""One-way conversion efficiency from electricity to the storage's state of charge. + description="""One-way conversion efficiency from the commodity (e.g. electricity) to the storage's state of charge. Can be a percentage, a ratio in the range [0,1], or a coefficient of performance (>1). Defaults to 100% (no conversion loss). """, example=".9", ) DISCHARGING_EFFICIENCY = MetaData( - description="""One-way conversion efficiency from the storage's state of charge to electricity. + description="""One-way conversion efficiency from the storage's state of charge to the commodity (e.g. electricity). Defaults to 100% (no conversion loss).""", example="90%", ) diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index 6e49cc9437..60d9da3448 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -20,7 +20,7 @@ from flexmeasures.data.schemas.scheduling import metadata from flexmeasures.data.schemas.sensors import ( SensorReference, - SensorReferenceSchema, + OutputSensorReferenceSchema, VariableQuantityField, ) from flexmeasures.utils.unit_utils import ( @@ -44,15 +44,6 @@ total=False, # not all are required (just value, which we can say in 3.11) ) -# Keys used by SensorReferenceSchema to carry source-filter options. -# Present as non-None values when the caller added a source filter. -_SENSOR_REFERENCE_SOURCE_FILTER_KEYS = ( - "source_types", - "exclude_source_types", - "sources", - "source_account", -) - class EfficiencyField(QuantityField): """Field that deserializes to a Quantity with % units. @@ -97,12 +88,18 @@ class StorageFlexModelSchema(Schema): metadata=dict(description="ID of the asset that is requested to be scheduled."), ) + commodity = fields.Str( + data_key="commodity", + load_default="electricity", + metadata=metadata.COMMODITY_FLEX_MODEL.to_dict(), + ) + consumption = fields.Nested( - SensorReferenceSchema, + OutputSensorReferenceSchema, metadata=metadata.CONSUMPTION.to_dict(), ) production = fields.Nested( - SensorReferenceSchema, + OutputSensorReferenceSchema, metadata=metadata.PRODUCTION.to_dict(), ) @@ -248,13 +245,6 @@ class StorageFlexModelSchema(Schema): validate=validate.Length(min=1), metadata=metadata.SOC_USAGE.to_dict(), ) - commodity = fields.Str( - required=False, - data_key="commodity", - load_default="electricity", - validate=OneOf(["electricity", "gas"]), - metadata=dict(description="Commodity label for this device/asset."), - ) def __init__( self, @@ -351,20 +341,6 @@ def validate_state_of_charge( "The `state-of-charge` field can only be a Sensor or a time series." ) - @validates("consumption") - def validate_consumption_has_no_source_filters(self, value: dict, **kwargs): - if isinstance(value, dict) and any( - value.get(key) is not None for key in _SENSOR_REFERENCE_SOURCE_FILTER_KEYS - ): - raise ValidationError("The `consumption` field cannot use source filters.") - - @validates("production") - def validate_production_has_no_source_filters(self, value: dict, **kwargs): - if isinstance(value, dict) and any( - value.get(key) is not None for key in _SENSOR_REFERENCE_SOURCE_FILTER_KEYS - ): - raise ValidationError("The `production` field cannot use source filters.") - @validates("asset") def validate_asset(self, asset: Asset, **kwargs): if self.sensor is not None and self.sensor.asset != asset: @@ -449,8 +425,8 @@ class DBStorageFlexModelSchema(Schema): Schema for flex-models stored in the db. Supports fixed quantities and sensor references, while disallowing time series specs. """ - consumption = fields.Nested(SensorReferenceSchema) - production = fields.Nested(SensorReferenceSchema) + consumption = fields.Nested(OutputSensorReferenceSchema) + production = fields.Nested(OutputSensorReferenceSchema) soc_min = VariableQuantityField( to_unit="MWh", @@ -592,20 +568,6 @@ def __init__(self, *args, **kwargs): for field in self.declared_fields } - @validates("consumption") - def validate_consumption_has_no_source_filters(self, value: dict, **kwargs): - if isinstance(value, dict) and any( - value.get(key) is not None for key in _SENSOR_REFERENCE_SOURCE_FILTER_KEYS - ): - raise ValidationError("The `consumption` field cannot use source filters.") - - @validates("production") - def validate_production_has_no_source_filters(self, value: dict, **kwargs): - if isinstance(value, dict) and any( - value.get(key) is not None for key in _SENSOR_REFERENCE_SOURCE_FILTER_KEYS - ): - raise ValidationError("The `production` field cannot use source filters.") - @validates_schema def forbid_time_series_specs(self, data: dict, **kwargs): """Do not allow time series specs for the flex-model fields saved in the db.""" diff --git a/flexmeasures/data/schemas/sensors.py b/flexmeasures/data/schemas/sensors.py index fc510adf05..72694534a6 100644 --- a/flexmeasures/data/schemas/sensors.py +++ b/flexmeasures/data/schemas/sensors.py @@ -979,11 +979,7 @@ def event_resolution(self) -> timedelta: return self.sensor.event_resolution -class SensorReferenceSchema(Schema): - """Sensor reference with optional source filters.""" - - class Meta: - description = "Sensor reference from which to look up a variable quantity." +class SharedSensorReferenceSchema(Schema): sensor = SensorIdField( required=True, @@ -991,6 +987,20 @@ class Meta: description="ID of the sensor on which the data is recorded.", ), ) + + +class OutputSensorReferenceSchema(SharedSensorReferenceSchema): + """Sensor reference for recording generated data.""" + + ... + + +class SensorReferenceSchema(SharedSensorReferenceSchema): + """Sensor reference with optional source filters.""" + + class Meta: + description = "Sensor reference from which to look up a variable quantity." + source_types = fields.List( fields.String(), load_default=None, diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 3d4450580e..aef5fbef9d 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -790,52 +790,6 @@ def test_flex_context_schema_rejects_filtered_aggregate_power( assert "cannot use source filters" in str(exc_info.value) -def test_storage_flex_model_schema_rejects_filtered_consumption( - setup_dummy_sensors, setup_sources, db -): - _, _, _, power_sensor = setup_dummy_sensors - seita_source = setup_sources["Seita"] - db.session.flush() - - for schema in [ - StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None), - DBStorageFlexModelSchema(), - ]: - with pytest.raises(ValidationError) as exc_info: - schema.load( - { - "consumption": { - "sensor": power_sensor.id, - "sources": [seita_source.id], - } - } - ) - assert "cannot use source filters" in str(exc_info.value) - - -def test_storage_flex_model_schema_rejects_filtered_production( - setup_dummy_sensors, setup_sources, db -): - _, _, _, power_sensor = setup_dummy_sensors - seita_source = setup_sources["Seita"] - db.session.flush() - - for schema in [ - StorageFlexModelSchema(start=datetime(2026, 6, 1), sensor=None), - DBStorageFlexModelSchema(), - ]: - with pytest.raises(ValidationError) as exc_info: - schema.load( - { - "production": { - "sensor": power_sensor.id, - "sources": [seita_source.id], - } - } - ) - assert "cannot use source filters" in str(exc_info.value) - - @pytest.mark.parametrize( ["flex_model", "fails"], [ @@ -953,22 +907,231 @@ def test_flex_model_schemas( for schema, fail in zip(schemas, fails): if fail: - with pytest.raises(ValidationError) as e_info: + with pytest.raises(ValidationError) as e_info: # noqa: F841 schema.load(flex_model) - for field_name, expected_message in fail.items(): - assert field_name in e_info.value.messages - if field_name in ["soc-gain", "soc-usage"]: - for index, message_list in e_info.value.messages[ - field_name - ].items(): - assert message_list[0] == expected_message[index][0] - else: - # Check all messages for the given field for the expected message - assert any( - [ - expected_message in message - for message in e_info.value.messages[field_name] - ] - ) + + +@pytest.mark.parametrize( + ["flex_context", "fails"], + [ + # Test aggregate-consumption field with sensor reference + ( + {"aggregate-consumption": {"sensor": "consumption-price in SEK/MWh"}}, + False, + ), + # Test aggregate-production field with sensor reference + ( + {"aggregate-production": {"sensor": "production-price in SEK/MWh"}}, + False, + ), + # Test both aggregate fields together + ( + { + "aggregate-consumption": {"sensor": "consumption-price in SEK/MWh"}, + "aggregate-production": {"sensor": "production-price in SEK/MWh"}, + }, + False, + ), + # Test that relax_constraints defaults to False in FlexContextSchema + ( + {"site-power-capacity": "1 MVA"}, + False, + ), + # Test breach prices moved to SharedSchema + ( + { + "consumption-breach-price": "100 EUR/MW", + "production-breach-price": "100 EUR/MW", + }, + False, + ), + # Test soc breach prices moved to SharedSchema + ( + { + "soc-minima-breach-price": "1000 EUR/MWh", + "soc-maxima-breach-price": "1000 EUR/MWh", + }, + False, + ), + ], +) +def test_shared_schema_fields_in_flex_context( + db, app, setup_site_capacity_sensor, setup_price_sensors, flex_context, fails +): + """Test that SharedSchema fields are accessible in FlexContextSchema.""" + schema = FlexContextSchema() + + # Replace sensor name with sensor ID + sensors_to_pick_from = {**setup_site_capacity_sensor, **setup_price_sensors} + for field_name, field_value in flex_context.items(): + if isinstance(field_value, dict) and "sensor" in field_value: + sensor_name = field_value["sensor"] + if sensor_name in sensors_to_pick_from: + flex_context[field_name]["sensor"] = sensors_to_pick_from[ + sensor_name + ].id + + check_schema_loads_data(schema=schema, data=flex_context, fails=fails) + + +@pytest.mark.parametrize( + ["commodity_contexts", "fails"], + [ + # Test single commodity pass validation and defaults relax_constraints to True + ( + [ + { + "commodity": "electricity", + "site-power-capacity": "1 MVA", + } + ], + False, + ), + # Likewise for multiple commodities, relax_constraints should default to True for each + ( + [ + { + "commodity": "electricity", + "site-power-capacity": "1 MVA", + }, + { + "commodity": "heat", + "site-power-capacity": "500 kW", + }, + ], + False, + ), + # Test aggregate fields in commodity context pass validation + ( + [ + { + "commodity": "electricity", + "aggregate-consumption": {"sensor": "consumption-price in SEK/MWh"}, + "aggregate-production": {"sensor": "production-price in SEK/MWh"}, + } + ], + False, + ), + # Test breach prices in commodity context pass validation + ( + [ + { + "commodity": "electricity", + "consumption-breach-price": "100 EUR/MW", + "production-breach-price": "100 EUR/MW", + } + ], + False, + ), + ], +) +def test_commodity_flex_context_defaults( + db, app, setup_site_capacity_sensor, setup_price_sensors, commodity_contexts, fails +): + """Test that CommodityFlexContextSchema has correct defaults, especially relax_constraints=True.""" + from flexmeasures.data.schemas.scheduling import CommodityFlexContextSchema + + # Replace sensor name with sensor ID + sensors_to_pick_from = {**setup_site_capacity_sensor, **setup_price_sensors} + for context in commodity_contexts: + for field_name, field_value in context.items(): + if isinstance(field_value, dict) and "sensor" in field_value: + sensor_name = field_value["sensor"] + if sensor_name in sensors_to_pick_from: + context[field_name]["sensor"] = sensors_to_pick_from[sensor_name].id + + # Test loading each commodity context + schema = CommodityFlexContextSchema() + for context in commodity_contexts: + if fails: + with pytest.raises(ValidationError) as e_info: + loaded = schema.load(context) + print(f"Returned error message: {e_info.value.messages}") else: - schema.load(flex_model) + loaded = schema.load(context) + # Verify relax_constraints defaults to True in CommodityFlexContextSchema + assert loaded.get("relax_constraints", True) is True + + +@pytest.mark.parametrize( + ["flex_context_listing", "fails"], + [ + # Test flex-context listing with mixed currencies should fail + ( + { + "commodities": [ + { + "commodity": "electricity", + "consumption-price": "1 EUR/MWh", + }, + { + "commodity": "heat", + "consumption-price": "1 USD/MWh", + }, + ] + }, + { + "commodities": "all prices in the flex-context must share the same currency unit" + }, + ), + # Test flex-context listing with same currencies should pass + ( + { + "commodities": [ + { + "commodity": "electricity", + "consumption-price": "1 EUR/MWh", + }, + { + "commodity": "heat", + "consumption-price": "2 EUR/MWh", + }, + ] + }, + False, + ), + # Test flex-context listing with breach prices sharing currency + ( + { + "commodities": [ + { + "commodity": "electricity", + "consumption-breach-price": "100 EUR/MW", + "production-breach-price": "10 cEUR/kW", + } + ] + }, + False, + ), + # Test flex-context listing with mixed breach price currencies should fail + ( + { + "commodities": [ + { + "commodity": "electricity", + "consumption-breach-price": "100 EUR/MW", + }, + { + "commodity": "heat", + "consumption-breach-price": "100 USD/MW", + }, + ] + }, + { + "commodities": "all prices in the flex-context must share the same currency unit" + }, + ), + ], +) +def test_flex_context_listing_shared_currency( + db, + app, + setup_site_capacity_sensor, + setup_price_sensors, + flex_context_listing, + fails, +): + """Test that flex-context listings enforce shared currency across commodities.""" + schema = FlexContextSchema() + + check_schema_loads_data(schema=schema, data=flex_context_listing, fails=fails) diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py index 5c7c7886fd..21fd7a75b4 100644 --- a/flexmeasures/data/services/scheduling.py +++ b/flexmeasures/data/services/scheduling.py @@ -800,7 +800,10 @@ def make_schedule( # noqa: C901 # Save any result that specifies a sensor to save it to for result in consumption_schedule: - if "sensor" not in result: + if rq_job and result["name"] == "commitment_costs": + rq_job.meta["scheduler_info"]["commitment_costs"] = result["data"] + continue + elif "sensor" not in result: continue # Ensure consumption_is_positive is set before resolving the sign. diff --git a/flexmeasures/data/tests/conftest.py b/flexmeasures/data/tests/conftest.py index c901aa9683..73820df4fa 100644 --- a/flexmeasures/data/tests/conftest.py +++ b/flexmeasures/data/tests/conftest.py @@ -229,7 +229,7 @@ def smart_building_types(app, fresh_db, setup_generic_asset_types_fresh_db): @pytest.fixture(scope="function") def smart_building(app, fresh_db, smart_building_types): """ - Topology of the sytstem: + Topology of the system: +---------+ | | @@ -414,6 +414,7 @@ def flex_description_sequential( "site-production-capacity": "2kW", "site-consumption-capacity": "5kW", # Cheap commitments that are not expected to affect the resulting schedule + # todo: CommitmentSchema should have a commodity field that defaults to electricity "commitments": [ { "name": "a sample commitment rewarding supply", diff --git a/flexmeasures/data/tests/test_scheduling_sequential.py b/flexmeasures/data/tests/test_scheduling_sequential.py index 53f67fa8c9..68d5689f2d 100644 --- a/flexmeasures/data/tests/test_scheduling_sequential.py +++ b/flexmeasures/data/tests/test_scheduling_sequential.py @@ -92,9 +92,12 @@ def test_create_sequential_jobs(db, app, flex_description_sequential, smart_buil work_on_rq(queue, handle_scheduling_exception) # Check that the jobs completed successfully - assert queued_jobs[0].get_status() == "finished" - assert deferred_jobs[0].get_status() == "finished" - assert deferred_jobs[1].get_status() == "finished" + ev_job = queued_jobs[0] + battery_job = deferred_jobs[0] + wrapup_job = deferred_jobs[1] + assert ev_job.get_status() == "finished" + assert battery_job.get_status() == "finished" + assert wrapup_job.get_status() == "finished" # check results ev_power = sensors["Test EV"].search_beliefs() @@ -131,14 +134,26 @@ def test_create_sequential_jobs(db, app, flex_description_sequential, smart_buil resolution = sensors["Test EV"].event_resolution.total_seconds() / 3600 ev_costs = (-ev_power * prices * resolution).sum().item() battery_costs = (-battery_power * prices * resolution).sum().item() - total_cost = ev_costs + battery_costs # Assert costs - assert ev_costs == 2.2375, f"EV cost should be 2.2375 €, got {ev_costs} €" + expected_ev_costs = 2.2375 + expected_battery_costs = -4.415 assert ( - battery_costs == -4.415 - ), f"Battery cost should be -4.415 €, got {battery_costs} €" - assert total_cost == -2.1775, f"Total cost should be -2.1775 €, got {total_cost} €" + ev_costs == expected_ev_costs + ), f"EV cost should be {expected_ev_costs} €, got {ev_costs} €" + assert ( + battery_costs == expected_battery_costs + ), f"Battery cost should be {expected_battery_costs} €, got {battery_costs} €" + + # todo: the ev job has scheduler_info and commitment costs, but the battery job has not + # Here, we want to check the electricity costs of the battery job, which takes into account the EV + # expected_total_cost = expected_ev_costs + expected_battery_costs + # np.testing.assert_approx_equal( + # battery_job.meta["scheduler_info"]["commitment_costs"]["electricity net energy"], + # expected_total_cost, + # 4, + # f"Reported costs should match our expectation", + # ) def test_create_sequential_jobs_fallback( diff --git a/flexmeasures/data/tests/test_scheduling_simultaneous.py b/flexmeasures/data/tests/test_scheduling_simultaneous.py index 6f307360c5..b5469d0e6b 100644 --- a/flexmeasures/data/tests/test_scheduling_simultaneous.py +++ b/flexmeasures/data/tests/test_scheduling_simultaneous.py @@ -11,7 +11,7 @@ def test_create_simultaneous_jobs( db, app, flex_description_sequential, smart_building, use_heterogeneous_resolutions ): - assets, sensors, _ = smart_building + assets, sensors, soc_sensors = smart_building queue = app.queues["scheduling"] start = pd.Timestamp("2015-01-03").tz_localize("Europe/Amsterdam") end = pd.Timestamp("2015-01-04").tz_localize("Europe/Amsterdam") @@ -20,6 +20,17 @@ def test_create_simultaneous_jobs( "module": "flexmeasures.data.models.planning.storage", "class": "StorageScheduler", } + flex_description_sequential["flex_model"][0]["sensor_flex_model"][ + "state-of-charge" + ] = {"sensor": soc_sensors["Test EV"].id} + if use_heterogeneous_resolutions: + flex_description_sequential["flex_model"][1]["sensor_flex_model"][ + "state-of-charge" + ] = {"sensor": soc_sensors["Test Battery 1h"].id} + else: + flex_description_sequential["flex_model"][1]["sensor_flex_model"][ + "state-of-charge" + ] = {"sensor": soc_sensors["Test Battery"].id} flex_description_sequential["start"] = start flex_description_sequential["end"] = end @@ -47,9 +58,17 @@ def test_create_simultaneous_jobs( ] ev_power = sensors["Test EV"].search_beliefs() - battery_power = sensors["Test Battery"].search_beliefs() + ev_soc = soc_sensors["Test EV"].search_beliefs() + if use_heterogeneous_resolutions: + battery_power = sensors["Test Battery 1h"].search_beliefs() + battery_soc = soc_sensors["Test Battery 1h"].search_beliefs() + else: + battery_power = sensors["Test Battery"].search_beliefs() + battery_soc = soc_sensors["Test Battery"].search_beliefs() assert ev_power.empty + assert ev_soc.empty assert battery_power.empty + assert battery_soc.empty # work tasks work_on_rq(queue) @@ -58,26 +77,33 @@ def test_create_simultaneous_jobs( job.perform() assert job.get_status() == "finished" - # Get power values + # Get power and SoC values ev_power = sensors["Test EV"].search_beliefs() assert ev_power.sources.unique()[0].model == "StorageScheduler" - ev_power = ev_power.droplevel([1, 2, 3]) + ev_soc = soc_sensors["Test EV"].search_beliefs() + assert ev_soc.sources.unique()[0].model == "StorageScheduler" if use_heterogeneous_resolutions: battery_power = sensors["Test Battery 1h"].search_beliefs() assert len(battery_power) == 24 + battery_soc = soc_sensors["Test Battery 1h"].search_beliefs() + assert len(battery_soc) == 97 else: battery_power = sensors["Test Battery"].search_beliefs() assert len(battery_power) == 96 + battery_soc = soc_sensors["Test Battery"].search_beliefs() + assert len(battery_soc) == 97 + + ev_power = ev_power.droplevel([1, 2, 3]) assert battery_power.sources.unique()[0].model == "StorageScheduler" battery_power = battery_power.droplevel([1, 2, 3]) - start_charging = start + pd.Timedelta(hours=8) - end_charging = start + pd.Timedelta(hours=10) - sensors["Test EV"].event_resolution # Check schedules - assert ( - ev_power.loc[start_charging:end_charging] != -0.005 - ).values.any(), "no charging at full device power capacity (5 kW) expected" + # start_charging = start + pd.Timedelta(hours=8) + # end_charging = start + pd.Timedelta(hours=10) - sensors["Test EV"].event_resolution + # assert ( + # ev_power.loc[start_charging:end_charging] != -0.005 + # ).values.any(), "no charging at full device power capacity (5 kW) expected, for target_no in (1, 2, 3): non_zero_target = flex_description_sequential["flex_model"][0][ "sensor_flex_model" @@ -96,7 +122,7 @@ def test_create_simultaneous_jobs( ] price_sensor = db.session.get(Sensor, price_sensor_id) prices = price_sensor.search_beliefs( - event_starts_after=start - pd.Timedelta(hours=1), event_ends_before=end + event_starts_after=start, event_ends_before=end ) prices = prices.droplevel([1, 2, 3]) prices.index = prices.index.tz_convert("Europe/Amsterdam") @@ -107,21 +133,32 @@ def test_create_simultaneous_jobs( total_cost = ev_costs + battery_costs # Define expected costs based on resolution - expected_ev_costs = 2.3125 - expected_battery_costs = -5.59 expected_total_cost = -3.2775 expected_ev_costs = 2.3125 expected_battery_costs = expected_total_cost - expected_ev_costs # Check costs - assert ( - round(total_cost, 4) == expected_total_cost - ), f"Total costs should be €{expected_total_cost}, got €{total_cost}" - - assert ( - round(ev_costs, 4) == expected_ev_costs - ), f"EV costs should be €{expected_ev_costs}, got €{ev_costs}" - - assert ( - round(battery_costs, 4) == expected_battery_costs - ), f"Battery costs should be €{expected_battery_costs}, got €{battery_costs}" + np.testing.assert_approx_equal( + total_cost, + expected_total_cost, + 4, + f"Total costs should be €{expected_total_cost}, got €{total_cost}", + ) + np.testing.assert_approx_equal( + ev_costs, + expected_ev_costs, + 4, + f"EV costs should be €{expected_ev_costs}, got €{ev_costs}", + ) + np.testing.assert_approx_equal( + battery_costs, + expected_battery_costs, + 4, + f"Battery costs should be €{expected_battery_costs}, got €{battery_costs}", + ) + np.testing.assert_approx_equal( + job.meta["scheduler_info"]["commitment_costs"]["electricity net energy"], + expected_total_cost, + 4, + "Reported costs should match our expectation", + ) diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 672bc6c985..9a6c7d9647 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -4556,74 +4556,54 @@ ], "additionalProperties": false }, + "OutputSensorReference": { + "type": "object", + "properties": { + "sensor": { + "type": "integer", + "description": "ID of the sensor on which the data is recorded." + } + }, + "required": [ + "sensor" + ], + "additionalProperties": false + }, "FlexContextOpenAPISchema": { "type": "object", "properties": { - "consumption-breach-price": { - "description": "This penalty value is used to discourage the violation of the consumption-capacity constraint in the flex-model.\nIt effectively treats the capacity as a soft constraint, allowing the scheduler to exceed it when necessary but with a high cost.\nThe scheduler will attempt to minimize this cost.\nIt must use the same currency as the other price settings and cannot be negative.\n", - "example": "10 EUR/kW", - "$ref": "#/components/schemas/VariableQuantityOpenAPI" - }, - "production-breach-price": { - "description": "This penalty value is used to discourage the violation of the production-capacity constraint in the flex-model.\nIt effectively treats the capacity as a soft constraint, allowing the scheduler to exceed it when necessary but with a high cost.\nThe scheduler will attempt to minimize this cost.\nIt must use the same currency as the other price settings and cannot be negative.\n", - "example": "10 EUR/kW", - "$ref": "#/components/schemas/VariableQuantityOpenAPI" + "commodity": { + "type": "string", + "default": "electricity", + "description": "Commodity to which this part of the flex-context applies.\nDefaults to \"electricity\".\n", + "examples": [ + "electricity", + "gas" + ] }, - "soc-minima-breach-price": { - "description": "This penalty value is used to discourage the violation of soc-minima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-minima become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed.\n", - "example": "120 EUR/kWh", + "consumption-price": { + "description": "The commodity price (e.g. electricity price) applied to the site's aggregate consumption. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem.", + "examples": [ + { + "sensor": 5 + }, + "0.29 EUR/kWh" + ], "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, - "soc-maxima-breach-price": { - "description": "This penalty value is used to discourage the violation of soc-maxima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-maxima become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed.\n", - "example": "120 EUR/kWh", + "production-price": { + "description": "The commodity price (e.g. electricity price) applied to the site's aggregate production. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem, as long as the unit matches the consumption-price unit.", + "example": "0.12 EUR/kWh", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, - "relax-constraints": { - "type": "boolean", - "default": false, - "description": "If True (default is False), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nWe recommend to set this field to True to enable the default prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", - "example": true - }, - "relax-soc-constraints": { - "type": "boolean", - "default": false, - "description": "If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", - "example": true - }, - "relax-capacity-constraints": { - "type": "boolean", - "default": false, - "description": "If True, avoids breaching the desired device consumption/production capacity as a relaxed constraint.", - "example": true - }, - "relax-site-capacity-constraints": { - "type": "boolean", - "default": false, - "description": "If True, avoids breaching the site consumption/production capacity as a relaxed constraint.", - "example": true - }, "site-power-capacity": { "description": "Maximum achievable power at the site's grid connection point, in either direction.\nBecomes a hard constraint in the optimization problem, which is especially suitable for physical limitations.\n", "example": "45kVA", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, - "consumption-price-sensor": { - "type": "integer" - }, - "production-price-sensor": { - "type": "integer" - }, - "consumption-price": { - "description": "The electricity price applied to the site's aggregate consumption. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem.", - "example": { - "sensor": 5 - }, - "$ref": "#/components/schemas/VariableQuantityOpenAPI" - }, - "production-price": { - "description": "The electricity price applied to the site's aggregate production. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem, as long as the unit matches the consumption-price unit.", - "example": "0.12 EUR/kWh", + "site-consumption-capacity": { + "description": "Maximum consumption power at the site's grid connection point.\nIf site-power-capacity is defined, the minimum between the site-power-capacity and site-consumption-capacity will be used.\nIf a site-consumption-breach-price is defined, the site-consumption-capacity becomes a soft constraint in the optimization problem.\nOtherwise, it becomes a hard constraint.\n", + "example": "45kW", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "site-production-capacity": { @@ -4631,11 +4611,6 @@ "example": "0kW", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, - "site-consumption-capacity": { - "description": "Maximum consumption power at the site's grid connection point.\nIf site-power-capacity is defined, the minimum between the site-power-capacity and site-consumption-capacity will be used.\nIf a site-consumption-breach-price is defined, the site-consumption-capacity becomes a soft constraint in the optimization problem.\nOtherwise, it becomes a hard constraint.\n", - "example": "45kW", - "$ref": "#/components/schemas/VariableQuantityOpenAPI" - }, "site-consumption-breach-price": { "description": "This penalty value is used to discourage the violation of the site-consumption-capacity constraint in the flex-context.\nIt effectively treats the capacity as a soft constraint, allowing the scheduler to exceed it when necessary but with a high cost.\nThe scheduler will attempt to minimize this cost.\nIt must use the same currency as the other price settings and cannot be negative.\nThe field may define (a sensor recording) contractual penalties, or a theoretical penalty influencing how badly breaches should be avoided.\n", "example": "1000 EUR/kW", @@ -4670,6 +4645,50 @@ "example": "260 EUR/MW", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, + "consumption-breach-price": { + "description": "This penalty value is used to discourage the violation of the consumption-capacity constraint in the flex-model.\nIt effectively treats the capacity as a soft constraint, allowing the scheduler to exceed it when necessary but with a high cost.\nThe scheduler will attempt to minimize this cost.\nIt must use the same currency as the other price settings and cannot be negative.\n", + "example": "10 EUR/kW", + "$ref": "#/components/schemas/VariableQuantityOpenAPI" + }, + "production-breach-price": { + "description": "This penalty value is used to discourage the violation of the production-capacity constraint in the flex-model.\nIt effectively treats the capacity as a soft constraint, allowing the scheduler to exceed it when necessary but with a high cost.\nThe scheduler will attempt to minimize this cost.\nIt must use the same currency as the other price settings and cannot be negative.\n", + "example": "10 EUR/kW", + "$ref": "#/components/schemas/VariableQuantityOpenAPI" + }, + "soc-minima-breach-price": { + "description": "This penalty value is used to discourage the violation of soc-minima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-minima become hard constraints, which means that any infeasible state-of-charge minima would prevent a complete schedule from being computed.\n", + "example": "120 EUR/kWh", + "$ref": "#/components/schemas/VariableQuantityOpenAPI" + }, + "soc-maxima-breach-price": { + "description": "This penalty value is used to discourage the violation of soc-maxima constraints in the flex-model, which the scheduler will attempt to minimize.\nIt must use the same currency as the other price settings and cannot be negative.\nWhile it's an internal nudge to steer the scheduler\u2014and doesn't represent a real-life cost\u2014it should still be chosen in proportion to the actual energy prices at your site.\nIf it's too high, it will overly dominate other constraints; if it's too low, it will have no effect.\nWithout this value, the soc-maxima become hard constraints, which means that any infeasible state-of-charge maxima would prevent a complete schedule from being computed.\n", + "example": "120 EUR/kWh", + "$ref": "#/components/schemas/VariableQuantityOpenAPI" + }, + "relax-constraints": { + "type": "boolean", + "default": true, + "description": "If True (default is False), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nWe recommend to set this field to True to enable the default prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", + "example": true + }, + "relax-soc-constraints": { + "type": "boolean", + "default": false, + "description": "If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", + "example": true + }, + "relax-capacity-constraints": { + "type": "boolean", + "default": false, + "description": "If True, avoids breaching the desired device consumption/production capacity as a relaxed constraint.", + "example": true + }, + "relax-site-capacity-constraints": { + "type": "boolean", + "default": false, + "description": "If True, avoids breaching the site consumption/production capacity as a relaxed constraint.", + "example": true + }, "commitments": { "description": "Prior commitments. Support for this field in the UI is still under further development, but you can find more information in the docs.", "example": [], @@ -4689,19 +4708,19 @@ "type": "integer" } }, - "aggregate-power": { - "description": "Sensor used to record the aggregate power schedule of all flexible and inflexible devices involved when scheduling this asset.", + "aggregate-consumption": { + "description": "Sensor used to record the aggregate consumption schedule of all flexible and inflexible devices involved when scheduling this asset.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nDepending on which output sensors are defined:\n\n- Only aggregate-consumption defined: the full aggregate power schedule is stored on this sensor using the\n consumption-positive sign convention (consumption positive, production negative).\n- Only aggregate-production defined: the full aggregate power schedule is stored on the aggregate-production sensor\n with the production-positive convention (production positive, consumption negative).\n- Both defined: only the non-negative part of the aggregate schedule is stored on this sensor (zero for\n time steps with net production), and only the non-positive part (sign-flipped) is stored on the\n aggregate-production sensor.\n", "example": { - "sensor": 9 + "sensor": 10 }, - "$ref": "#/components/schemas/SensorReference" + "$ref": "#/components/schemas/OutputSensorReference" }, - "gas-price": { - "description": "The gas price applied to the site's aggregate gas consumption. Can be (a sensor recording) market prices, but also CO\u2082 intensity\u2014whatever fits your optimization problem", + "aggregate-production": { + "description": "Sensor used to record the aggregate production schedule of all flexible and inflexible devices involved when scheduling this asset.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nSee the aggregate-consumption field for the full description of the split logic when both sensors are defined.\n", "example": { - "sensor": 6 + "sensor": 11 }, - "$ref": "#/components/schemas/VariableQuantityOpenAPI" + "$ref": "#/components/schemas/OutputSensorReference" } }, "additionalProperties": false @@ -6070,19 +6089,28 @@ "StorageFlexModelSchemaOpenAPI": { "type": "object", "properties": { + "commodity": { + "type": "string", + "default": "electricity", + "description": "Commodity on which this device acts.\nDefaults to \"electricity\".\n", + "examples": [ + "electricity", + "gas" + ] + }, "consumption": { "description": "Sensor used to record the scheduled power as seen from a consumption perspective.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nDepending on which output sensors are defined:\n\n- Only consumption defined: the full power schedule is stored on this sensor using the\n consumption-positive sign convention (consumption positive, production negative).\n- Only production defined: the full power schedule is stored on the production sensor\n with the production-positive convention (production positive, consumption negative).\n- Both defined: only the non-negative part of the schedule is stored on this sensor (zero for\n time steps with net production), and only the non-positive part (sign-flipped) is stored on the\n production sensor.\n", "example": { "sensor": 14 }, - "$ref": "#/components/schemas/SensorReference" + "$ref": "#/components/schemas/OutputSensorReference" }, "production": { - "description": "Sensor used to record the scheduled power as seen from a production perspective.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nSee consumption for the full description of the split logic when both sensors are defined.\n", + "description": "Sensor used to record the scheduled power as seen from a production perspective.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nSee the consumption field for the full description of the split logic when both sensors are defined.\n", "example": { "sensor": 15 }, - "$ref": "#/components/schemas/SensorReference" + "$ref": "#/components/schemas/OutputSensorReference" }, "soc-at-start": { "type": "string", @@ -6182,12 +6210,12 @@ "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "charging-efficiency": { - "description": "One-way conversion efficiency from electricity to the storage's state of charge.\nCan be a percentage, a ratio in the range [0,1], or a coefficient of performance (>1).\nDefaults to 100% (no conversion loss).\n", + "description": "One-way conversion efficiency from the commodity (e.g. electricity) to the storage's state of charge.\nCan be a percentage, a ratio in the range [0,1], or a coefficient of performance (>1).\nDefaults to 100% (no conversion loss).\n", "example": ".9", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "discharging-efficiency": { - "description": "One-way conversion efficiency from the storage's state of charge to electricity.\nDefaults to 100% (no conversion loss).", + "description": "One-way conversion efficiency from the storage's state of charge to the commodity (e.g. electricity).\nDefaults to 100% (no conversion loss).", "example": "90%", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, @@ -6227,15 +6255,6 @@ ], "items": {} }, - "commodity": { - "type": "string", - "default": "electricity", - "enum": [ - "electricity", - "gas" - ], - "description": "Commodity label for this device/asset." - }, "sensor": { "type": "integer", "description": "ID of the device's power sensor." @@ -6270,16 +6289,21 @@ "example": "PT2H", "format": "duration" }, - "flex_model": { + "flex-model": { "type": "array", + "default": [], + "description": "Flex-model per device (identified by `sensor`). The flex-model validation is handled by the scheduler. What follows is the schema used by the `StorageScheduler`.", "items": { - "description": "Flex-model per device (identified by `sensor`). The flex-model validation is handled by the scheduler. What follows is the schema used by the `StorageScheduler`.", "$ref": "#/components/schemas/StorageFlexModelSchemaOpenAPI" } }, "flex-context": { - "description": "The flex-context is validated according to the scheduler's `FlexContextSchema`.", - "$ref": "#/components/schemas/FlexContextOpenAPISchema" + "type": "array", + "default": [], + "description": "Flex-context per commodity. The flex-context is validated according to the scheduler's `FlexContextSchema`.", + "items": { + "$ref": "#/components/schemas/FlexContextOpenAPISchema" + } }, "sequential": { "type": "boolean", @@ -6292,7 +6316,6 @@ } }, "required": [ - "flex-context", "start" ], "additionalProperties": false diff --git a/flexmeasures/ui/templates/assets/asset_context.html b/flexmeasures/ui/templates/assets/asset_context.html index 5c84a5a607..09a1348dfb 100644 --- a/flexmeasures/ui/templates/assets/asset_context.html +++ b/flexmeasures/ui/templates/assets/asset_context.html @@ -380,7 +380,7 @@
Edit {{ asset.name }}'s flex-context
setTimeout(() => { const card = document.getElementById(`${fieldName}-control`); - const isOnlySensorField = (fieldName === "inflexible-device-sensors" || fieldName === "consumption-price" || fieldName === "production-price" || fieldName === "aggregate-power"); + const isOnlySensorField = (fieldName === "inflexible-device-sensors" || fieldName === "consumption-price" || fieldName === "production-price" || fieldName === "aggregate-power" || fieldName === "aggregate-consumption" || fieldName === "aggregate-production"); card.classList.add('border-on-click'); // Add border to the clicked card flexOptionsContainer.appendChild(renderFlexInputOptions(fieldName, isOnlySensorField)); handleFlexSelectChange(fieldName); @@ -608,7 +608,7 @@
Edit {{ asset.name }}'s flex-context
document.querySelectorAll(".card-highlight").forEach(el => el.classList.remove("border-on-click")); // Remove border from all cards card.classList.add("border-on-click"); // Add border to the clicked card handleFlexSelectChange(fieldName); - const isOnlySensorField = (fieldName === "inflexible-device-sensors" || fieldName === "consumption-price" || fieldName === "production-price" || fieldName === "aggregate-power"); + const isOnlySensorField = (fieldName === "inflexible-device-sensors" || fieldName === "consumption-price" || fieldName === "production-price" || fieldName === "aggregate-power" || fieldName === "aggregate-consumption" || fieldName === "aggregate-production"); flexOptionsContainer.appendChild(renderFlexInputOptions(fieldName, (isOnlySensorField))); setActiveCard(card); // Store active card in local storage @@ -720,7 +720,7 @@
Edit {{ asset.name }}'s flex-context
if (storedActiveCard && activeCard()) { const flexId = (activeCard() ? activeCard().id : null).slice(0, -8); if (assetFlexContext[storedActiveCard]) { - const isOnlySensorField = (flexId === "inflexible-device-sensors" || flexId === "consumption-price" || flexId === "production-price" || flexId === "aggregate-power"); + const isOnlySensorField = (flexId === "inflexible-device-sensors" || flexId === "consumption-price" || flexId === "production-price" || flexId === "aggregate-power" || flexId === "aggregate-consumption" || flexId === "aggregate-production"); setTimeout(() => { flexOptionsContainer.innerHTML = ""; flexOptionsContainer.appendChild(renderFlexInputOptions(flexId, isOnlySensorField)); diff --git a/flexmeasures/ui/tests/test_utils.py b/flexmeasures/ui/tests/test_utils.py index 09a68ff005..eff2114fc6 100644 --- a/flexmeasures/ui/tests/test_utils.py +++ b/flexmeasures/ui/tests/test_utils.py @@ -79,6 +79,7 @@ def test_ui_flexcontext_schema(): "relax-site-capacity-constraints", "consumption-price-sensor", "production-price-sensor", + "commodities", # todo: https://github.com/FlexMeasures/flexmeasures/issues/2230 ] schema_keys = [] @@ -91,7 +92,10 @@ def test_ui_flexcontext_schema(): assert ( schema_keys == ui_flexcontext_schema_fields - ), "If this test fails, you may have added a new flex-context field, but forgot about UI support." + ), "If this fails, you may have added a new flex-context field, but forgot about UI support." + assert ( + schema_keys - set(exclude_fields) == schema_keys + ), "If this fails, you may have added UI support for a new flex-context field, but forgot to remove it from exclude_fields." def test_ui_flexmodel_schema(): diff --git a/flexmeasures/utils/coding_utils.py b/flexmeasures/utils/coding_utils.py index 1ed43ce1f2..894846aa10 100644 --- a/flexmeasures/utils/coding_utils.py +++ b/flexmeasures/utils/coding_utils.py @@ -2,6 +2,7 @@ from __future__ import annotations +from typing import Any import functools import time import inspect @@ -11,6 +12,34 @@ from flask import current_app +def merge_or_append( + item: dict[str, Any], + items: list[dict[str, Any]], + match_key: str, + match_value: str | None = None, +) -> None: + """Merge `item` into the first dictionary in `items` with the same value for `key`, preserving its position in the sequence. + + Values from `item` take precedence when keys overlap. If no matching + dictionary is found, `item` is appended to the end of `items`. + + :param item: The dictionary to merge or append. + :param items: A mutable sequence of dictionaries to update. + :param match_key: The dictionary key used to determine whether two items match. + :param match_value: The value used to determine whether two items match. + + :returns: None. The `items` sequence is modified in place. + """ + match_value = item.get(match_key) or match_value + + for i, existing in enumerate(items): + if existing.get(match_key) == match_value: + items[i] = existing | item + return + + items.append(item) + + def delete_key_recursive(value, key): """Delete key in a multilevel dictionary""" if isinstance(value, dict): diff --git a/tests/documentation/test_schemas.py b/tests/documentation/test_schemas.py index 0a85803103..fab2946647 100644 --- a/tests/documentation/test_schemas.py +++ b/tests/documentation/test_schemas.py @@ -13,6 +13,8 @@ "RELAX_CAPACITY_CONSTRAINTS", "RELAX_SITE_CAPACITY_CONSTRAINTS", "RELAX_SOC_CONSTRAINTS", + "COMMODITY_FLEX_CONTEXT", # Documented as "commodity" in flex-context section + "COMMODITY_FLEX_MODEL", # Documented as "commodity" in flex-model section } From 75392b9b0619b2b649fcf6bbdbe2aa5927490416 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 17:50:21 +0200 Subject: [PATCH 101/136] docs: changelog entry Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 108a647e0f..5cc669e98b 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -17,7 +17,7 @@ New features * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 `_] * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 `_] * Support multiple feeders to a shared storage [see `PR #2001 `_ ] -* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #2172 `_ and `PR #2235 `_] +* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_ and `PR #2235 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] Infrastructure / Support From b3e12bd564db2977185a12b26c7768a7ad3d477e Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:51:48 +0200 Subject: [PATCH 102/136] fix(tests): unique public asset names in commitment tests to satisfy generic_asset_public_root_name_key Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index d9ffc8a2dc..a58e4919b9 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -552,7 +552,7 @@ def test_two_flexible_assets_with_commodity(app, db): # ---- assets battery = GenericAsset( - name="Battery", + name="Battery (two flexible assets with commodity)", generic_asset_type=battery_type, attributes={"energy-capacity": "100 kWh"}, ) @@ -706,13 +706,13 @@ def test_mixed_gas_and_electricity_assets(app, db): resolution = pd.Timedelta("1h") battery = GenericAsset( - name="Battery", + name="Battery (mixed gas and electricity)", generic_asset_type=battery_type, attributes={"energy-capacity": "100 kWh"}, ) gas_boiler = GenericAsset( - name="Gas Boiler", + name="Gas Boiler (mixed gas and electricity)", generic_asset_type=boiler_type, ) From 107ab37ba8bdfd00c56e7d56e5b7885a0746facf Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:58:00 +0200 Subject: [PATCH 103/136] fix: consolidate currency validation across commodity flex-contexts Replace the broken split("/")-based cross-commodity currency validator (which skipped Sensor-valued prices and never actually normalized units correctly) with a robust comparison of each commodity context's already normalized shared_currency_unit, computed by the inherited _try_to_convert_price_units validator. Also cross-check the top-level flex-context's currency against per-commodity contexts, and remove the now-redundant relax-constraints override on CommodityFlexContextSchema. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 63 +++++++++---------- 1 file changed, 31 insertions(+), 32 deletions(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 5bd8021e6a..2a4156de99 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -393,13 +393,6 @@ class CommodityFlexContextSchema(SharedSchema): metadata=metadata.COMMODITY_FLEX_CONTEXT.to_dict(), ) - # For flex-context listings (per commodity), default relax_constraints to True - relax_constraints = fields.Bool( - data_key="relax-constraints", - load_default=True, - metadata=metadata.RELAX_CONSTRAINTS.to_dict(), - ) - def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) @@ -452,37 +445,28 @@ def validate_aggregate_power_is_sensor( def validate_commodity_contexts_shared_currency( self, commodity_contexts: list[dict], **kwargs ): - """Validate that all prices across commodity contexts share the same currency.""" + """Validate that all prices across commodity contexts share the same currency. + + Each commodity context already computed its own normalized ``shared_currency_unit`` + (a base-unit currency string, e.g. "EUR") via the inherited + ``_try_to_convert_price_units`` schema-level validator. We simply compare those. + """ if not commodity_contexts: return shared_currency_unit = None for context in commodity_contexts: - # Check all price fields in this context - for field_name, field_value in context.items(): - if field_name.endswith("_price") and field_value is not None: - # Get the price unit - if hasattr(field_value, "units"): - price_unit = str(field_value.units) - elif isinstance(field_value, ur.Quantity): - price_unit = str(field_value.units) - else: - continue - - # Extract currency from the price unit - # Price units are typically like "EUR/MWh" or "USD/MW" - # Split by "/" and take first part as currency - currency_unit = price_unit.split("/")[0].strip() - - if shared_currency_unit is None: - shared_currency_unit = str( - ur.Quantity(currency_unit).to_base_units().units - ) - elif not units_are_convertible(currency_unit, shared_currency_unit): - raise ValidationError( - "all prices in the flex-context must share the same currency unit" - ) + context_currency_unit = context.get("shared_currency_unit") + if context_currency_unit is None: + continue + if shared_currency_unit is None: + shared_currency_unit = context_currency_unit + elif not units_are_convertible(context_currency_unit, shared_currency_unit): + raise ValidationError( + "all prices in the flex-context must share the same currency unit" + f" (found both '{shared_currency_unit}' and '{context_currency_unit}')" + ) @validates_schema(pass_original=True) def check_prices(self, data: dict, original_data: dict, **kwargs): @@ -539,6 +523,21 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): data = self._try_to_convert_price_units(data, original_data) shared_currency = ur.Quantity(data["shared_currency_unit"]) + # Also check that top-level prices share their currency with any per-commodity contexts + for context in data.get("commodity_contexts", []) or []: + context_currency_unit = context.get("shared_currency_unit") + if context_currency_unit is None: + continue + if not units_are_convertible( + context_currency_unit, data["shared_currency_unit"] + ): + raise ValidationError( + "all prices in the flex-context must share the same currency unit" + f" (found both '{data['shared_currency_unit']}' at the top level and" + f" '{context_currency_unit}' in a commodity context)", + field_name="commodities", + ) + # Fill in default soc breach prices when asked to relax SoC constraints, unless already set explicitly. if ( data["relax_soc_constraints"] From 4d7eef560d09d13fe49dad66a8e63bd490a68cb9 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:58:08 +0200 Subject: [PATCH 104/136] feat!: default top-level relax-constraints to True Flip the default of SharedSchema.relax_constraints (used by the top-level FlexContextSchema) to True, matching the default already used for per-commodity CommodityFlexContextSchema entries. This changes behaviour for existing single-commodity users who did not explicitly set relax-constraints: constraints are now relaxed (with default breach prices) by default. See changelog for details. Signed-off-by: F.N. Claessen --- flexmeasures/data/schemas/scheduling/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 2a4156de99..3494ca1ace 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -269,7 +269,7 @@ class SharedSchema(Schema): # Relaxation fields relax_constraints = fields.Bool( data_key="relax-constraints", - load_default=False, + load_default=True, metadata=metadata.RELAX_CONSTRAINTS.to_dict(), ) relax_soc_constraints = fields.Bool( From cec885b2789f3abae502dcdbf7804b854206bc09 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:58:39 +0200 Subject: [PATCH 105/136] fix: support per-commodity inflexible-device-sensors and all-gas portfolios _prepare() only read the top-level inflexible-device-sensors key, so sensors declared inside a commodity context got no device constraints and did not count against that commodity's site capacity, even though _compute_commodity_aggregate_schedules already assumed they were enumerated. Collect and constrain each commodity context's inflexible sensors too, appended after the top-level (electricity) ones, in listed order, matching that enumeration. Extract the shared commodity-to-device-index reconstruction into _reconstruct_commodity_to_devices()/_electricity_device_indices() to avoid duplicating this logic. Also switch commodity_to_devices["electricity"] += ... to setdefault(), so a flex-model without any electricity device (e.g. all-gas) no longer raises a KeyError. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 99 ++++++++++++++------ 1 file changed, 71 insertions(+), 28 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index af1626a89a..40c4c598a2 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -353,13 +353,30 @@ def device_list_series( self.flex_context.get("inflexible_device_sensors", []) ) num_flexible_devices = len(flex_model) - commodity_to_devices["electricity"] += list( + commodity_to_devices.setdefault("electricity", []).extend( range( number_flexible_devices, number_flexible_devices + number_inflexible_devices, ) ) + # Per-commodity inflexible-device-sensors, enumerated after the top-level + # (electricity) inflexible devices, in the order the commodity contexts are + # given. This mirrors the enumeration that + # `_compute_commodity_aggregate_schedules` already assumes. + commodity_context_inflexible_sensors: list[Sensor] = [] + num_devices = number_flexible_devices + number_inflexible_devices + for commodity_context in self.flex_context.get("commodity_contexts", []): + commodity = commodity_context["commodity"] + commodity_inflexible_sensors = commodity_context.get( + "inflexible_device_sensors", [] + ) + commodity_to_devices.setdefault(commodity, []).extend( + range(num_devices, num_devices + len(commodity_inflexible_sensors)) + ) + commodity_context_inflexible_sensors.extend(commodity_inflexible_sensors) + num_devices += len(commodity_inflexible_sensors) + commodity_contexts = self._get_commodity_contexts() price_frames_by_commodity = {} @@ -724,12 +741,20 @@ def device_list_series( ) commitments.append(commitment) - # Set up device constraints: scheduled flexible devices for this EMS (from index 0 to D-1), plus the forecasted inflexible devices (at indices D to n). + # Set up device constraints: scheduled flexible devices for this EMS (from index 0 to D-1), + # plus the forecasted top-level (electricity) inflexible devices, plus each commodity + # context's own inflexible devices, in that order. device_constraints = [ initialize_df(StorageScheduler.COLUMNS, start, end, resolution) - for i in range(num_flexible_devices + len(inflexible_device_sensors)) + for i in range( + num_flexible_devices + + len(inflexible_device_sensors) + + len(commodity_context_inflexible_sensors) + ) ] - for i, inflexible_sensor in enumerate(inflexible_device_sensors): + for i, inflexible_sensor in enumerate( + inflexible_device_sensors + commodity_context_inflexible_sensors + ): device_constraints[i + num_flexible_devices]["derivative equals"] = ( get_power_values( query_window=(start, end), @@ -2199,30 +2224,17 @@ def _build_consumption_production_schedules( ) return schedules - def _compute_commodity_aggregate_schedules( - self, - storage_schedule: dict, - ems_schedule: pd.DataFrame, - # sensors: list[Sensor | None], - ) -> None: - """Compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors. - - This method populates the storage_schedule dict with aggregate schedules for each commodity - that defines aggregate-consumption and/or aggregate-production sensors in its commodity context. - - The sign convention and split logic follows the same pattern as _build_consumption_production_schedules: - - Only aggregate-consumption defined: full aggregate schedule (consumption +, production -) - - Only aggregate-production defined: full aggregate schedule (consumption +, production -) - (sign will be flipped by make_schedule based on consumption_is_positive=False) - - Both defined: consumption sensor gets non-negative part, production sensor gets non-positive part - (sign will be flipped for production by make_schedule) + def _reconstruct_commodity_to_devices(self) -> dict[str, list[int]]: + """Reconstruct the mapping of commodity -> device indices as enumerated by `_prepare()`. - For backwards compatibility, when no commodity_contexts are defined, all devices are treated - as electricity devices and use the top-level flex-context fields. + Device enumeration order: + 1. flexible devices (from the flex-model), in order, + 2. top-level (electricity) inflexible-device-sensors, in order, + 3. each commodity context's own inflexible-device-sensors, in the order the + commodity contexts are given. - :param storage_schedule: Dict to populate with aggregate schedules (will be modified in-place) - :param ems_schedule: DataFrame of per-device power schedules in MW (consumption positive) - :param sensors: List of sensors corresponding to device indices + This mirrors `_prepare()`'s device enumeration exactly, so the returned device + indices line up with entries of `ems_schedule` / `device_constraints`. """ # Get the device models to reconstruct commodity_to_devices mapping flex_model = getattr(self, "_device_models", None) @@ -2237,7 +2249,7 @@ def _compute_commodity_aggregate_schedules( flex_model = [flex_model] # Reconstruct commodity_to_devices mapping - commodity_to_devices = {} + commodity_to_devices: dict[str, list[int]] = {} for d, flex_model_d in enumerate(flex_model): commodity = flex_model_d.get("commodity", "electricity") commodity_to_devices.setdefault(commodity, []).append(d) @@ -2247,7 +2259,7 @@ def _compute_commodity_aggregate_schedules( # - top-level inflexible-device-sensors go to electricity (backwards compat), # - then each commodity context's own inflexible-device-sensors are appended to # that commodity, in the order the commodity contexts are given. - # Without step 2 below, a commodity's inflexible demand (e.g. a heat load) is left + # Without this, a commodity's inflexible demand (e.g. a heat load) is left # out of its aggregate schedule, so an aggregate-consumption sensor only reflects # the flexible devices of that commodity. inflexible_device_sensors = self.flex_context.get( @@ -2276,6 +2288,37 @@ def _compute_commodity_aggregate_schedules( ) num_devices += len(commodity_inflexible_device_sensors) + return commodity_to_devices + + def _electricity_device_indices(self) -> list[int]: + """Return the device indices (flexible and inflexible) belonging to the electricity commodity.""" + return self._reconstruct_commodity_to_devices().get("electricity", []) + + def _compute_commodity_aggregate_schedules( + self, + storage_schedule: dict, + ems_schedule: pd.DataFrame, + ) -> None: + """Compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors. + + This method populates the storage_schedule dict with aggregate schedules for each commodity + that defines aggregate-consumption and/or aggregate-production sensors in its commodity context. + + The sign convention and split logic follows the same pattern as _build_consumption_production_schedules: + - Only aggregate-consumption defined: full aggregate schedule (consumption +, production -) + - Only aggregate-production defined: full aggregate schedule (consumption +, production -) + (sign will be flipped by make_schedule based on consumption_is_positive=False) + - Both defined: consumption sensor gets non-negative part, production sensor gets non-positive part + (sign will be flipped for production by make_schedule) + + For backwards compatibility, when no commodity_contexts are defined, all devices are treated + as electricity devices and use the top-level flex-context fields. + + :param storage_schedule: Dict to populate with aggregate schedules (will be modified in-place) + :param ems_schedule: DataFrame of per-device power schedules in MW (consumption positive) + """ + commodity_to_devices = self._reconstruct_commodity_to_devices() + # Get commodity contexts (handles backwards compatibility) commodity_contexts = self._get_commodity_contexts() From 95965a99ff7d7d06da74986f1ac74785861b7064 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:58:56 +0200 Subject: [PATCH 106/136] fix: default device commodity to electricity and match commitment commodity flex_model_d["commodity"] raised a KeyError for devices without an explicit commodity; default to "electricity" instead. Also only apply a commitment spec to devices of a matching commodity (defaulting the commitment's commodity to "electricity" too), so a future CommitmentSchema commodity field can be wired through without further changes here. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 40c4c598a2..9fe305b7ed 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -1337,12 +1337,14 @@ def convert_to_commitments( commitment_spec["index"] = initialize_index( start, end, timing_kwargs["resolution"] ) + commitment_commodity = commitment_spec.get("commodity", "electricity") for d, flex_model_d in enumerate(flex_model): + device_commodity = flex_model_d.get("commodity", "electricity") + if device_commodity != commitment_commodity: + continue commitment = FlowCommitment( device=d, - # todo: is flex_model_d guaranteed to have "commodity? Consider defaulting the device commodity to "electricity" - # todo: should there not be something matching the "commodity" from the commitment_spec (default to "electricity") to the device commodity? - device_group=flex_model_d["commodity"], + device_group=device_commodity, **commitment_spec, ) commitments.append(commitment) From c2e4f4ee4a2fcfa56cb19366e924d6bbbebb0960 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:59:04 +0200 Subject: [PATCH 107/136] fix: fix dead cross-commodity currency check in list-form flex-context shared_currency_unit was assigned before the is-None check on every iteration, so the mismatch branch could never fire when loading the list-of-dicts flex-context form. Compare against the previously seen currency instead. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 9fe305b7ed..335b7d673e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -1396,18 +1396,12 @@ def _deserialize_flex_context(self): ) # Ensure all flex-contexts share the same currency unit - # todo: move this into a validator for FlexContextSchema.commodity_contexts? shared_currency_unit = None for commodity_flex_context in self.flex_context: - shared_currency_unit = commodity_flex_context["shared_currency_unit"] + context_currency_unit = commodity_flex_context["shared_currency_unit"] if shared_currency_unit is None: - shared_currency_unit = commodity_flex_context[ - "shared_currency_unit" - ] - elif ( - commodity_flex_context["shared_currency_unit"] - != shared_currency_unit - ): + shared_currency_unit = context_currency_unit + elif context_currency_unit != shared_currency_unit: raise ValidationError( f"All prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." ) From 6e890a6e605b1293930d1910757d579f843dddd5 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Tue, 7 Jul 2026 23:59:12 +0200 Subject: [PATCH 108/136] fix: restrict aggregate-power to electricity devices aggregate-power previously summed every device's schedule regardless of commodity, mixing e.g. gas devices into an electricity aggregate. Sum only the flexible and inflexible devices belonging to the electricity commodity, per decision. Also drop the resolved todo comment and the unused commented-out sensors parameter of _compute_commodity_aggregate_schedules and its call site. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 335b7d673e..1dc696502f 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -2436,16 +2436,16 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: storage_schedule[sensor] += ems_schedule[d] # Obtain the aggregate power schedule, too, if the flex-context states the associated sensor. Fill with the sum of schedules made here. + # Restricted to electricity devices (flexible and inflexible), per decision. aggregate_power_sensor = self.flex_context.get("aggregate_power", None) if isinstance(aggregate_power_sensor, Sensor): + electricity_devices = self._electricity_device_indices() storage_schedule[aggregate_power_sensor] = pd.concat( - ems_schedule, - axis=1, # todo: select only electric devices (flexible and inflexible) + [ems_schedule[d] for d in electricity_devices if d < len(ems_schedule)], + axis=1, ).sum(axis=1) # Compute per-commodity aggregate power flows for aggregate-consumption and aggregate-production sensors - self._compute_commodity_aggregate_schedules( - storage_schedule, ems_schedule # , sensors - ) + self._compute_commodity_aggregate_schedules(storage_schedule, ems_schedule) # Convert each device schedule to the unit of the device's power sensor storage_schedule = { From b976b538e4d31b220e47055a074cd5b98c9627f2 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:01:31 +0200 Subject: [PATCH 109/136] fix: reject duplicate commodities in flex-context commodities list _get_commodity_contexts (storage.py) keys commodity contexts by commodity, so a duplicate entry in the `commodities` list silently overwrote an earlier one. Reject duplicates explicitly. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 3494ca1ace..bbdb9fa54b 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -441,6 +441,27 @@ def validate_aggregate_power_is_sensor( if not isinstance(aggregate_power, Sensor): raise ValidationError("The `aggregate-power` field can only be a Sensor.") + @validates("commodity_contexts") + def validate_commodity_contexts_unique( + self, commodity_contexts: list[dict], **kwargs + ): + """Validate that each commodity is listed at most once. + + `_get_commodity_contexts` (storage.py) builds a dict keyed by commodity, so + duplicate entries would otherwise silently overwrite each other. + """ + commodities = [context["commodity"] for context in commodity_contexts] + seen = set() + duplicates = set() + for commodity in commodities: + if commodity in seen: + duplicates.add(commodity) + seen.add(commodity) + if duplicates: + raise ValidationError( + f"Each commodity may only be listed once in `commodities`. Duplicate(s): {sorted(duplicates)}." + ) + @validates("commodity_contexts") def validate_commodity_contexts_shared_currency( self, commodity_contexts: list[dict], **kwargs From 1381b1485a3090e513f326d031ac27f23e0037d9 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:01:46 +0200 Subject: [PATCH 110/136] feat: reject non-electricity semantics in the single-dict flex-context A single flex-context dict only supports the electricity commodity. Reject a top-level `commodity` key with any value other than "electricity", and reject the case where the `commodities` list is combined with commodity-specific top-level SharedSchema fields, since it is then ambiguous which commodity those fields belong to. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index bbdb9fa54b..1d1733e85c 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -489,6 +489,55 @@ def validate_commodity_contexts_shared_currency( f" (found both '{shared_currency_unit}' and '{context_currency_unit}')" ) + # Fields defined on SharedSchema (i.e. not commodity_contexts, the price-sensor + # deprecations, nor aggregate_power) are commodity-specific semantics: they only + # make sense for a single commodity (electricity, in the single-dict form). + _FLEX_CONTEXT_ONLY_FIELDS = frozenset( + { + "commodity_contexts", + "consumption_price_sensor", + "production_price_sensor", + "aggregate_power", + } + ) + + @validates_schema(pass_original=True) + def check_single_dict_is_electricity_only( + self, data: dict, original_data: dict, **kwargs + ): + """A single flex-context dict may only define the electricity context. + + Reject: + - a `commodities` list combined with commodity-specific top-level fields + (ambiguous: which context do the top-level fields belong to?), + - a top-level `commodity` key with a value other than "electricity". + """ + if not isinstance(original_data, dict): + return + + top_level_commodity = original_data.get("commodity") + if top_level_commodity is not None and top_level_commodity != "electricity": + raise ValidationError( + "The top-level flex-context dict only supports the 'electricity' " + "commodity. Use the `commodities` list to define other commodities.", + field_name="commodity", + ) + + if data.get("commodity_contexts"): + shared_field_data_keys = { + field.data_key + for field_var, field in self.declared_fields.items() + if field_var not in self._FLEX_CONTEXT_ONLY_FIELDS + } + offending_keys = shared_field_data_keys & set(original_data.keys()) + if offending_keys: + raise ValidationError( + "When using the `commodities` list, commodity-specific fields " + "must be set inside each commodity's dict, not at the top level. " + f"Offending top-level field(s): {sorted(offending_keys)}.", + field_name="commodities", + ) + @validates_schema(pass_original=True) def check_prices(self, data: dict, original_data: dict, **kwargs): """Check assumptions about prices. From 777bc084e225183b57b893b9214ceaeee8324ff5 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:01:58 +0200 Subject: [PATCH 111/136] fix: raise ValidationError for malformed flex-context input Extend the existing normalize_flex_context_format pre_load hook to raise a proper 422 ValidationError when the normalized flex-context is neither a dict nor a list of dicts, instead of letting downstream code fail with a TypeError. Signed-off-by: F.N. Claessen --- flexmeasures/data/schemas/scheduling/__init__.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 1d1733e85c..cf2cb715dd 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -1245,6 +1245,15 @@ def normalize_flex_context_format(self, data, **kwargs): # Regular list case data["flex-context"] = {"commodities": raw_flex_context} # else: already a dict, leave as-is + + # By now, flex-context should always be normalized to a dict. If it isn't + # (e.g. a bare string or number was passed), raise a 422 here instead of + # letting downstream code fail with a TypeError. + if not isinstance(data["flex-context"], dict): + raise ValidationError( + "`flex-context` must be an object, or a list of objects.", + field_name="flex-context", + ) return data @validates_schema From 8210c6811d099b4e1bcf2b1b7bd4517efd483cb2 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:02:07 +0200 Subject: [PATCH 112/136] refactor: drop dead 'types' comment blocks in UI_FLEX_CONTEXT_SCHEMA Per-commodity UI editing of aggregate-consumption/aggregate-production field types is a follow-up; remove the commented-out placeholders. Signed-off-by: F.N. Claessen --- flexmeasures/data/schemas/scheduling/__init__.py | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index cf2cb715dd..7401e89ae6 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -664,21 +664,11 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "aggregate-consumption": { "default": None, "description": rst_to_openapi(metadata.AGGREGATE_CONSUMPTION.description), - # todo: the field type is defined in asset_context.html in 3 places? - # "types": { - # "backend": "typeTwo", - # "ui": "A sensor which records the scheduled aggregate consumption.", - # }, "example-units": EXAMPLE_UNIT_TYPES["power"], }, "aggregate-production": { "default": None, "description": rst_to_openapi(metadata.AGGREGATE_PRODUCTION.description), - # todo: the field type is defined in asset_context.html in 3 places? - # "types": { - # "backend": "typeTwo", - # "ui": "A sensor which records the scheduled aggregate production.", - # }, "example-units": EXAMPLE_UNIT_TYPES["power"], }, "consumption-price": { From 04dfdd0ef693798c756fcaa520b3f5de219062a7 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:07:38 +0200 Subject: [PATCH 113/136] test: add schema-level regression tests for hardening fixes Cover duplicate commodities in the commodities list, non-electricity semantics in the single-dict flex-context form, mixing commodities with top-level shared fields, and malformed flex-context input rejected with a ValidationError instead of a TypeError. Signed-off-by: F.N. Claessen --- .../data/schemas/tests/test_scheduling.py | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index aef5fbef9d..619ae98e76 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -1135,3 +1135,63 @@ def test_flex_context_listing_shared_currency( schema = FlexContextSchema() check_schema_loads_data(schema=schema, data=flex_context_listing, fails=fails) + + +def test_flex_context_listing_rejects_duplicate_commodities(db, app): + """test_flex_context_listing_rejects_duplicate_commodities: a commodity listed twice must be rejected.""" + schema = FlexContextSchema() + flex_context = { + "commodities": [ + {"commodity": "electricity", "consumption-price": "1 EUR/MWh"}, + {"commodity": "electricity", "production-price": "1 EUR/MWh"}, + ] + } + check_schema_loads_data( + schema=schema, + data=flex_context, + fails={"commodities": "may only be listed once"}, + ) + + +def test_flex_context_single_dict_rejects_non_electricity_commodity(db, app): + """test_flex_context_single_dict_rejects_non_electricity_commodity: the single-dict form only supports electricity.""" + schema = FlexContextSchema() + flex_context = {"commodity": "gas", "consumption-price": "1 EUR/MWh"} + check_schema_loads_data( + schema=schema, + data=flex_context, + fails={"commodity": "only supports the 'electricity' commodity"}, + ) + + +def test_flex_context_single_dict_allows_explicit_electricity_commodity(db, app): + """test_flex_context_single_dict_allows_explicit_electricity_commodity: explicit electricity is fine.""" + schema = FlexContextSchema() + flex_context = {"commodity": "electricity", "consumption-price": "1 EUR/MWh"} + check_schema_loads_data(schema=schema, data=flex_context, fails=False) + + +def test_flex_context_rejects_commodities_with_top_level_shared_fields(db, app): + """test_flex_context_rejects_commodities_with_top_level_shared_fields: ambiguous mixed forms are rejected.""" + schema = FlexContextSchema() + flex_context = { + "consumption-price": "1 EUR/MWh", + "commodities": [ + {"commodity": "electricity", "consumption-price": "1 EUR/MWh"}, + ], + } + check_schema_loads_data( + schema=schema, + data=flex_context, + fails={"commodities": "must be set inside each commodity's dict"}, + ) + + +def test_asset_trigger_schema_rejects_malformed_flex_context(app): + """test_asset_trigger_schema_rejects_malformed_flex_context: a non-dict/list flex-context must raise a ValidationError, not a TypeError.""" + from flexmeasures.data.schemas.scheduling import AssetTriggerSchema + + schema = AssetTriggerSchema() + with pytest.raises(ValidationError) as e_info: + schema.normalize_flex_context_format({"flex-context": "not-a-dict-or-list"}) + assert "flex-context" in str(e_info.value) From 5ec17ae1512f6462ed2881de117e757949d414f0 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:09:03 +0200 Subject: [PATCH 114/136] test: add scheduler-level regression tests for gas-only and per-commodity inflexible sensors Add test_all_gas_flex_model_without_electricity_device, covering a flex-model with no electricity device (guarding the setdefault fix), and test_per_commodity_inflexible_device_sensors, covering an inflexible-device-sensor declared inside a (non-electricity) commodity context. Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 179 +++++++++++++++++- 1 file changed, 178 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index a58e4919b9..05b605e0c4 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -12,7 +12,8 @@ initialize_index, ) from flexmeasures.data.models.planning.storage import StorageScheduler -from flexmeasures.data.models.time_series import Sensor +from flexmeasures.data.models.time_series import Sensor, TimedBelief +from flexmeasures.data.models.data_sources import DataSource from flexmeasures.data.models.planning.linear_optimization import device_scheduler from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType from flexmeasures.data.utils import save_to_db @@ -1550,3 +1551,179 @@ def test_simulation_with_dynamic_consumption_capacity(app, db): assert heater_schedule.loc["2026-04-07T08:00:00+00:00"] == pytest.approx( 80.0 ), "Electric heater should have one expected partial 80 kW dispatch step before the first cheap-electricity window." + + +def test_all_gas_flex_model_without_electricity_device(app, db): + """test_all_gas_flex_model_without_electricity_device: a flex-model with only gas + devices (no electricity device at all) should not raise a KeyError, now that + commodity_to_devices["electricity"] is built with setdefault(). + """ + boiler_type = get_or_create_model(GenericAssetType, name="gas-boiler") + + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-02T00:00:00+01:00") + resolution = pd.Timedelta("1h") + + gas_boiler = GenericAsset( + name="Gas Boiler (all-gas flex-model test)", + generic_asset_type=boiler_type, + ) + db.session.add(gas_boiler) + db.session.commit() + + boiler_power = Sensor( + name="boiler power", + unit="kW", + event_resolution=resolution, + generic_asset=gas_boiler, + ) + db.session.add(boiler_power) + db.session.commit() + + flex_model = [ + { + "sensor": boiler_power.id, + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0, + }, + ] + + flex_context = [ + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh", + "production-price": "50 EUR/MWh", + }, + ] + + scheduler = StorageScheduler( + asset_or_sensor=gas_boiler, + start=start, + end=end, + resolution=resolution, + belief_time=start, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + # This used to raise KeyError("electricity") in _prepare(). + schedules = scheduler.compute(skip_validation=True) + + storage_schedules = [ + entry for entry in schedules if entry.get("name") == "storage_schedule" + ] + assert len(storage_schedules) == 1 + boiler_schedule = storage_schedules[0]["data"] + assert (boiler_schedule == 1.0).all() + + +def test_per_commodity_inflexible_device_sensors(app, db): + """test_per_commodity_inflexible_device_sensors: an inflexible-device-sensor declared + inside a (non-electricity) commodity context should constrain that commodity's site + capacity and be reflected in the flexible device's schedule (since its consumption + capacity leaves no room for the inflexible load plus more). + """ + boiler_type = get_or_create_model(GenericAssetType, name="gas-boiler") + + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-01T04:00:00+01:00") + resolution = pd.Timedelta("1h") + + gas_site = GenericAsset( + name="Gas Site (per-commodity inflexible sensor test)", + generic_asset_type=boiler_type, + ) + db.session.add(gas_site) + db.session.commit() + + flexible_boiler_power = Sensor( + name="flexible boiler power", + unit="kW", + event_resolution=resolution, + generic_asset=gas_site, + ) + inflexible_gas_load = Sensor( + name="inflexible gas load", + unit="kW", + event_resolution=resolution, + generic_asset=gas_site, + ) + db.session.add_all([flexible_boiler_power, inflexible_gas_load]) + db.session.commit() + + # A constant 8 kW inflexible gas load, recorded as beliefs. + index = initialize_index(start, end, resolution) + + source = get_or_create_model( + DataSource, name="test source", source_type="forecaster" + ) + beliefs = [ + TimedBelief( + sensor=inflexible_gas_load, + source=source, + event_start=dt, + belief_time=start, + event_value=8.0, + ) + for dt in index + ] + db.session.add_all(beliefs) + db.session.commit() + + flex_model = [ + { + "sensor": flexible_boiler_power.id, + "commodity": "gas", + "power-capacity": "30 kW", + "consumption-capacity": "30 kW", + "production-capacity": "0 kW", + "soc-usage": ["1 kW"], + "soc-min": 0.0, + "soc-max": 0.0, + "soc-at-start": 0.0, + }, + ] + + flex_context = [ + { + "commodity": "gas", + "consumption-price": "50 EUR/MWh", + "production-price": "50 EUR/MWh", + "site-consumption-capacity": "10 kW", + "inflexible-device-sensors": [inflexible_gas_load.id], + }, + ] + + scheduler = StorageScheduler( + asset_or_sensor=gas_site, + start=start, + end=end, + resolution=resolution, + belief_time=start, + flex_model=flex_model, + flex_context=flex_context, + return_multiple=True, + ) + + schedules = scheduler.compute(skip_validation=True) + + storage_schedules = [ + entry for entry in schedules if entry.get("name") == "storage_schedule" + ] + boiler_schedule = next( + entry for entry in storage_schedules if entry["sensor"] == flexible_boiler_power + )["data"] + + # With an 8 kW inflexible gas load counted against the 10 kW site-consumption-capacity, + # the flexible boiler (which otherwise consumes a constant 1 kW) is left at most 2 kW of + # headroom. Since the boiler's own consumption is fixed at 1 kW via soc-usage, this test + # mainly asserts the schedule was computed without infeasibility, i.e. the per-commodity + # inflexible sensor was taken into account as a device (not simply dropped). + assert (boiler_schedule <= 2.0 + 1e-6).all() From 65f0fe786c16381e7c79896149c65ddfd86744ab Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:09:42 +0200 Subject: [PATCH 115/136] docs: changelog entry for multi-commodity hardening fixes Document the correctness fixes and the relax-constraints default change under the v1.0.0 section. Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 5cc669e98b..8ed7d2d4ac 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -36,6 +36,7 @@ Bugfixes ----------- * Let storage scheduling treat missing constant SoC bounds as unconstrained lower or upper bounds [see `PR #2221 `_] * Allow root assets belonging to different accounts to share the same name, while keeping asset names unique among root assets within the same account and among children of the same parent [see `PR #2226 `_] +* Fix several multi-commodity scheduling issues found in review of the flex-context ``commodities`` list: a broken cross-commodity currency check that could never fire, a ``KeyError`` when a flex-model contains no electricity device, ``inflexible-device-sensors`` declared inside a (non-electricity) commodity context being silently ignored (they now constrain that commodity's site capacity and are included in its aggregate schedule), devices without an explicit ``commodity`` key defaulting to ``electricity``, ``aggregate-power`` now summing only electricity devices, and duplicate commodities or non-electricity semantics in the single-dict flex-context form now being rejected with a 422 instead of behaving unpredictably. As part of this, the top-level flex-context's ``relax-constraints`` field now also defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] v0.33.1 | July 1, 2026 From d39a6a2f22206fdc1679ad94341b6e1a0c70061c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:14:46 +0200 Subject: [PATCH 116/136] fix: count only device models, not stock entries, when enumerating devices in _prepare The commodity_to_devices mapping and the num_flexible_devices count were built from the re-copied flex_model, which includes stock-model entries. This overran the sensors list (IndexError at sensor_d = sensors[d]) whenever the flex-model contained a stock entry, and gave stock entries bogus device indices in commodity groups. Enumerate device_models instead, keeping the earlier num_flexible_devices = len(device_models) and matching _compute_commodity_aggregate_schedules, which already uses self._device_models. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index af1626a89a..97cc4b48c8 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -342,17 +342,18 @@ def device_list_series( ) -> pd.Series: return pd.Series([tuple(devices)] * len(index), index=index, name="device") + # Enumerate only device models (not stock entries), so device indices line up + # with the sensors and device_constraints lists. commodity_to_devices = {} - for d, flex_model_d in enumerate(flex_model): + for d, flex_model_d in enumerate(device_models): commodity = flex_model_d.get("commodity", "electricity") commodity_to_devices.setdefault(commodity, []).append(d) # inflexible devices are electricity by default - number_flexible_devices = len(flex_model) + number_flexible_devices = len(device_models) number_inflexible_devices = len( self.flex_context.get("inflexible_device_sensors", []) ) - num_flexible_devices = len(flex_model) commodity_to_devices["electricity"] += list( range( number_flexible_devices, From a00ab9a93ee16a4ab3451084027c8dae4099269a Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:32:35 +0200 Subject: [PATCH 117/136] fix: declare commodity field on FlexContextSchema The single-dict electricity-only check never ran: FlexContextSchema declares no commodity field and marshmallow raises 'Unknown field.' for it before schema-level validators fire. Declare the field (load_default 'electricity') with a OneOf validator carrying a helpful message pointing to the commodities list. Not part of the documented UI/OpenAPI fields. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 7401e89ae6..a2945ae904 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -405,6 +405,23 @@ def __init__(self, *args, **kwargs): class FlexContextSchema(SharedSchema): """This schema defines fields that provide context to the portfolio to be optimized.""" + # The single-dict flex-context form only supports the electricity commodity. + # Other commodities must be defined via the `commodities` list. + # Not part of the documented UI/OpenAPI fields. + commodity = fields.Str( + required=False, + load_default="electricity", + data_key="commodity", + validate=validate.OneOf( + ["electricity"], + error="The top-level flex-context dict only supports the 'electricity' " + "commodity. Use the `commodities` list to define other commodities.", + ), + metadata=dict( + description="Commodity of the single-dict flex-context form; only 'electricity' is supported here. Use the `commodities` list to define other commodities.", + ), + ) + commodity_contexts = fields.Nested( CommodityFlexContextSchema, data_key="commodities", From 5d190544050ddd5cb61762be48f0aac8f7989167 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:32:52 +0200 Subject: [PATCH 118/136] fix: tolerate commodities list combined with top-level flex-context fields The mixed-fields rejection caused false positives: in the API path, a multi-commodity list is normalized to {"commodities": [...]} and collect_flex_config then dict-merges the asset's db-stored flex-context fields (e.g. site-power-capacity) at the top level, so any asset with stored electricity flex-context fields would get a 422. Remove the rejection; semantics stay as designed: top-level fields serve as the electricity context only when the commodities list has no electricity entry (see _get_commodity_contexts). Documented in the changelog and as a code comment. Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 3 +- .../data/schemas/scheduling/__init__.py | 56 +++---------------- .../data/schemas/tests/test_scheduling.py | 18 +++--- 3 files changed, 20 insertions(+), 57 deletions(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 8ed7d2d4ac..bd7fb20dec 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -19,6 +19,7 @@ New features * Support multiple feeders to a shared storage [see `PR #2001 `_ ] * The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_ and `PR #2235 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] +* Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] Infrastructure / Support ---------------------- @@ -36,7 +37,7 @@ Bugfixes ----------- * Let storage scheduling treat missing constant SoC bounds as unconstrained lower or upper bounds [see `PR #2221 `_] * Allow root assets belonging to different accounts to share the same name, while keeping asset names unique among root assets within the same account and among children of the same parent [see `PR #2226 `_] -* Fix several multi-commodity scheduling issues found in review of the flex-context ``commodities`` list: a broken cross-commodity currency check that could never fire, a ``KeyError`` when a flex-model contains no electricity device, ``inflexible-device-sensors`` declared inside a (non-electricity) commodity context being silently ignored (they now constrain that commodity's site capacity and are included in its aggregate schedule), devices without an explicit ``commodity`` key defaulting to ``electricity``, ``aggregate-power`` now summing only electricity devices, and duplicate commodities or non-electricity semantics in the single-dict flex-context form now being rejected with a 422 instead of behaving unpredictably. As part of this, the top-level flex-context's ``relax-constraints`` field now also defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] +* Fix several multi-commodity scheduling issues found in review of the flex-context ``commodities`` list: a broken cross-commodity currency check that could never fire, a ``KeyError`` when a flex-model contains no electricity device, ``inflexible-device-sensors`` declared inside a (non-electricity) commodity context being silently ignored (they now constrain that commodity's site capacity and are included in its aggregate schedule), devices without an explicit ``commodity`` key defaulting to ``electricity``, ``aggregate-power`` now summing only electricity devices, and duplicate commodities or a non-electricity ``commodity`` in the single-dict flex-context form now being rejected with a 422 instead of behaving unpredictably. Note: combining the ``commodities`` list with top-level flex-context fields is deliberately tolerated (top-level fields serve as the electricity context only when the list has no electricity entry), because assets with a db-stored flex-context get their stored fields merged at the top level [see `PR #2172 `_] v0.33.1 | July 1, 2026 diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index a2945ae904..e4ebed8f4a 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -506,54 +506,14 @@ def validate_commodity_contexts_shared_currency( f" (found both '{shared_currency_unit}' and '{context_currency_unit}')" ) - # Fields defined on SharedSchema (i.e. not commodity_contexts, the price-sensor - # deprecations, nor aggregate_power) are commodity-specific semantics: they only - # make sense for a single commodity (electricity, in the single-dict form). - _FLEX_CONTEXT_ONLY_FIELDS = frozenset( - { - "commodity_contexts", - "consumption_price_sensor", - "production_price_sensor", - "aggregate_power", - } - ) - - @validates_schema(pass_original=True) - def check_single_dict_is_electricity_only( - self, data: dict, original_data: dict, **kwargs - ): - """A single flex-context dict may only define the electricity context. - - Reject: - - a `commodities` list combined with commodity-specific top-level fields - (ambiguous: which context do the top-level fields belong to?), - - a top-level `commodity` key with a value other than "electricity". - """ - if not isinstance(original_data, dict): - return - - top_level_commodity = original_data.get("commodity") - if top_level_commodity is not None and top_level_commodity != "electricity": - raise ValidationError( - "The top-level flex-context dict only supports the 'electricity' " - "commodity. Use the `commodities` list to define other commodities.", - field_name="commodity", - ) - - if data.get("commodity_contexts"): - shared_field_data_keys = { - field.data_key - for field_var, field in self.declared_fields.items() - if field_var not in self._FLEX_CONTEXT_ONLY_FIELDS - } - offending_keys = shared_field_data_keys & set(original_data.keys()) - if offending_keys: - raise ValidationError( - "When using the `commodities` list, commodity-specific fields " - "must be set inside each commodity's dict, not at the top level. " - f"Offending top-level field(s): {sorted(offending_keys)}.", - field_name="commodities", - ) + # Note: we deliberately tolerate a `commodities` list combined with top-level + # commodity-specific (SharedSchema) fields. In the API path, a multi-commodity + # list is normalized to {"commodities": [...]} and collect_flex_config then + # dict-merges the asset's db-stored flex-context (e.g. "site-power-capacity", + # "consumption-price") at the top level, so rejecting this mix would 422 any + # asset with stored electricity flex-context fields. Semantics: top-level fields + # serve as the electricity context only when the commodities list has no + # electricity entry (see _get_commodity_contexts in storage.py). @validates_schema(pass_original=True) def check_prices(self, data: dict, original_data: dict, **kwargs): diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 619ae98e76..f0cf26c134 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -1171,20 +1171,22 @@ def test_flex_context_single_dict_allows_explicit_electricity_commodity(db, app) check_schema_loads_data(schema=schema, data=flex_context, fails=False) -def test_flex_context_rejects_commodities_with_top_level_shared_fields(db, app): - """test_flex_context_rejects_commodities_with_top_level_shared_fields: ambiguous mixed forms are rejected.""" +def test_flex_context_tolerates_commodities_with_top_level_shared_fields(db, app): + """test_flex_context_tolerates_commodities_with_top_level_shared_fields: mixing must be tolerated. + + The API path dict-merges an asset's db-stored (electricity) flex-context fields at the + top level after normalizing a multi-commodity list to {"commodities": [...]}, so this + mix must load fine. Top-level fields serve as the electricity context only when the + commodities list has no electricity entry (see _get_commodity_contexts in storage.py). + """ schema = FlexContextSchema() flex_context = { "consumption-price": "1 EUR/MWh", "commodities": [ - {"commodity": "electricity", "consumption-price": "1 EUR/MWh"}, + {"commodity": "gas", "consumption-price": "1 EUR/MWh"}, ], } - check_schema_loads_data( - schema=schema, - data=flex_context, - fails={"commodities": "must be set inside each commodity's dict"}, - ) + check_schema_loads_data(schema=schema, data=flex_context, fails=False) def test_asset_trigger_schema_rejects_malformed_flex_context(app): From f4a82624da2bc40c7285a626c931e2c81ed45b65 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:33:03 +0200 Subject: [PATCH 119/136] refactor: use units_are_convertible for scheduler-side currency check; fix stale comment Match the schema-side currency comparison in _deserialize_flex_context, and update a test comment still claiming relax_constraints defaults to False. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 6 ++++-- flexmeasures/data/schemas/tests/test_scheduling.py | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 1d995298c3..feb30f90d3 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -42,7 +42,7 @@ ) from flexmeasures.utils.time_utils import get_max_planning_horizon from flexmeasures.utils.time_utils import determine_minimum_resampling_resolution -from flexmeasures.utils.unit_utils import ur, convert_units +from flexmeasures.utils.unit_utils import ur, convert_units, units_are_convertible storage_asset_types = ["one-way_evse", "two-way_evse", "battery", "heat-storage"] @@ -1402,7 +1402,9 @@ def _deserialize_flex_context(self): context_currency_unit = commodity_flex_context["shared_currency_unit"] if shared_currency_unit is None: shared_currency_unit = context_currency_unit - elif context_currency_unit != shared_currency_unit: + elif not units_are_convertible( + context_currency_unit, shared_currency_unit + ): raise ValidationError( f"All prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." ) diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index f0cf26c134..90842d2cf9 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -932,7 +932,7 @@ def test_flex_model_schemas( }, False, ), - # Test that relax_constraints defaults to False in FlexContextSchema + # Test that relax_constraints defaults to True in FlexContextSchema ( {"site-power-capacity": "1 MVA"}, False, From 3c544f256052af7bd012d9031110580682f05e72 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 00:33:12 +0200 Subject: [PATCH 120/136] test: strengthen per-commodity inflexible sensor test; cover electricity-only aggregate indices Assert via an aggregate-consumption sensor that the gas commodity's inflexible load (8 kW) is included in the aggregate schedule on top of the flexible boiler's schedule; before the fix, the per-commodity inflexible sensor's device index was silently dropped, so this assertion would have failed. Also add a DB-free unit test that _electricity_device_indices (used by the aggregate-power sum) covers only electricity devices, flexible and inflexible. Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 60 +++++++++++++++++-- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 05b605e0c4..ab834bcb62 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1655,7 +1655,15 @@ def test_per_commodity_inflexible_device_sensors(app, db): event_resolution=resolution, generic_asset=gas_site, ) - db.session.add_all([flexible_boiler_power, inflexible_gas_load]) + gas_aggregate_consumption = Sensor( + name="gas aggregate consumption", + unit="kW", + event_resolution=resolution, + generic_asset=gas_site, + ) + db.session.add_all( + [flexible_boiler_power, inflexible_gas_load, gas_aggregate_consumption] + ) db.session.commit() # A constant 8 kW inflexible gas load, recorded as beliefs. @@ -1698,6 +1706,7 @@ def test_per_commodity_inflexible_device_sensors(app, db): "production-price": "50 EUR/MWh", "site-consumption-capacity": "10 kW", "inflexible-device-sensors": [inflexible_gas_load.id], + "aggregate-consumption": {"sensor": gas_aggregate_consumption.id}, }, ] @@ -1723,7 +1732,50 @@ def test_per_commodity_inflexible_device_sensors(app, db): # With an 8 kW inflexible gas load counted against the 10 kW site-consumption-capacity, # the flexible boiler (which otherwise consumes a constant 1 kW) is left at most 2 kW of - # headroom. Since the boiler's own consumption is fixed at 1 kW via soc-usage, this test - # mainly asserts the schedule was computed without infeasibility, i.e. the per-commodity - # inflexible sensor was taken into account as a device (not simply dropped). + # headroom. assert (boiler_schedule <= 2.0 + 1e-6).all() + + # The aggregate-consumption schedule for the gas commodity must include the inflexible + # gas load. Before the fix, the per-commodity inflexible sensor got no device constraints + # (its device index was silently dropped), so the aggregate would only reflect the + # flexible boiler's ~1 kW instead of 8 + 1 = 9 kW. + aggregate_schedule = next( + entry + for entry in storage_schedules + if entry["sensor"] == gas_aggregate_consumption + )["data"] + expected_aggregate = boiler_schedule + 8.0 + assert aggregate_schedule.values == pytest.approx( + expected_aggregate.values, abs=1e-6 + ), ( + "Aggregate gas consumption should include the 8 kW inflexible gas load " + "on top of the flexible boiler's schedule." + ) + + +def test_electricity_device_indices_exclude_other_commodities(): + """test_electricity_device_indices_exclude_other_commodities: the device indices used + for the aggregate-power sum should cover only electricity devices (flexible and + inflexible), not gas devices, nor per-commodity inflexible devices of other commodities. + """ + scheduler = object.__new__(StorageScheduler) + # Flexible devices: 0 = electricity, 1 = gas, 2 = electricity (implicit default) + scheduler._device_models = [ + {"commodity": "electricity"}, + {"commodity": "gas"}, + {}, # defaults to electricity + ] + scheduler.flex_model = scheduler._device_models + # Top-level inflexible sensors (electricity): indices 3 and 4. + # Gas commodity context with one inflexible sensor: index 5. + scheduler.flex_context = { + "inflexible_device_sensors": ["el_sensor_a", "el_sensor_b"], + "commodity_contexts": [ + {"commodity": "gas", "inflexible_device_sensors": ["gas_sensor"]}, + ], + } + + mapping = scheduler._reconstruct_commodity_to_devices() + assert mapping["electricity"] == [0, 2, 3, 4] + assert mapping["gas"] == [1, 5] + assert scheduler._electricity_device_indices() == [0, 2, 3, 4] From 2724920b1a561f09bd1c6494ab96d82bb70e649a Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 07:51:40 +0200 Subject: [PATCH 121/136] fix: keep deprecated price sensors working and fix breach-price error attribution Two regressions from defaulting relax-constraints to True: 1. The 'please switch to consumption-price' guard counted the load_default-filled relax_constraints as an explicitly used new-style field, breaking the deprecated consumption/production-price-sensor path with a hard 422. Count only fields explicitly present in the original input. 2. Default breach prices were filled even when the shared currency unit was derived from a mis-united price field (e.g. '6 kWh' passed as a breach price), causing DBFlexContextSchema unit validation to flag the filled soc-*-breach-price fields instead of the actual offending field. Skip the fills when the deprecated price sensor fields are used (legacy behaviour unchanged) or when the derived shared currency is not an actual currency. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index e4ebed8f4a..cf3077e613 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -41,6 +41,7 @@ is_capacity_price_unit, is_energy_price_unit, is_power_unit, + is_currency_unit, is_energy_unit, ) from flexmeasures.utils.validation_utils import validate_variable_quantity @@ -538,8 +539,10 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): field.data_key: field_var for field_var, field in self.declared_fields.items() } + # Only count fields that were explicitly passed (not filled in by a load_default, + # such as relax-constraints, which defaults to True). if any( - field_map[field] in data and data[field_map[field]] + field in original_data and data.get(field_map[field]) for field in ( "soc-minima-breach-price", "soc-maxima-breach-price", @@ -585,6 +588,20 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): field_name="commodities", ) + # Skip filling default breach prices when: + # - the deprecated price sensor fields are used (those predate relaxation + # support; filling defaults would silently change legacy behaviour), or + # - the shared currency is not an actual currency (e.g. a mis-united price + # field slipped through _try_to_convert_price_units); filling defaults in a + # nonsense currency would misattribute unit errors to the breach price + # fields in downstream validation (e.g. DBFlexContextSchema). + if ( + "consumption_price_sensor" in data + or "production_price_sensor" in data + or not is_currency_unit(data["shared_currency_unit"]) + ): + return data + # Fill in default soc breach prices when asked to relax SoC constraints, unless already set explicitly. if ( data["relax_soc_constraints"] From 3c7bf96c22b7db3d6d16011ff586c15fbb8fa2bb Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 07:53:25 +0200 Subject: [PATCH 122/136] fix: skip commodity device groups without devices in _prepare An empty group could be created for a commodity that no device refers to (e.g. the always-created electricity entry in an all-gas flex-model, or a commodity context listed in the flex-context but unused by the flex-model). Such groups demanded a consumption price and produced empty-device commitments and EMS constraint groups, making the problem unbounded. Skip them: they need no prices, commitments or constraints. Also normalize inflexible_device_sensors to a list (tests bypassing the schema may pass dict_values, which broke concatenation with the per-commodity inflexible sensors), and fix the DataSource keyword (type, not source_type) in the new per-commodity inflexible sensor test. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 11 +++++++++-- .../data/models/planning/tests/test_commitments.py | 4 +--- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index feb30f90d3..5bc1f72363 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -295,8 +295,9 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ] # Get info from flex-context - inflexible_device_sensors = self.flex_context.get( - "inflexible_device_sensors", [] + # (normalize to a list; tests may pass e.g. dict_values when bypassing the schema) + inflexible_device_sensors = list( + self.flex_context.get("inflexible_device_sensors", []) ) # Fetch the device's power capacity (required to keep the optimization problem bounded) @@ -382,6 +383,12 @@ def device_list_series( price_frames_by_commodity = {} for commodity, devices in commodity_to_devices.items(): + # Skip commodities without any devices (e.g. no electricity devices in an + # all-gas flex-model, or a commodity context that no device refers to): + # they need no prices, commitments or EMS constraints, and empty device + # groups would make the optimization problem unbounded. + if not devices: + continue commodity_devices = device_list_series(devices, index) commodity_context = commodity_contexts.get(commodity, {}) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index ab834bcb62..cf38a31670 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1669,9 +1669,7 @@ def test_per_commodity_inflexible_device_sensors(app, db): # A constant 8 kW inflexible gas load, recorded as beliefs. index = initialize_index(start, end, resolution) - source = get_or_create_model( - DataSource, name="test source", source_type="forecaster" - ) + source = get_or_create_model(DataSource, name="test source", type="forecaster") beliefs = [ TimedBelief( sensor=inflexible_gas_load, From 97b8b5fca24a012a4d09d1270071b3af74462275 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 07:55:44 +0200 Subject: [PATCH 123/136] test: pin relax-constraints False in tests asserting hard-capacity semantics test_capacity, test_battery_power_capacity_as_sensor and test_device_power_capacity_uses_directional_capacity_before_site_fallback assert the hard constraint bounds (derivative min/max DataFrames) equal the contracted capacities. With relax-constraints defaulting to True, default breach prices are filled and the hard bound becomes the physical capacity (contracted capacities are then only softly penalized), so these tests must opt out explicitly. Note the default breach prices (10000/kW site, 100/kW device, plus soc prices) dwarf typical energy prices, so optimal schedules still respect contracted capacities in ordinary price-arbitrage cases; the relaxation only kicks in when the problem would otherwise be infeasible. Signed-off-by: F.N. Claessen --- .../data/models/planning/tests/test_solver.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index edb0ad45a8..182f2cc400 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -1523,6 +1523,10 @@ def test_capacity( flex_context = { "production-price": {"sensor": add_market_prices["epex_da_production"].id}, "consumption-price": {"sensor": add_market_prices["epex_da"].id}, + # This test asserts hard-capacity semantics (constraint DataFrames), so opt out + # of the default constraint relaxation (which would use the physical capacity + # as the hard bound and penalize the contracted capacity only softly). + "relax-constraints": False, } def set_if_not_none(dictionary, key, value): @@ -1621,7 +1625,8 @@ def test_device_power_capacity_uses_directional_capacity_before_site_fallback( "soc-max": 5, **configured_capacities, }, - flex_context={"consumption-price": "1 EUR/MWh"}, + # relax-constraints False: this test asserts hard-capacity semantics. + flex_context={"consumption-price": "1 EUR/MWh", "relax-constraints": False}, ) scheduler.deserialize_config() @@ -1879,7 +1884,11 @@ def test_battery_power_capacity_as_sensor( resolution = timedelta(minutes=15) soc_at_start = 10 - flex_context = {"site-power-capacity": "1100 kVA"} # 1.1 MW + flex_context = { + "site-power-capacity": "1100 kVA", # 1.1 MW + # relax-constraints False: this test asserts hard-capacity semantics. + "relax-constraints": False, + } if site_consumption_capacity_sensor: flex_context["site-consumption-capacity"] = { "sensor": capacity_sensors["site_power_capacity"].id From 7e22cffaf67d12e05c1b3263f7906923d1825a1d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 07:55:57 +0200 Subject: [PATCH 124/136] test: exclude the commodity field from the UI flex-context schema guard The new FlexContextSchema.commodity field (single-dict form is electricity-only) is deliberately not exposed in the UI, like the relax-* developer fields. Signed-off-by: F.N. Claessen --- flexmeasures/ui/tests/test_utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/flexmeasures/ui/tests/test_utils.py b/flexmeasures/ui/tests/test_utils.py index eff2114fc6..b9925823b7 100644 --- a/flexmeasures/ui/tests/test_utils.py +++ b/flexmeasures/ui/tests/test_utils.py @@ -80,6 +80,7 @@ def test_ui_flexcontext_schema(): "consumption-price-sensor", "production-price-sensor", "commodities", # todo: https://github.com/FlexMeasures/flexmeasures/issues/2230 + "commodity", # single-dict form is electricity-only; not exposed in the UI ] schema_keys = [] From 9592a8eac796fc98bd6e584aab6aca359b90ec29 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 08:16:55 +0200 Subject: [PATCH 125/136] test: fix sign convention of inflexible gas load beliefs Power sensors store consumption as negative values by default; get_power_values flips the sign to the scheduler's consumption-positive convention. The test wrote +8 kW beliefs, which entered the problem as 8 kW production, making the aggregate 1 - 8 = -7 kW instead of the expected 1 + 8 = 9 kW. Record the load as -8 kW. The per-commodity inflexible path in _prepare shares the exact same get_power_values loop as the top-level path, so no production-code change is needed. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/tests/test_commitments.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index cf38a31670..6ad735f906 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -1667,6 +1667,8 @@ def test_per_commodity_inflexible_device_sensors(app, db): db.session.commit() # A constant 8 kW inflexible gas load, recorded as beliefs. + # By default, power sensors store consumption as negative values + # (get_power_values flips the sign to the scheduler's consumption-positive convention). index = initialize_index(start, end, resolution) source = get_or_create_model(DataSource, name="test source", type="forecaster") @@ -1676,7 +1678,7 @@ def test_per_commodity_inflexible_device_sensors(app, db): source=source, event_start=dt, belief_time=start, - event_value=8.0, + event_value=-8.0, ) for dt in index ] From e78a0d3508cbef427645e14c05bf72a00e953e1c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 08:30:37 +0200 Subject: [PATCH 126/136] Add commodity field to CommitmentSchema Wire an undocumented commodity field (defaulting to "electricity") into CommitmentSchema, so a flex-context commitment only binds devices of its own commodity. StorageScheduler.convert_to_commitments already matched commitment and device commodities; this adds the schema-level field so the commodity can actually be set on a commitment spec. Signed-off-by: F.N. Claessen --- .../models/planning/tests/test_commitments.py | 61 +++++++++++++++++++ .../data/schemas/scheduling/__init__.py | 9 +++ 2 files changed, 70 insertions(+) diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index 6ad735f906..29ac245187 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -17,6 +17,7 @@ from flexmeasures.data.models.planning.linear_optimization import device_scheduler from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType from flexmeasures.data.utils import save_to_db +from flexmeasures.utils.unit_utils import ur def test_multi_feed_device_scheduler_shared_buffer(): @@ -1779,3 +1780,63 @@ def test_electricity_device_indices_exclude_other_commodities(): assert mapping["electricity"] == [0, 2, 3, 4] assert mapping["gas"] == [1, 5] assert scheduler._electricity_device_indices() == [0, 2, 3, 4] + + +def test_commitment_commodity_does_not_bind_other_commodity_devices(): + """test_commitment_commodity_does_not_bind_other_commodity_devices: a commitment + listed under the flex-context's `commitments` should only bind devices of its own + `commodity` (defaulting to "electricity", like devices do). A gas commitment + should therefore not create a FlowCommitment against an electricity device, and + vice versa. + + This is a DB-free, unit-level test of StorageScheduler.convert_to_commitments. + """ + scheduler = object.__new__(StorageScheduler) + scheduler.flex_context = { + "shared_currency_unit": "EUR", + "commitments": [ + { + "name": "gas commitment", + "commodity": "gas", + "baseline": ur.Quantity("1 MW"), + }, + { + # No `commodity` given: defaults to "electricity", like devices do. + "name": "electricity commitment", + "baseline": ur.Quantity("2 MW"), + }, + ], + } + # Flexible devices: 0 = electricity, 1 = gas. + flex_model = [ + {"commodity": "electricity"}, + {"commodity": "gas"}, + ] + + start = pd.Timestamp("2024-01-01T00:00:00+01:00") + end = pd.Timestamp("2024-01-01T03:00:00+01:00") + resolution = pd.Timedelta("1h") + + commitments = scheduler.convert_to_commitments( + flex_model=flex_model, + query_window=(start, end), + resolution=resolution, + beliefs_before=None, + ) + + assert len(commitments) == 2 + + gas_commitment = next(c for c in commitments if c.name == "gas commitment") + electricity_commitment = next( + c for c in commitments if c.name == "electricity commitment" + ) + + # The gas commitment binds only the gas device (index 1), not the electricity + # device (index 0). + assert (gas_commitment.device == 1).all() + assert set(gas_commitment.device_group.unique()) == {"gas"} + + # The electricity commitment (commodity defaulting to "electricity") binds only + # the electricity device (index 0), not the gas device (index 1). + assert (electricity_commitment.device == 0).all() + assert set(electricity_commitment.device_group.unique()) == {"electricity"} diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index cf3077e613..9c7fa3f457 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -71,6 +71,15 @@ def forbid_time_series_specs(self, data: dict, **kwargs): class CommitmentSchema(Schema): name = fields.Str(required=True, data_key="name") + # Undocumented for now (not part of UI_FLEX_CONTEXT_SCHEMA, OpenAPI or Sphinx docs). + # Determines which commodity's devices this commitment binds (see + # StorageScheduler.convert_to_commitments, which matches this against each + # device's own `commodity`, defaulting to "electricity" as well). + commodity = fields.Str( + required=False, + load_default="electricity", + data_key="commodity", + ) baseline = VariableQuantityField("MW", required=False, data_key="baseline") up_price = VariableQuantityField("/MW", required=False, data_key="up-price") down_price = VariableQuantityField( From dbe922f7c196060e0b556b1e5e418568ae7ccc97 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 08:30:46 +0200 Subject: [PATCH 127/136] Add smarter defaults for commodity contexts A commodities-list entry that omits some or all grid-connection fields (consumption-price, production-price, site-consumption- capacity, site-production-capacity, site-power-capacity) previously either failed (a missing consumption-price crashes the scheduler) or left ambiguous, error-prone combinations to fill in by hand. Add CommodityFlexContextSchema.fill_grid_connection_defaults, a post-load step that derives sensible defaults from which of those five fields were explicitly given: - none given: no grid connection (both site capacities default to 0, as soft constraints; site-power-capacity stays unlimited) - only a price given: assume a grid connection in that direction (unlimited capacity there), 0 capacity in the other direction - only a capacity given: 0 price in that direction, and the other direction defaults to a 0 capacity (and price) - only site-power-capacity given: a hard constraint at that capacity for both directions, with 0 prices Combinations of explicitly given fields apply the same rules per direction independently. A 0 capacity filled in by this method is enforced as a soft constraint (a default breach price is filled in too), not a hard, potentially infeasible, one; explicit hard constraints (e.g. via site-power-capacity alone) get no breach price. Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 1 + .../data/schemas/scheduling/__init__.py | 142 +++++++++++++++++ .../data/schemas/tests/test_scheduling.py | 145 ++++++++++++++++++ 3 files changed, 288 insertions(+) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index bd7fb20dec..6a3f4af265 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -20,6 +20,7 @@ New features * The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_ and `PR #2235 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] * Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] +* A ``commodities`` entry in the flex-context that omits some or all of its grid-connection fields (``consumption-price``, ``production-price``, ``site-consumption-capacity``, ``site-production-capacity`` and ``site-power-capacity``) now gets smarter defaults instead of failing or silently defaulting to a fully-connected, unconstrained grid. In short: a fully bare commodity context (e.g. just ``{"commodity": "gas"}``) is treated as having no grid connection at all (both site capacities default to 0, as soft constraints); giving only a price for one direction assumes a grid connection in that direction (with an unlimited capacity, unless a capacity is also given) and a 0 capacity in the other direction; giving only a capacity for one direction implies a 0 price for that direction (and a 0 capacity/price for the other direction); and giving only ``site-power-capacity`` sets a hard constraint at that capacity for both directions, with 0 prices. See ``CommodityFlexContextSchema.fill_grid_connection_defaults`` for the full precedence rules. Infrastructure / Support ---------------------- diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 9c7fa3f457..ebe7e76ecb 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -411,6 +411,148 @@ def __init__(self, *args, **kwargs): [("commodity", commodity_field), *self.fields.items()] ) + @post_load(pass_original=True) + def fill_grid_connection_defaults(self, data: dict, original_data: dict, **kwargs): + """Fill in smarter defaults for a commodity context's grid-connection fields. + + A commodity context (an entry of the top-level `commodities` list) may omit + some or all of the grid-connection fields (`consumption-price`, + `production-price`, `site-consumption-capacity`, `site-production-capacity`, + `site-power-capacity`). Rather than leaving those simply unset (which, for + `consumption-price`, would make the scheduler fail, since it requires one), + we derive sensible defaults from *which* of those five fields were explicitly + given (inspecting the original input, not post-default-fill presence). + + Precedence (single-field triggers; see the plan in `documentation/changelog.rst` + for cases 13/14): + + 1. None of the five given (e.g. just `{"commodity": "gas"}`): no grid + connection at all. `site-consumption-capacity` and + `site-production-capacity` default to 0, as *soft* constraints (a default + breach price is filled in, so breaching is possible but penalized -- this + relies on `relax-constraints`/`relax-site-capacity-constraints`, which + default to True). `site-power-capacity` is left unlimited (unset). + 2. Only `consumption-price` given: assume a grid connection for consumption. + `site-power-capacity` and `site-consumption-capacity` stay unlimited. + `site-production-capacity` defaults to 0 (soft). + 3. Only `production-price` given: the mirror image of (2), for production. + 4. Only `site-consumption-capacity` given: `site-power-capacity` stays + unlimited; `consumption-price` defaults to 0; `site-production-capacity` + (and, transitively, `production-price`) default to 0. + 5. Only `site-production-capacity` given: the mirror image of (4). + 6. Only `site-power-capacity` given: a *hard* constraint at that capacity. + `site-consumption-capacity` and `site-production-capacity` are both set + equal to it (no breach price is filled in, so the constraint stays hard); + `consumption-price` and `production-price` default to 0. + + For any *combination* of explicitly given fields, only the fields not + determined by one of the rules above are filled in, applying the same rules + per direction (consumption / production) independently: a price given for a + direction implies a grid connection in that direction (its capacity is left + unlimited unless also given explicitly); a capacity given for a direction + (and no price) implies a 0 price in that direction. The "hard constraint" + rule (6) only applies when `site-power-capacity` is the *sole* field given. + As a safety net (since the scheduler requires a resolvable consumption + price), `consumption-price` defaults to 0 if still unset after applying the + rules above (`production-price` already falls back to `consumption-price` + at the scheduler level, so no separate safety net is needed for it). + """ + + has_consumption_price = "consumption-price" in original_data + has_production_price = "production-price" in original_data + has_consumption_capacity = "site-consumption-capacity" in original_data + has_production_capacity = "site-production-capacity" in original_data + has_power_capacity = "site-power-capacity" in original_data + + any_given = ( + has_consumption_price + or has_production_price + or has_consumption_capacity + or has_production_capacity + or has_power_capacity + ) + + currency = data.get("shared_currency_unit") or "EUR" + zero_price = ur.Quantity(f"0 {currency}/MWh") + zero_capacity = ur.Quantity("0 MW") + + # Case 6: site-power-capacity is the only field given -> hard constraint. + if has_power_capacity and not ( + has_consumption_price + or has_production_price + or has_consumption_capacity + or has_production_capacity + ): + power_capacity = data["ems_power_capacity_in_mw"] + data.setdefault("ems_consumption_capacity_in_mw", power_capacity) + data.setdefault("ems_production_capacity_in_mw", power_capacity) + data.setdefault("consumption_price", zero_price) + data.setdefault("production_price", zero_price) + return data + + # Case 1: nothing given at all -> fully disconnected commodity. + if not any_given: + self._default_zero_capacity_as_soft_constraint( + data, "ems_consumption_capacity_in_mw", zero_capacity + ) + self._default_zero_capacity_as_soft_constraint( + data, "ems_production_capacity_in_mw", zero_capacity + ) + data.setdefault("consumption_price", zero_price) + return data + + # Cases 2-5 and combinations thereof: fill in what's still missing, per + # direction (consumption/production), independently. + if not has_consumption_price and not has_consumption_capacity: + self._default_zero_capacity_as_soft_constraint( + data, "ems_consumption_capacity_in_mw", zero_capacity + ) + if has_consumption_capacity and not has_consumption_price: + data.setdefault("consumption_price", zero_price) + + if not has_production_price and not has_production_capacity: + self._default_zero_capacity_as_soft_constraint( + data, "ems_production_capacity_in_mw", zero_capacity + ) + if has_production_capacity and not has_production_price: + data.setdefault("production_price", zero_price) + + # Safety net: the scheduler requires a resolvable consumption price. + data.setdefault("consumption_price", zero_price) + + return data + + def _default_zero_capacity_as_soft_constraint( + self, data: dict, field: str, zero_capacity: ur.Quantity + ): + """Default a site capacity field to 0, as a *soft* constraint. + + Also fills in a default breach price for that direction (unless one was + already set), so the 0 capacity is enforced as a soft constraint (breaching + is possible, but penalized) rather than a hard, potentially infeasible, one. + This mirrors FlexContextSchema.check_prices, but scoped to a single + commodity context, and only fired for capacities defaulted here (not for + capacities the caller explicitly set to 0). + """ + if field in data: + # Already set (e.g. by an earlier rule in this method); leave it as-is. + return + data[field] = zero_capacity + + breach_price_field = { + "ems_consumption_capacity_in_mw": "ems_consumption_breach_price", + "ems_production_capacity_in_mw": "ems_production_breach_price", + }[field] + if data.get("relax_site_capacity_constraints") or data.get("relax_constraints"): + if not data.get(breach_price_field): + currency = data.get("shared_currency_unit") or "EUR" + shared_currency = ur.Quantity(currency) + self.set_default_breach_prices( + data, + fields=[breach_price_field], + price=10000 * shared_currency / ur.Quantity("kW"), + ) + class FlexContextSchema(SharedSchema): """This schema defines fields that provide context to the portfolio to be optimized.""" diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 90842d2cf9..b40c637fe4 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -16,6 +16,7 @@ ) from flexmeasures.data.models.time_series import Sensor from flexmeasures.data.schemas.sensors import TimedEventSchema, VariableQuantityField +from flexmeasures.utils.unit_utils import ur @pytest.mark.parametrize( @@ -1053,6 +1054,150 @@ def test_commodity_flex_context_defaults( assert loaded.get("relax_constraints", True) is True +def _assert_quantity_or_none(actual, expected): + """Compare an (optionally None) ur.Quantity against an expected ur.Quantity or None.""" + if expected is None: + assert actual is None + else: + assert actual is not None + assert actual.to(expected.units).magnitude == pytest.approx(expected.magnitude) + + +@pytest.mark.parametrize( + ["context_input", "expected"], + [ + # Case 1: none of the 5 grid-connection fields given -> fully disconnected + # commodity. Both site capacities default to 0 as *soft* constraints (a + # default breach price is filled in); site-power-capacity stays unlimited. + ( + {"commodity": "gas"}, + { + "ems_consumption_capacity_in_mw": ur.Quantity("0 MW"), + "ems_production_capacity_in_mw": ur.Quantity("0 MW"), + "ems_power_capacity_in_mw": None, + "consumption_price": ur.Quantity("0 EUR/MWh"), + "ems_consumption_breach_price_set": True, + "ems_production_breach_price_set": True, + }, + ), + # Case 2: only consumption-price given -> assume a grid connection for + # consumption (unlimited site-power/consumption-capacity); 0 + # site-production-capacity (soft). + ( + {"commodity": "gas", "consumption-price": "10 EUR/MWh"}, + { + "ems_consumption_capacity_in_mw": None, + "ems_production_capacity_in_mw": ur.Quantity("0 MW"), + "ems_power_capacity_in_mw": None, + "consumption_price": ur.Quantity("10 EUR/MWh"), + "ems_production_breach_price_set": True, + }, + ), + # Case 3: only production-price given -> mirror image of case 2. + ( + {"commodity": "gas", "production-price": "10 EUR/MWh"}, + { + "ems_consumption_capacity_in_mw": ur.Quantity("0 MW"), + "ems_production_capacity_in_mw": None, + "ems_power_capacity_in_mw": None, + "consumption_price": ur.Quantity("0 EUR/MWh"), + "production_price": ur.Quantity("10 EUR/MWh"), + "ems_consumption_breach_price_set": True, + }, + ), + # Case 4: only site-consumption-capacity given -> unlimited + # site-power-capacity, 0 consumption-price, 0 site-production-capacity + # (soft), (and thereby 0 production-price). + ( + {"commodity": "gas", "site-consumption-capacity": "5 MW"}, + { + "ems_consumption_capacity_in_mw": ur.Quantity("5 MW"), + "ems_production_capacity_in_mw": ur.Quantity("0 MW"), + "ems_power_capacity_in_mw": None, + "consumption_price": ur.Quantity("0 EUR/MWh"), + "ems_production_breach_price_set": True, + }, + ), + # Case 5: only site-production-capacity given -> mirror image of case 4. + ( + {"commodity": "gas", "site-production-capacity": "5 MW"}, + { + "ems_consumption_capacity_in_mw": ur.Quantity("0 MW"), + "ems_production_capacity_in_mw": ur.Quantity("5 MW"), + "ems_power_capacity_in_mw": None, + "consumption_price": ur.Quantity("0 EUR/MWh"), + "production_price": ur.Quantity("0 EUR/MWh"), + "ems_consumption_breach_price_set": True, + }, + ), + # Case 6: only site-power-capacity given -> a *hard* constraint at that + # capacity (both site capacities set equal to it; no breach price filled + # in); 0 consumption- and production-price. + ( + {"commodity": "gas", "site-power-capacity": "5 MW"}, + { + "ems_consumption_capacity_in_mw": ur.Quantity("5 MW"), + "ems_production_capacity_in_mw": ur.Quantity("5 MW"), + "ems_power_capacity_in_mw": ur.Quantity("5 MW"), + "consumption_price": ur.Quantity("0 EUR/MWh"), + "production_price": ur.Quantity("0 EUR/MWh"), + "ems_consumption_breach_price_set": False, + "ems_production_breach_price_set": False, + }, + ), + # A multi-field combination: consumption-price given together with an + # explicit site-power-capacity. The site-power-capacity is not the *sole* + # field given, so it does not trigger the hard-constraint case; instead, + # each direction is filled in independently: consumption-price given -> + # site-consumption-capacity stays unlimited (implicitly bounded by + # site-power-capacity at the scheduler level); production side untouched + # -> 0 site-production-capacity (soft). + ( + { + "commodity": "gas", + "consumption-price": "10 EUR/MWh", + "site-power-capacity": "5 MW", + }, + { + "ems_consumption_capacity_in_mw": None, + "ems_production_capacity_in_mw": ur.Quantity("0 MW"), + "ems_power_capacity_in_mw": ur.Quantity("5 MW"), + "consumption_price": ur.Quantity("10 EUR/MWh"), + "ems_production_breach_price_set": True, + }, + ), + ], +) +def test_commodity_flex_context_smart_defaults(context_input, expected): + """Test the smarter defaults for commodity contexts (see + CommodityFlexContextSchema.fill_grid_connection_defaults). + + These are DB-free, direct schema loads (no sensors involved). + """ + from flexmeasures.data.schemas.scheduling import CommodityFlexContextSchema + + loaded = CommodityFlexContextSchema().load(context_input) + + for field in ( + "ems_consumption_capacity_in_mw", + "ems_production_capacity_in_mw", + "ems_power_capacity_in_mw", + "consumption_price", + "production_price", + ): + if field in expected: + _assert_quantity_or_none(loaded.get(field), expected[field]) + + if "ems_consumption_breach_price_set" in expected: + assert (loaded.get("ems_consumption_breach_price") is not None) == expected[ + "ems_consumption_breach_price_set" + ] + if "ems_production_breach_price_set" in expected: + assert (loaded.get("ems_production_breach_price") is not None) == expected[ + "ems_production_breach_price_set" + ] + + @pytest.mark.parametrize( ["flex_context_listing", "fails"], [ From 5fa8b902e5e71e34320d72ad7e64f6af59949cc0 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:32:22 +0200 Subject: [PATCH 128/136] Fix spurious cross-currency error for price-free commodity contexts A commodity context with no user-given price fields (e.g. a bare {"commodity": "gas"}) was stamped with a fallback "EUR" currency by _try_to_convert_price_units, which then tripped the cross-context and top-level/context currency checks against a differently-currencied portfolio, even though the context had no real price constraint of its own. Track when a context's shared_currency_unit is just a fallback, skip such contexts in the currency comparisons, and let their 0-price/breach-price fills inherit the portfolio's real currency where determinable. Also warn when a smart-defaulted 0-capacity soft constraint ends up hard because relax-constraints was explicitly set to False. Signed-off-by: F.N. Claessen --- .../data/schemas/scheduling/__init__.py | 146 ++++++++++++++++-- .../data/schemas/tests/test_scheduling.py | 65 ++++++++ 2 files changed, 195 insertions(+), 16 deletions(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index ebe7e76ecb..387efccbbe 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -4,6 +4,8 @@ from datetime import timedelta from typing import Any, Callable, Dict +from flask import current_app + from marshmallow import ( Schema, fields, @@ -379,9 +381,57 @@ def _try_to_convert_price_units(self, data: dict, original_data: dict, **kwargs) elif sensor := data.get("production_price_sensor"): data["shared_currency_unit"] = self._to_currency_per_mwh(sensor.unit) else: + # No user-given price fields at all: fall back to "EUR", but flag this + # as a default (not user-given), so cross-context/cross-schema currency + # comparisons can skip it (a price-free context should never trip a + # currency-mismatch error against the rest of a differently-currencied + # portfolio; see CommodityFlexContextSchema.fill_grid_connection_defaults + # and FlexContextSchema.validate_commodity_contexts_shared_currency). data["shared_currency_unit"] = "EUR" + data["shared_currency_unit_is_default"] = True return data + # Currency-denominated fields that CommodityFlexContextSchema's smart defaults + # (fill_grid_connection_defaults) may fill with a fallback "EUR" price/breach + # price when a context has no user-given price fields at all. + _CURRENCY_DENOMINATED_FIELDS = ( + "consumption_price", + "production_price", + "ems_consumption_breach_price", + "ems_production_breach_price", + "consumption_breach_price", + "production_breach_price", + "soc_minima_breach_price", + "soc_maxima_breach_price", + "ems_peak_consumption_price", + "ems_peak_production_price", + ) + + @classmethod + def _rebase_default_context_currency(cls, context: dict, new_currency: str): + """Re-express a price-free context's fallback-currency fields in another currency. + + Only called for a commodity context that had no user-given price fields + (``shared_currency_unit_is_default`` is True), once a real portfolio + currency becomes known (e.g. from the top-level flex-context, or from a + sibling commodity context). All of that context's currency-denominated + fields were filled with plain quantities in a fallback "EUR", so their + magnitudes carry over unchanged under the new currency label (no FX + conversion is implied or attempted). + """ + for field in cls._CURRENCY_DENOMINATED_FIELDS: + value = context.get(field) + if not isinstance(value, ur.Quantity): + continue + old_units = str(value.units) + denominator = old_units.split("/", 1)[1] if "/" in old_units else None + new_unit = ( + new_currency if denominator is None else f"{new_currency}/{denominator}" + ) + context[field] = ur.Quantity(value.magnitude, new_unit) + context["shared_currency_unit"] = new_currency + context["shared_currency_unit_is_default"] = False + @staticmethod def _to_currency_per_mwh(price_unit: str) -> str: """Convert a price unit to a base currency used to express that price per MWh. @@ -552,6 +602,19 @@ def _default_zero_capacity_as_soft_constraint( fields=[breach_price_field], price=10000 * shared_currency / ur.Quantity("kW"), ) + elif data.get("relax_constraints") is False: + # relax-constraints defaults to True, so False here can only be an + # explicit user choice. Since relax-site-capacity-constraints is also + # not set/true, this 0 capacity ends up as a *hard* constraint, which + # is likely infeasible for any commodity with actual devices/flow. + current_app.logger.warning( + f"Commodity context '{data.get('commodity', 'electricity')}' has" + f" its '{field}' defaulted to a 0 capacity, but" + " 'relax-constraints' was explicitly set to False (and" + " 'relax-site-capacity-constraints' was not set to True), so this" + " ends up as a hard 0-capacity constraint, which is likely" + " infeasible." + ) class FlexContextSchema(SharedSchema): @@ -647,6 +710,11 @@ def validate_commodity_contexts_shared_currency( shared_currency_unit = None for context in commodity_contexts: + if context.get("shared_currency_unit_is_default"): + # No user-given price fields in this context: its "EUR" currency is + # just a fallback, not a real constraint, so don't let it clash with + # a differently-currencied portfolio. + continue context_currency_unit = context.get("shared_currency_unit") if context_currency_unit is None: continue @@ -667,25 +735,44 @@ def validate_commodity_contexts_shared_currency( # serve as the electricity context only when the commodities list has no # electricity entry (see _get_commodity_contexts in storage.py). - @validates_schema(pass_original=True) - def check_prices(self, data: dict, original_data: dict, **kwargs): - """Check assumptions about prices. + def _reconcile_commodity_context_currencies(self, data: dict) -> str: + """Backfill price-free contexts' currency with the portfolio's real currency. - 1. The flex-context must contain at most 1 consumption price and at most 1 production price field. - 2. All prices must share the same currency. + Determines the portfolio's real (user-given) shared currency, if any: the + top-level one, unless it's itself just a fallback (no user-given price + fields at the top level), in which case falls back to the first + non-default commodity context's currency, if any. Then rebases any + price-free ("default currency") commodity context onto that real currency, + so their 0-price/breach-price fills inherit it. Returns the (possibly + just-updated) top-level `shared_currency_unit`. """ + commodity_contexts = data.get("commodity_contexts", []) or [] + real_shared_currency_unit = None + if not data.get("shared_currency_unit_is_default"): + real_shared_currency_unit = data["shared_currency_unit"] + else: + for context in commodity_contexts: + if not context.get("shared_currency_unit_is_default"): + real_shared_currency_unit = context["shared_currency_unit"] + break - # The flex-context must contain at most 1 consumption price and at most 1 production price field - if "consumption_price_sensor" in data and "consumption_price" in data: - raise ValidationError( - "Must pass either consumption-price or consumption-price-sensor." - ) - if "production_price_sensor" in data and "production_price" in data: - raise ValidationError( - "Must pass either production-price or production-price-sensor." - ) + if real_shared_currency_unit is not None and data.get( + "shared_currency_unit_is_default" + ): + data["shared_currency_unit"] = real_shared_currency_unit + data["shared_currency_unit_is_default"] = False + + if real_shared_currency_unit is not None: + for context in commodity_contexts: + if context.get("shared_currency_unit_is_default"): + self._rebase_default_context_currency( + context, real_shared_currency_unit + ) + + return data["shared_currency_unit"] - # New price fields can only be used after updating to the new consumption-price and production-price fields + def _check_deprecated_price_sensor_migration(self, data: dict, original_data: dict): + """New price fields can only be used after updating to consumption-price/production-price.""" field_map = { field.data_key: field_var for field_var, field in self.declared_fields.items() @@ -718,14 +805,41 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): f"""Please switch to using `production-price: {{"sensor": {data[field_map["production-price-sensor"]].id}}}`.""" ) + @validates_schema(pass_original=True) + def check_prices(self, data: dict, original_data: dict, **kwargs): + """Check assumptions about prices. + + 1. The flex-context must contain at most 1 consumption price and at most 1 production price field. + 2. All prices must share the same currency. + """ + + # The flex-context must contain at most 1 consumption price and at most 1 production price field + if "consumption_price_sensor" in data and "consumption_price" in data: + raise ValidationError( + "Must pass either consumption-price or consumption-price-sensor." + ) + if "production_price_sensor" in data and "production_price" in data: + raise ValidationError( + "Must pass either production-price or production-price-sensor." + ) + + self._check_deprecated_price_sensor_migration(data, original_data) + # make sure that the prices fields are valid price units # All prices must share the same unit data = self._try_to_convert_price_units(data, original_data) - shared_currency = ur.Quantity(data["shared_currency_unit"]) + shared_currency = ur.Quantity( + self._reconcile_commodity_context_currencies(data) + ) # Also check that top-level prices share their currency with any per-commodity contexts for context in data.get("commodity_contexts", []) or []: + if context.get("shared_currency_unit_is_default"): + # Already reconciled (or left as a harmless fallback, if no real + # currency was determinable anywhere) by + # _reconcile_commodity_context_currencies above. + continue context_currency_unit = context.get("shared_currency_unit") if context_currency_unit is None: continue diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index b40c637fe4..ed94d711a6 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -1282,6 +1282,71 @@ def test_flex_context_listing_shared_currency( check_schema_loads_data(schema=schema, data=flex_context_listing, fails=fails) +def test_flex_context_listing_tolerates_price_free_context_in_other_currency(): + """test_flex_context_listing_tolerates_price_free_context_in_other_currency: + a bare (price-free) commodity context must not trip the shared-currency check + against a differently-currencied portfolio, since it has no user-given prices + of its own -- its 0-price/breach-price fills should just inherit the + portfolio's real currency. + """ + schema = FlexContextSchema() + + # Case A: top-level price sets the portfolio currency. + loaded = schema.load( + { + "consumption-price": "10 USD/MWh", + "commodities": [ + {"commodity": "electricity", "consumption-price": "10 USD/MWh"}, + {"commodity": "gas"}, + ], + } + ) + assert loaded["shared_currency_unit"] == "USD" + gas_context = next( + c for c in loaded["commodity_contexts"] if c["commodity"] == "gas" + ) + assert gas_context["shared_currency_unit"] == "USD" + assert str(gas_context["consumption_price"].units) == "USD/MWh" + + # Case B: no top-level price; a sibling commodity context sets the currency. + loaded = schema.load( + { + "commodities": [ + {"commodity": "electricity", "consumption-price": "10 USD/MWh"}, + {"commodity": "gas"}, + ], + } + ) + assert loaded["shared_currency_unit"] == "USD" + gas_context = next( + c for c in loaded["commodity_contexts"] if c["commodity"] == "gas" + ) + assert gas_context["shared_currency_unit"] == "USD" + assert str(gas_context["consumption_price"].units) == "USD/MWh" + + # Case C: no price given anywhere -> falls back to EUR everywhere. + loaded = schema.load({"commodities": [{"commodity": "gas"}]}) + assert loaded["shared_currency_unit"] == "EUR" + gas_context = loaded["commodity_contexts"][0] + assert gas_context["shared_currency_unit"] == "EUR" + + # A genuine mismatch (both contexts have explicit, different currencies) must + # still be rejected. + check_schema_loads_data( + schema=schema, + data={ + "consumption-price": "10 USD/MWh", + "commodities": [ + {"commodity": "electricity", "consumption-price": "10 USD/MWh"}, + {"commodity": "gas", "consumption-price": "10 EUR/MWh"}, + ], + }, + fails={ + "commodities": "all prices in the flex-context must share the same currency unit" + }, + ) + + def test_flex_context_listing_rejects_duplicate_commodities(db, app): """test_flex_context_listing_rejects_duplicate_commodities: a commodity listed twice must be rejected.""" schema = FlexContextSchema() From d074831aed716ad79722aa7ebca21d9824bea486 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:32:29 +0200 Subject: [PATCH 129/136] Warn when a commitment's commodity matches no devices Helps catch a typo in a commitment's or device's commodity field; the commitment is still silently skipped (no binding, no error), matching the existing deferred-validation approach for commodity mismatches. Signed-off-by: F.N. Claessen --- flexmeasures/data/models/planning/storage.py | 35 +++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 5bc1f72363..516229f73e 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -35,6 +35,7 @@ CommodityFlexContextSchema, FlexContextSchema, MultiSensorFlexModelSchema, + SharedSchema, ) from flexmeasures.data.schemas.sensors import SensorReference, VariableQuantityField from flexmeasures.utils.calculations import ( @@ -1346,6 +1347,7 @@ def convert_to_commitments( start, end, timing_kwargs["resolution"] ) commitment_commodity = commitment_spec.get("commodity", "electricity") + bound_device_count = 0 for d, flex_model_d in enumerate(flex_model): device_commodity = flex_model_d.get("commodity", "electricity") if device_commodity != commitment_commodity: @@ -1356,6 +1358,15 @@ def convert_to_commitments( **commitment_spec, ) commitments.append(commitment) + bound_device_count += 1 + if bound_device_count == 0: + current_app.logger.warning( + f"Commitment '{commitment_spec.get('name')}' has commodity" + f" '{commitment_commodity}', which matches none of the devices" + " in the flex-model. This commitment will not bind any device" + " (check for a typo in the commitment's `commodity` field, or in" + " a device's `commodity` field in the flex-model)." + ) return commitments @@ -1403,9 +1414,17 @@ def _deserialize_flex_context(self): commodity_flex_context ) - # Ensure all flex-contexts share the same currency unit + # Ensure all flex-contexts share the same currency unit. Contexts with + # no user-given price fields at all (shared_currency_unit_is_default) + # only carry a fallback "EUR" currency, which isn't a real constraint, + # so they're skipped here and instead backfilled below, once a real + # portfolio currency is known. shared_currency_unit = None + default_currency_contexts = [] for commodity_flex_context in self.flex_context: + if commodity_flex_context.get("shared_currency_unit_is_default"): + default_currency_contexts.append(commodity_flex_context) + continue context_currency_unit = commodity_flex_context["shared_currency_unit"] if shared_currency_unit is None: shared_currency_unit = context_currency_unit @@ -1416,6 +1435,20 @@ def _deserialize_flex_context(self): f"All prices in the flex-context must share the same currency unit (in this case: '{shared_currency_unit}')." ) + # Let price-free contexts inherit the portfolio's actual currency, + # where determinable (i.e. when at least one other context set one). + if shared_currency_unit is not None: + for commodity_flex_context in default_currency_contexts: + SharedSchema._rebase_default_context_currency( + commodity_flex_context, shared_currency_unit + ) + elif default_currency_contexts: + # No context anywhere gave an explicit price: fall back to the + # (shared) default currency already stamped on each of them. + shared_currency_unit = default_currency_contexts[0][ + "shared_currency_unit" + ] + # Nest the flex-contexts per commodity under the commodity_contexts field self.flex_context = dict( commodity_contexts=self.flex_context, From 02713b2f81344d282382553318940adc2546ed10 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:32:44 +0200 Subject: [PATCH 130/136] Note commodity-context defaults behaviour change in changelog Flag explicitly that partially-specified commodity contexts used to leave capacities unlimited and hard-error on a missing consumption price, whereas they now get 0-capacity soft constraints and 0 prices; and that price-free contexts no longer trip a spurious cross-currency error against a differently-currencied portfolio. Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 6a3f4af265..c3269b4f2b 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -20,7 +20,7 @@ New features * The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_ and `PR #2235 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] * Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] -* A ``commodities`` entry in the flex-context that omits some or all of its grid-connection fields (``consumption-price``, ``production-price``, ``site-consumption-capacity``, ``site-production-capacity`` and ``site-power-capacity``) now gets smarter defaults instead of failing or silently defaulting to a fully-connected, unconstrained grid. In short: a fully bare commodity context (e.g. just ``{"commodity": "gas"}``) is treated as having no grid connection at all (both site capacities default to 0, as soft constraints); giving only a price for one direction assumes a grid connection in that direction (with an unlimited capacity, unless a capacity is also given) and a 0 capacity in the other direction; giving only a capacity for one direction implies a 0 price for that direction (and a 0 capacity/price for the other direction); and giving only ``site-power-capacity`` sets a hard constraint at that capacity for both directions, with 0 prices. See ``CommodityFlexContextSchema.fill_grid_connection_defaults`` for the full precedence rules. +* A ``commodities`` entry in the flex-context that omits some or all of its grid-connection fields (``consumption-price``, ``production-price``, ``site-consumption-capacity``, ``site-production-capacity`` and ``site-power-capacity``) now gets smarter defaults instead of failing or silently defaulting to a fully-connected, unconstrained grid. In short: a fully bare commodity context (e.g. just ``{"commodity": "gas"}``) is treated as having no grid connection at all (both site capacities default to 0, as soft constraints); giving only a price for one direction assumes a grid connection in that direction (with an unlimited capacity, unless a capacity is also given) and a 0 capacity in the other direction; giving only a capacity for one direction implies a 0 price for that direction (and a 0 capacity/price for the other direction); and giving only ``site-power-capacity`` sets a hard constraint at that capacity for both directions, with 0 prices. See ``CommodityFlexContextSchema.fill_grid_connection_defaults`` for the full precedence rules. **Behaviour change:** previously, a partially-specified commodity context left unmentioned capacities unlimited and would hard-error if no resolvable ``consumption-price`` could be determined; now, unmentioned capacities/prices are filled in per the rules above (typically a 0-capacity soft constraint plus a 0 price), so such contexts load successfully and no longer raise that error. A commodity context with no user-given price fields at all also no longer trips a spurious cross-currency error against a differently-currencied portfolio; its 0-price/breach-price fills instead inherit the portfolio's real currency where determinable. Infrastructure / Support ---------------------- From 6937bee48f68aae8158305c6b01c095a290de635 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:35:56 +0200 Subject: [PATCH 131/136] docs: reorder and consolidate changelog entries per self-review Move the relax-constraints breaking-change entry to the top of its section, and fold the PR #2271 hardening-fix summary into the existing multi-commodity feature entry (appending the PR reference) instead of listing it as a separate bugfix bullet. Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index bd7fb20dec..e4dcea6405 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -12,14 +12,14 @@ v1.0.0 | July XX, 2026 New features ------------- +* Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] * In the UI, asset and sensor lists can be filtered by ID prefix through API-backed search fields [see `PR #2231 `_] * Floor off-clock API datetimes to a non-instantaneous sensor's resolution by default when ingesting sensor data, uploading sensor data, and handling scheduler flex-model timed events; configurable with the ``floor_datetimes_to_resolution`` sensor attribute [see `PR #2146 `_] * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 `_] * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 `_] * Support multiple feeders to a shared storage [see `PR #2001 `_ ] -* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_ and `PR #2235 `_] +* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_, `PR #2235 `_ and `PR #2271 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] -* Breaking behaviour change: the top-level flex-context's ``relax-constraints`` field now defaults to ``True`` (matching the default already used within each ``commodities`` entry), so constraint violations are softly penalized by default instead of being hard constraints, unless explicitly set to ``False`` [see `PR #2172 `_] Infrastructure / Support ---------------------- @@ -37,7 +37,6 @@ Bugfixes ----------- * Let storage scheduling treat missing constant SoC bounds as unconstrained lower or upper bounds [see `PR #2221 `_] * Allow root assets belonging to different accounts to share the same name, while keeping asset names unique among root assets within the same account and among children of the same parent [see `PR #2226 `_] -* Fix several multi-commodity scheduling issues found in review of the flex-context ``commodities`` list: a broken cross-commodity currency check that could never fire, a ``KeyError`` when a flex-model contains no electricity device, ``inflexible-device-sensors`` declared inside a (non-electricity) commodity context being silently ignored (they now constrain that commodity's site capacity and are included in its aggregate schedule), devices without an explicit ``commodity`` key defaulting to ``electricity``, ``aggregate-power`` now summing only electricity devices, and duplicate commodities or a non-electricity ``commodity`` in the single-dict flex-context form now being rejected with a 422 instead of behaving unpredictably. Note: combining the ``commodities`` list with top-level flex-context fields is deliberately tolerated (top-level fields serve as the electricity context only when the list has no electricity entry), because assets with a db-stored flex-context get their stored fields merged at the top level [see `PR #2172 `_] v0.33.1 | July 1, 2026 From 50b1b1d15ede2adcf4beb9d62a12e29244bc7786 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:37:16 +0200 Subject: [PATCH 132/136] docs: restore intentional 'types' notes on UI_FLEX_CONTEXT_SCHEMA aggregate fields Per self-review: these commented-out notes on aggregate-consumption and aggregate-production are intentional placeholders for planned future UI support, not dead code to remove. Signed-off-by: F.N. Claessen --- flexmeasures/data/schemas/scheduling/__init__.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index cf3077e613..93c86ac555 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -658,11 +658,21 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): "aggregate-consumption": { "default": None, "description": rst_to_openapi(metadata.AGGREGATE_CONSUMPTION.description), + # todo: the field type is defined in asset_context.html in 3 places? + # "types": { + # "backend": "typeTwo", + # "ui": "A sensor which records the scheduled aggregate consumption.", + # }, "example-units": EXAMPLE_UNIT_TYPES["power"], }, "aggregate-production": { "default": None, "description": rst_to_openapi(metadata.AGGREGATE_PRODUCTION.description), + # todo: the field type is defined in asset_context.html in 3 places? + # "types": { + # "backend": "typeTwo", + # "ui": "A sensor which records the scheduled aggregate production.", + # }, "example-units": EXAMPLE_UNIT_TYPES["power"], }, "consumption-price": { From 1c5809e9bc92d3c9e118fddbf993de0234e5f8f4 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 13:40:51 +0200 Subject: [PATCH 133/136] Point CommitmentSchema.commodity docs to the commodities list The field's comment previously read like it documented a supported way to bind a commitment to a commodity. Clarify that it's internal bookkeeping, and that the documented way to associate a commitment with a commodity is to place it under the relevant entry of the multi-commodity flex-context's commodities list. Signed-off-by: F.N. Claessen --- flexmeasures/data/schemas/scheduling/__init__.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 387efccbbe..83d56de0f2 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -74,9 +74,12 @@ def forbid_time_series_specs(self, data: dict, **kwargs): class CommitmentSchema(Schema): name = fields.Str(required=True, data_key="name") # Undocumented for now (not part of UI_FLEX_CONTEXT_SCHEMA, OpenAPI or Sphinx docs). - # Determines which commodity's devices this commitment binds (see - # StorageScheduler.convert_to_commitments, which matches this against each - # device's own `commodity`, defaulting to "electricity" as well). + # Internal bookkeeping only: not the documented way to associate a commitment + # with a commodity. API users should instead place the commitment under the + # relevant entry of the multi-commodity `commodities` list (one flex-context + # per commodity) -- see StorageScheduler.convert_to_commitments, which matches + # this field against each device's own `commodity`, defaulting to + # "electricity" as well. commodity = fields.Str( required=False, load_default="electricity", From 022e5c0b6997f1c86e106086ce3e8e4226679826 Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Wed, 8 Jul 2026 17:29:50 +0200 Subject: [PATCH 134/136] fix: resolve CI failures from merge conflict between #1946 and #2072 (#2274) Context: - PR #2072 added scheduling_result output to StorageScheduler.compute() when return_multiple=True, and added _compute_unresolved_targets. - In single-sensor mode (self.sensor is set), flex_model entries lack a "sensor" key, so _compute_unresolved_targets was skipping every device and returning empty unresolved/resolved lists. Changes: - storage.py: fall back to self.sensor in _compute_unresolved_targets when the flex_model entry has no "sensor" key (single-sensor mode) - test_commitments.py: update schedule-count assertions (+1 for the new scheduling_result entry added by PR #2072) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> --- flexmeasures/data/models/planning/storage.py | 4 +++- .../models/planning/tests/test_commitments.py | 15 +++++++++------ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 21c9f5f647..cc1f9786bb 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -2219,7 +2219,9 @@ def _compute_unresolved_targets( # Devices without a state-of-charge sensor are included as long as a # key can be derived from the power sensor's generic asset (or the # power sensor itself). - power_sensor = flex_model_d.get("sensor") + # In single-sensor mode the flex_model entry has no "sensor" key; + # fall back to self.sensor (set when the scheduler was given a Sensor). + power_sensor = flex_model_d.get("sensor") or self.sensor if ( power_sensor is not None and hasattr(power_sensor, "generic_asset") diff --git a/flexmeasures/data/models/planning/tests/test_commitments.py b/flexmeasures/data/models/planning/tests/test_commitments.py index a58e4919b9..d8b8f54ff1 100644 --- a/flexmeasures/data/models/planning/tests/test_commitments.py +++ b/flexmeasures/data/models/planning/tests/test_commitments.py @@ -629,7 +629,9 @@ def test_two_flexible_assets_with_commodity(app, db): schedules = scheduler.compute(skip_validation=True) assert isinstance(schedules, list) - assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs + assert ( + len(schedules) == 4 + ) # 2 storage schedules + 1 commitment costs + 1 scheduling_result # Extract schedules by type storage_schedules = [ @@ -648,7 +650,6 @@ def test_two_flexible_assets_with_commodity(app, db): ) battery_data = battery_schedule["data"] - # Get heat pump schedule hp_schedule = next( entry for entry in storage_schedules if entry["sensor"] == hp_power ) @@ -790,7 +791,9 @@ def test_mixed_gas_and_electricity_assets(app, db): schedules = scheduler.compute(skip_validation=True) assert isinstance(schedules, list) - assert len(schedules) == 3 # 2 storage schedules + 1 commitment costs + assert ( + len(schedules) == 4 + ) # 2 storage schedules + 1 commitment costs + 1 scheduling_result # Extract schedules by type storage_schedules = [ @@ -989,9 +992,9 @@ def test_two_devices_shared_stock(app, db): "(device schedules, commitment costs, SOC)." ) - assert len(schedules) == 4, ( - "Expected 4 outputs: two inverter schedules, one commitment_costs " - "object, and one state_of_charge schedule." + assert len(schedules) == 5, ( + "Expected 5 outputs: two inverter schedules, one commitment_costs " + "object, one state_of_charge schedule, and one scheduling_result." ) # ---- extract schedules From 49abd524e43fa7d949e542c79c13e9b7600eeb26 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 20:51:58 +0200 Subject: [PATCH 135/136] feat(ci): suggest subagent default models (Claude can override) Signed-off-by: F.N. Claessen --- .claude/agents/api-backward-compatibility-specialist.md | 1 + .claude/agents/architecture-domain-specialist.md | 1 + .claude/agents/coordinator.md | 1 + .claude/agents/data-time-semantics-specialist.md | 1 + .claude/agents/documentation-developer-experience-specialist.md | 1 + .claude/agents/performance-scalability-specialist.md | 1 + .claude/agents/test-specialist.md | 1 + .claude/agents/tooling-ci-specialist.md | 1 + .claude/agents/ui-specialist.md | 1 + 9 files changed, 9 insertions(+) diff --git a/.claude/agents/api-backward-compatibility-specialist.md b/.claude/agents/api-backward-compatibility-specialist.md index ab3aac110b..71b93448e6 100644 --- a/.claude/agents/api-backward-compatibility-specialist.md +++ b/.claude/agents/api-backward-compatibility-specialist.md @@ -1,6 +1,7 @@ --- name: api-backward-compatibility-specialist description: Protects users and integrators by ensuring API changes are backwards compatible, properly versioned, and well-documented +model: sonnet --- # Agent: API & Backward Compatibility Specialist diff --git a/.claude/agents/architecture-domain-specialist.md b/.claude/agents/architecture-domain-specialist.md index 4cbdad7a06..8654d89815 100644 --- a/.claude/agents/architecture-domain-specialist.md +++ b/.claude/agents/architecture-domain-specialist.md @@ -1,6 +1,7 @@ --- name: architecture-domain-specialist description: Guards domain model, invariants, and architecture to maintain model clarity and prevent erosion of core principles +model: opus --- # Agent: Architecture & Domain Specialist diff --git a/.claude/agents/coordinator.md b/.claude/agents/coordinator.md index 9077ea39d4..0ad4ce5e41 100644 --- a/.claude/agents/coordinator.md +++ b/.claude/agents/coordinator.md @@ -1,6 +1,7 @@ --- name: coordinator description: Meta-agent that manages agent lifecycle, enforces structural standards, and maintains coherence across the agent system +model: sonnet --- # Agent: Coordinator diff --git a/.claude/agents/data-time-semantics-specialist.md b/.claude/agents/data-time-semantics-specialist.md index b01e477c4a..90aad5ed70 100644 --- a/.claude/agents/data-time-semantics-specialist.md +++ b/.claude/agents/data-time-semantics-specialist.md @@ -1,6 +1,7 @@ --- name: data-time-semantics-specialist description: Prevents subtle bugs in time handling, units, and data semantics with focus on timezone-aware operations and unit conversions +model: sonnet --- # Agent: Data & Time Semantics Specialist diff --git a/.claude/agents/documentation-developer-experience-specialist.md b/.claude/agents/documentation-developer-experience-specialist.md index 9d0831d298..615bb3e1ea 100644 --- a/.claude/agents/documentation-developer-experience-specialist.md +++ b/.claude/agents/documentation-developer-experience-specialist.md @@ -1,6 +1,7 @@ --- name: documentation-developer-experience-specialist description: Ensures excellent documentation, clear error messages, and smooth developer workflows to keep FlexMeasures accessible +model: sonnet --- # Agent: Documentation & Developer Experience Specialist diff --git a/.claude/agents/performance-scalability-specialist.md b/.claude/agents/performance-scalability-specialist.md index 9786881e2f..605d7993fb 100644 --- a/.claude/agents/performance-scalability-specialist.md +++ b/.claude/agents/performance-scalability-specialist.md @@ -1,6 +1,7 @@ --- name: performance-scalability-specialist description: Identifies performance bottlenecks, inefficient algorithms, and scalability issues to keep FlexMeasures fast under load +model: sonnet --- # Agent: Performance & Scalability Specialist diff --git a/.claude/agents/test-specialist.md b/.claude/agents/test-specialist.md index 39fcafcadf..e00364ff6e 100644 --- a/.claude/agents/test-specialist.md +++ b/.claude/agents/test-specialist.md @@ -1,6 +1,7 @@ --- name: test-specialist description: Focuses on test coverage, quality, and testing best practices without modifying production code +model: sonnet --- # Agent: Test Specialist diff --git a/.claude/agents/tooling-ci-specialist.md b/.claude/agents/tooling-ci-specialist.md index 02bf4b0620..bace95aadf 100644 --- a/.claude/agents/tooling-ci-specialist.md +++ b/.claude/agents/tooling-ci-specialist.md @@ -1,6 +1,7 @@ --- name: tooling-ci-specialist description: Reviews GitHub Actions workflows, pre-commit hooks, and CI/CD pipelines to ensure automation reliability +model: sonnet --- # Agent: Tooling & CI Specialist diff --git a/.claude/agents/ui-specialist.md b/.claude/agents/ui-specialist.md index a985cf6f09..725673a7ea 100644 --- a/.claude/agents/ui-specialist.md +++ b/.claude/agents/ui-specialist.md @@ -1,6 +1,7 @@ --- name: ui-specialist description: Guards UI consistency, permission patterns, JavaScript interaction patterns, and template quality in the FlexMeasures web interface +model: sonnet --- # Agent: UI Specialist From cb9505db18145d0982eb8d088b32bf7c2de9683c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 8 Jul 2026 21:02:05 +0200 Subject: [PATCH 136/136] docs: move explanation to proper docs section Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 3 +-- documentation/features/scheduling.rst | 26 +++++++++++++++++++ .../data/schemas/scheduling/__init__.py | 22 +++++++++------- 3 files changed, 40 insertions(+), 11 deletions(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index e9259ed94c..fba88b9a65 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -18,10 +18,9 @@ New features * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 `_] * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 `_] * Support multiple feeders to a shared storage [see `PR #2001 `_ ] -* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_, `PR #2235 `_ and `PR #2271 `_] +* The flex-context can now define multiple commodities, each specifying their own prices and grid capacities [see `PR #1946 `_, `PR #2172 `_, `PR #2235 `_, `PR #2271 `_ and `PR #2272 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] * Extended ``GET /api/v3_0/jobs/`` with a ``result`` field containing ``unresolved`` and ``resolved`` soft state-of-charge constraint analysis (``soc-minima``/``soc-maxima`` violations or satisfied constraints, keyed by asset ID) for scheduling jobs; both arrays are empty when no SoC constraints were defined [see `PR #2072 `_] -* A ``commodities`` entry in the flex-context that omits some or all of its grid-connection fields (``consumption-price``, ``production-price``, ``site-consumption-capacity``, ``site-production-capacity`` and ``site-power-capacity``) now gets smarter defaults instead of failing or silently defaulting to a fully-connected, unconstrained grid. In short: a fully bare commodity context (e.g. just ``{"commodity": "gas"}``) is treated as having no grid connection at all (both site capacities default to 0, as soft constraints); giving only a price for one direction assumes a grid connection in that direction (with an unlimited capacity, unless a capacity is also given) and a 0 capacity in the other direction; giving only a capacity for one direction implies a 0 price for that direction (and a 0 capacity/price for the other direction); and giving only ``site-power-capacity`` sets a hard constraint at that capacity for both directions, with 0 prices. See ``CommodityFlexContextSchema.fill_grid_connection_defaults`` for the full precedence rules. **Behaviour change:** previously, a partially-specified commodity context left unmentioned capacities unlimited and would hard-error if no resolvable ``consumption-price`` could be determined; now, unmentioned capacities/prices are filled in per the rules above (typically a 0-capacity soft constraint plus a 0 price), so such contexts load successfully and no longer raise that error. A commodity context with no user-given price fields at all also no longer trips a spurious cross-currency error against a differently-currencied portfolio; its 0-price/breach-price fills instead inherit the portfolio's real currency where determinable. Infrastructure / Support ---------------------- diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index aec2ce3397..77831da391 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -149,6 +149,32 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul The flexible device can still have its own power limit defined in its flex-model. +.. _commodity_context_defaults: + +Smart defaults for commodity-context grid connections +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For multi-commodity scheduling problems, each entry of the top-level ``commodities`` list is itself a flex-context (a "commodity context") describing the grid connection for that commodity. +A commodity context that leaves out some or all of its grid-connection fields (``consumption-price``, ``production-price``, ``site-consumption-capacity``, ``site-production-capacity`` and ``site-power-capacity``) gets sensible defaults for the missing fields, rather than failing or silently leaving the grid unconstrained. + +As a rule of thumb, a price given for a direction (consumption or production) implies a grid connection in that direction, with an unlimited capacity unless a capacity is also given; a capacity given for a direction (without a price) implies a 0 price in that direction; and anything not implied by a given field defaults to "no connection" (0 capacity, as a soft constraint). +The exception is ``site-power-capacity`` given on its own, which sets a *hard* (symmetric) capacity limit instead. + +This leads to the following defaults, depending on which fields are explicitly given: + +- **Nothing given** (e.g. just ``{"commodity": "gas"}``): both ``site-consumption-capacity`` and ``site-production-capacity`` default to 0, as soft constraints (a breach is possible, but penalized). ``site-power-capacity`` stays unlimited. +- **Only** ``consumption-price``: ``site-power-capacity`` and ``site-consumption-capacity`` stay unlimited; ``site-production-capacity`` defaults to 0 (soft). +- **Only** ``production-price``: the mirror image, for production. +- **Only** ``site-consumption-capacity``: ``site-power-capacity`` stays unlimited; ``consumption-price`` defaults to 0; ``site-production-capacity`` (and, transitively, ``production-price``) default to 0. +- **Only** ``site-production-capacity``: the mirror image, for production. +- **Only** ``site-power-capacity``: a *hard* constraint at that capacity, with ``site-consumption-capacity`` and ``site-production-capacity`` both set equal to it, and ``consumption-price``/``production-price`` defaulting to 0. + +For any combination of explicitly given fields, these rules apply per direction (consumption/production) independently, filling in only the fields not already determined by a given field. +As a safety net, ``consumption-price`` still defaults to 0 if it remains unset after applying the rules above, since the scheduler requires a resolvable consumption price. + +.. note:: Setting ``relax-constraints`` to ``False`` on a commodity context that ends up with a smart-defaulted 0 hard capacity can make the schedule infeasible; FlexMeasures logs a warning in that case. + + .. _flex_models_and_schedulers: The flex-models & corresponding schedulers diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 0339f2da0f..cf15dda15c 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -476,8 +476,16 @@ def fill_grid_connection_defaults(self, data: dict, original_data: dict, **kwarg we derive sensible defaults from *which* of those five fields were explicitly given (inspecting the original input, not post-default-fill presence). - Precedence (single-field triggers; see the plan in `documentation/changelog.rst` - for cases 13/14): + A price given for a direction (consumption or production) implies a grid + connection in that direction, with an unlimited capacity unless a capacity + is also given; a capacity given for a direction (without a price) implies a + 0 price in that direction; and anything not implied by a given field + defaults to "no connection" (0 capacity, as a soft constraint). The + exception is `site-power-capacity` given on its own, which sets a *hard* + (symmetric) capacity limit instead. See :ref:`commodity_context_defaults` + for the full user-facing explanation, including worked examples. + + Precedence (single-field triggers): 1. None of the five given (e.g. just `{"commodity": "gas"}`): no grid connection at all. `site-consumption-capacity` and @@ -498,17 +506,13 @@ def fill_grid_connection_defaults(self, data: dict, original_data: dict, **kwarg equal to it (no breach price is filled in, so the constraint stays hard); `consumption-price` and `production-price` default to 0. - For any *combination* of explicitly given fields, only the fields not - determined by one of the rules above are filled in, applying the same rules - per direction (consumption / production) independently: a price given for a - direction implies a grid connection in that direction (its capacity is left - unlimited unless also given explicitly); a capacity given for a direction - (and no price) implies a 0 price in that direction. The "hard constraint" - rule (6) only applies when `site-power-capacity` is the *sole* field given. As a safety net (since the scheduler requires a resolvable consumption price), `consumption-price` defaults to 0 if still unset after applying the rules above (`production-price` already falls back to `consumption-price` at the scheduler level, so no separate safety net is needed for it). + + A commodity context with no user-given price fields does not trip a spurious cross-currency error against a differently-currencied portfolio; + its 0-price/breach-price fields instead inherit the portfolio's real currency where determinable (from a top-level price or a sibling commodity context). """ has_consumption_price = "consumption-price" in original_data