docs: switch to source-types in tutorial, and explain the source filters

documentation/api/notation.rst

@@ -64,6 +64,23 @@ Unless stated otherwise, values of such fields can take one of the following for
 
   The unit of the data is specified on the sensor.
 
+  A sensor reference can optionally include a source filter, so it keeps pointing at the right data even when multiple sources (e.g. a forecast and a schedule) record beliefs on the same sensor:
+
+  .. code-block:: json
+
+     {
+         "site-consumption-capacity": {"sensor": 55, "source-types": ["forecaster"]}
+     }
+
+  The supported filter keys are:
+
+  - ``source-types`` / ``exclude-source-types``: include or exclude sources by type (e.g. ``"forecaster"``, ``"scheduler"``, ``"user"``). **Recommended** over a specific ``source`` or ``sources`` ID, because forecasters and schedulers are versioned — a version bump gives new data a new source ID, but the source-type stays the same, so filters based on it don't need updating.
+  - ``source-account``: a list of account IDs, to filter by the account(s) linked to data sources. Useful in multi-tenant setups where several accounts run their own forecasters or schedulers.
+  - ``sources``: a list of specific data source IDs.
+  - ``source``: a single specific data source ID.
+
+  This is the same source filtering mechanism described under :ref:`sources`, just scoped to sensor references inside flex-model/flex-context fields rather than GET data endpoints.
+
 
 Timeseries
 ^^^^^^^^^^

documentation/changelog.rst

@@ -24,6 +24,7 @@ Infrastructure / Support
 * Prepare the ``device_scheduler`` to deal with commitments per device group [see `PR #1934 <https://www.github.com/FlexMeasures/flexmeasures/pull/1934>`_]
 * Support storing encrypted connection secrets on organisations and assets, including utility functions, encryption key configuration, CLI commands to set and delete secrets, and UI tables that show stored secret names and optional expiration times without exposing their values [see `PR #2236 <https://www.github.com/FlexMeasures/flexmeasures/pull/2236>`_]
 * Documentation section on the modelling choice for recording measurements, forecasts and schedules under one or multiple sensors [see `PR #2217 <https://www.github.com/FlexMeasures/flexmeasures/pull/2217>`_]
+* Document source filters better, and make use of the source-types filter in the PV curtailment tutorial [see `PR #2261 <https://www.github.com/FlexMeasures/flexmeasures/pull/2261>`_]
 * Document suggested cloud architecture [see `PR #2245 <https://www.github.com/FlexMeasures/flexmeasures/pull/2245>`_]
 * Document the ``TRUSTED_HOSTS`` setting to safeguard against host poisoning from clients [see `PR #2246 <https://www.github.com/FlexMeasures/flexmeasures/pull/2246>`_]
 * Document the various ways to inspect a (scheduling) job [see `PR #2247 <https://www.github.com/FlexMeasures/flexmeasures/pull/2247>`_]

documentation/concepts/data-model.rst

@@ -152,6 +152,13 @@ A common modeling choice is whether measurements, forecasts and schedules for an
 Using a single sensor is advisable when these data represent different beliefs about the same physical quantity at the same connection point.
 In that case, keeping them on one sensor makes it explicit that measurements, forecasts and schedules all refer to the same events, and therefore should share the same unit, event resolution and sign convention.
 Different origins of the data can still be distinguished through their data source.
+Sensor references in flex-model/flex-context fields can filter by source, e.g. to point a field at forecast data only, even after a schedule has been written to the same sensor:
+
+.. code-block:: json
+
+   {"sensor": 3, "exclude-source-types": ["scheduler"]}
+
+See :ref:`variable_quantities` for the full set of source filter options.
 
 Using separate sensors is advisable when scheduled power should be modeled separately from measured or forecast power.
 This can be useful when you want to avoid filtering by source in queries, when schedules should use a different sign convention, when they should be stored at a different resolution, or when the scheduled quantity has a different operational meaning from the measured quantity.

documentation/tut/toy-example-multiasset-curtailment.rst

@@ -25,6 +25,11 @@ We start by curtailing the PV asset only.
 To make the PV asset curtailable, we tell FlexMeasures that the PV (represented by sensor 3) can only pick production values between 0 and the production forecast recorded on sensor 3.
 We store the resulting schedule on sensor 3, as well (the FlexMeasures UI will still be able to distinguish forecasts from schedules).
 
+Since sensor 3 will end up holding both the forecast and the schedule, ``production-capacity`` needs to reference only the forecast, not the schedule this very run is about to write.
+We do this with a source filter: ``{"sensor": 3, "source-types": ["forecaster"]}`` scopes the reference to data recorded by a "forecaster"-type source, so it keeps pointing at the forecast even after the schedule lands on the same sensor.
+We filter by source *type* rather than a specific source ID because forecasters and schedulers are versioned — a version bump gives new data a new source ID, but the source-type stays the same, so this reference doesn't need updating when that happens.
+See :ref:`variable_quantities` for the full set of source filter options.
+
 Also, we want to create a situation with negative prices, so curtailment makes sense. We can pass into the ``flex-context`` a price profile with negative prices during some hours of the day.
 
 .. tabs::
@@ -50,7 +55,7 @@ Also, we want to create a situation with negative prices, so curtailment makes s
             $ # Scheduling only the PV sensor
             $ docker exec -it flexmeasures-server-1 flexmeasures add schedule --sensor 3 \
             --start ${TOMORROW}T07:00+01:00 --duration PT12H \
-            --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}'\
+            --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source-types": ["forecaster"]}}'\
             --flex-context tutorial3-priceprofile-flex-context.json 
 
     .. tab:: API
@@ -66,7 +71,7 @@ Also, we want to create a situation with negative prices, so curtailment makes s
                 "flex-model": [
                     {
                         "consumption-capacity": "0 kW",
-                        "production-capacity": {"sensor": 3, "source": 4},
+                        "production-capacity": {"sensor": 3, "source-types": ["forecaster"]},
                     }
                 ],
                 "flex-context": {
@@ -104,7 +109,7 @@ Also, we want to create a situation with negative prices, so curtailment makes s
                     flex_model=[
                         {
                             "consumption-capacity": "0 kW",
-                            "production-capacity": {"sensor": 3, "source": 4},
+                            "production-capacity": {"sensor": 3, "source-types": ["forecaster"]},
                         }
                     ],
                     flex_context={
@@ -178,7 +183,7 @@ Note that we are still passing in the flex-context with block price profiles her
                 --asset 2 \
                 --start ${TOMORROW}T07:00+01:00 \
                 --duration PT12H \
-                --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\
+                --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source-types": ["forecaster"]}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\
                 --flex-context tutorial3-priceprofile-flex-context.json 
             New schedule is stored.
 
@@ -195,7 +200,7 @@ Note that we are still passing in the flex-context with block price profiles her
                     {
                         "sensor": 3,
                         "consumption-capacity": "0 kW",
-                        "production-capacity": {"sensor": 3, "source": 4},
+                        "production-capacity": {"sensor": 3, "source-types": ["forecaster"]},
                     }
                     {
                         "sensor": 2,
@@ -234,7 +239,7 @@ Note that we are still passing in the flex-context with block price profiles her
                     {
                         "sensor": 3,  # solar production (sensor ID)
                         "consumption-capacity": "0 kW",
-                        "production-capacity": {"sensor": 3, "source": 4},
+                        "production-capacity": {"sensor": 3, "source-types": ["forecaster"]},
                     },
                     {
                         "sensor": 2,  # battery power (sensor ID)