From 466163f263d80c080ff0fcacdd54781ee030cc89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicolas=20H=C3=B6ning?= Date: Mon, 6 Jul 2026 15:04:23 +0200 Subject: [PATCH 1/3] chore: make toy tutorials robust against pre-existing asset or sensor IDs, also less work for users MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/docker-build.yml | 12 +- documentation/dev/docker-compose.rst | 4 +- documentation/index.rst | 10 +- documentation/tut/building_uis.rst | 2 +- .../tut/scripts/run-tutorial-in-docker.sh | 8 +- .../tut/scripts/run-tutorial2-in-docker.sh | 18 +- .../tut/scripts/run-tutorial3-in-docker.sh | 27 +- .../tut/scripts/run-tutorial4-in-docker.sh | 14 +- .../tut/scripts/run-tutorial5-in-docker.sh | 25 +- documentation/tut/toy-example-expanded.rst | 6 +- .../tut/toy-example-from-scratch.rst | 6 +- .../toy-example-multiasset-curtailment.rst | 22 +- documentation/tut/toy-example-process.rst | 14 +- documentation/tut/toy-example-reporter.rst | 18 +- documentation/tut/toy-example-setup.rst | 17 +- flexmeasures/cli/data_add.py | 564 ++++++++++-------- .../cli/tests/test_data_add_fresh_db.py | 30 + 17 files changed, 446 insertions(+), 351 deletions(-) diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index 10cd23ef76..e2bb55af4b 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -47,7 +47,7 @@ jobs: db upgrade - name: Add toy user run: docker exec --env-file .env fm-container flexmeasures - add toy-account + add toy-account --kind battery --shell-vars >> $GITHUB_ENV - name: Generate prices dummy data run: ci/generate-dummy-price.sh - name: Copy prices dummy data @@ -55,19 +55,19 @@ jobs: - name: Add beliefs run: | docker exec --env-file .env fm-container flexmeasures \ - add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam + add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam - name: Export TOMORROW run: echo "TOMORROW=$(date --date="next day" '+%Y-%m-%d')" >> $GITHUB_ENV - name: Add schedule run: | docker exec --env-file .env fm-container flexmeasures \ - add schedule --sensor 2 --start ${TOMORROW}T07:00+01:00 \ + add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00+01:00 \ --duration PT12H --soc-at-start 50% --flex-model '{"roundtrip-efficiency": "90%"}' - name: Add Toy Account for process - run: docker exec --env-file .env fm-container flexmeasures add toy-account --kind process + run: docker exec --env-file .env fm-container flexmeasures add toy-account --kind process --shell-vars >> $GITHUB_ENV - name: Add Process schedule run: | - docker exec --env-file .env fm-container flexmeasures add schedule --sensor 5 --scheduler ProcessScheduler \ - --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --flex-context '{"consumption-price": {"sensor": 1}}' \ + docker exec --env-file .env fm-container flexmeasures add schedule --sensor ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID} --scheduler ProcessScheduler \ + --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --flex-context '{"consumption-price": {"sensor": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model "{\"duration\": \"PT4H\", \"process-type\": \"BREAKABLE\", \"power\": 0.2, \"time-restrictions\": [{\"start\": \"${TOMORROW}T15:00:00+02:00\", \"duration\": \"PT1H\"}]}" diff --git a/documentation/dev/docker-compose.rst b/documentation/dev/docker-compose.rst index 759e33ec19..f74e651f85 100644 --- a/documentation/dev/docker-compose.rst +++ b/documentation/dev/docker-compose.rst @@ -99,7 +99,7 @@ Next, we put a scheduling job in the worker's queue. This only works because we .. code-block:: bash - $ flexmeasures add schedule --sensor 2 \ + $ flexmeasures add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \ --flex-model '{"soc-min": "50 kWh"}' --as-job @@ -116,7 +116,7 @@ We'll not go into the server container this time, but simply send a command: .. code-block:: bash $ TOMORROW=$(date --date="next day" '+%Y-%m-%d') - $ docker exec -it flexmeasures-server-1 bash -c "flexmeasures show beliefs --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H" + $ docker exec -it flexmeasures-server-1 bash -c "flexmeasures show beliefs --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H" The charging/discharging schedule should be there: diff --git a/documentation/index.rst b/documentation/index.rst index e987794db8..6919aa067c 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -53,13 +53,13 @@ The main purpose of FlexMeasures is to create optimized schedules. Let's have a $ docker pull postgres; docker run --name pg-docker -e POSTGRES_PASSWORD=docker -e POSTGRES_DB=flexmeasures-db -d -p 5433:5432 postgres:latest $ export SQLALCHEMY_DATABASE_URI="postgresql://postgres:docker@127.0.0.1:5433/flexmeasures-db" && export SECRET_KEY=notsecret && export SECURITY_TOTP_SECRETS={"1":"something-secret"} $ flexmeasures db upgrade # create tables - $ flexmeasures add toy-account --kind battery # setup account incl. a user, battery (ID 2) and market (ID 1) - $ flexmeasures add beliefs --sensor 2 --source toy-user prices-tomorrow.csv --timezone utc # load prices, also possible per API - $ flexmeasures add schedule --sensor 2 \ + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" # setup account incl. a user, battery, solar and market + $ flexmeasures add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone utc # load prices, also possible per API + $ flexmeasures add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --soc-at-start 50% --flex-context '{"consumption-price": {"sensor": 1}}' \ + --soc-at-start 50% --flex-context '{"consumption-price": {"sensor": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{"roundtrip-efficiency": "90%"}' # this is also possible per API - $ flexmeasures show beliefs --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H # also visible per UI, of course + $ flexmeasures show beliefs --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H # also visible per UI, of course A short explanation of the optimization shown above: This battery is optimized to buy power cheaply and sell it at expensive times - the red-dotted line is what FlexMeasures computed to be the best schedule, given all knowledge (in this case, the prices shown in blue). However, in the example in the middle tab, the battery has to store local solar power as well (orange line), which constrains how much it can do with its capacity (that's why the schedule is limited in capacity and thus cycling less energy overall than on the left). diff --git a/documentation/tut/building_uis.rst b/documentation/tut/building_uis.rst index f326a41cd0..63c9f7515f 100644 --- a/documentation/tut/building_uis.rst +++ b/documentation/tut/building_uis.rst @@ -284,7 +284,7 @@ Now let's call this function when the HTML page is opened, to embed our chart: } The parameters we pass in describe what we want to see: all data for sensor 3 since 2022. -If you followed our :ref:`toy tutorial` on a fresh FlexMeasures installation, sensor 1 contains market prices (authenticate with the toy-user to gain access). +If you followed our :ref:`toy tutorial` on a fresh FlexMeasures installation, ``FM_TOY_PRICE_SENSOR_ID`` contains the market-price sensor ID (authenticate with the toy-user to gain access). The result looks like this in your browser: diff --git a/documentation/tut/scripts/run-tutorial-in-docker.sh b/documentation/tut/scripts/run-tutorial-in-docker.sh index 624a91adc7..c143cdaab6 100755 --- a/documentation/tut/scripts/run-tutorial-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial-in-docker.sh @@ -6,6 +6,8 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 1 (SIMPLE BATTERY SCHEDULE)..." echo "-----------------------------------------------------------------" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" + echo "[TUTORIAL-RUNNER] loading prices..." TOMORROW=$(date --date="next day" '+%Y-%m-%d') echo "Hour,Price @@ -37,14 +39,14 @@ ${TOMORROW}T23:00:00,7" > prices-tomorrow.csv docker cp prices-tomorrow.csv $CONTAINER_NAME:/app docker exec -it $CONTAINER_NAME flexmeasures add beliefs \ - --sensor 1 --source toy-user /app/prices-tomorrow.csv --timezone Europe/Amsterdam + --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user /app/prices-tomorrow.csv --timezone Europe/Amsterdam echo "[TUTORIAL-RUNNER] creating schedule ..." docker exec -it $CONTAINER_NAME flexmeasures add schedule \ - --sensor 2 \ + --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \ --flex-model '{"soc-min": "50 kWh"}' echo "[TUTORIAL-RUNNER] displaying schedule..." docker exec -it $CONTAINER_NAME flexmeasures show beliefs \ - --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H + --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H diff --git a/documentation/tut/scripts/run-tutorial2-in-docker.sh b/documentation/tut/scripts/run-tutorial2-in-docker.sh index 006c0b0e9e..6f77baf16e 100755 --- a/documentation/tut/scripts/run-tutorial2-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial2-in-docker.sh @@ -6,6 +6,8 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 2 (ADDING SOLAR FORECAST) ..." echo "------------------------------------------------------------" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" + echo "[TUTORIAL-RUNNER] loading solar production data..." TOMORROW=$(date --date="next day" '+%Y-%m-%d') @@ -42,18 +44,18 @@ echo "[TUTORIAL-RUNNER] adding source ..." docker exec -it $CONTAINER_NAME flexmeasures add source --name "toy-forecaster" --type forecaster echo "[TUTORIAL-RUNNER] adding beliefs ..." -docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor 3 --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam +docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam echo "[TUTORIAL-RUNNER] showing beliefs ..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 3 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H echo "[TUTORIAL-RUNNER] update schedule taking solar into account ..." -docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 2 \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \ - --flex-context '{"inflexible-device-sensors": [3]}' \ + --flex-context '{"inflexible-device-sensors": ['"${FM_TOY_SOLAR_SENSOR_ID}"']}' \ --flex-model '{"soc-min": "50 kWh"}' echo "[TUTORIAL-RUNNER] showing schedule ..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H #echo "" #echo "[TUTORIAL-RUNNER] DEMONSTRATING CUSTOM SCHEDULING RESOLUTION ..." @@ -63,12 +65,12 @@ docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 2 --start ${T #echo "" # #echo "[TUTORIAL-RUNNER] creating hourly-resolution schedule ..." -#docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 2 \ +#docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ # --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \ # --resolution PT1H \ -# --flex-context '{"inflexible-device-sensors": [3]}' \ +# --flex-context '{"inflexible-device-sensors": ['"${FM_TOY_SOLAR_SENSOR_ID}"']}' \ # --flex-model '{"soc-min": "50 kWh"}' # #echo "[TUTORIAL-RUNNER] showing hourly-resolution schedule ..." #echo "[TUTORIAL-RUNNER] Notice the schedule still has 15-minute data points, but values only change each hour." -#docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +#docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H diff --git a/documentation/tut/scripts/run-tutorial3-in-docker.sh b/documentation/tut/scripts/run-tutorial3-in-docker.sh index 5d18e11002..152a6daa69 100755 --- a/documentation/tut/scripts/run-tutorial3-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial3-in-docker.sh @@ -7,6 +7,7 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 3 (PV CURTAILMENT / MORE THAN ONE FLEXIBLE ASSET) ..." echo "----------------------------------------------------------------------------------------" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" TOMORROW=$(date --date="next day" '+%Y-%m-%d') @@ -25,34 +26,34 @@ echo '''{ docker cp tutorial3-priceprofile-flex-context.json $CONTAINER_NAME:/app/ # Running only the PV sensor -docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 3 \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_SOLAR_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}'\ + --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "source": 4}}'\ --flex-context tutorial3-priceprofile-flex-context.json echo "[TUTORIAL-RUNNER] showing PV schedule ..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 3 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H echo "[TUTORIAL-RUNNER] Cleaning solar data for the next steps ..." # remove all previous beliefs on PV sensor so we don't have schedules mixed in the next run (issue 1807 can help with this, so selection by source works) -docker exec -it $CONTAINER_NAME flexmeasures delete beliefs --sensor 3 --force -docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor 3 --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam +docker exec -it $CONTAINER_NAME flexmeasures delete beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --force +docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam echo "[TUTORIAL-RUNNER] Now running both battery and PV together, still using block price profiles ..." -docker exec -it $CONTAINER_NAME flexmeasures add schedule --asset 2 \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --asset ${FM_TOY_BUILDING_ASSET_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ + --flex-model '[{"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "consumption-capacity": "0 kW", "production-capacity": {"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "source": 4}}, {"sensor": '"${FM_TOY_BATTERY_SENSOR_ID}"', "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ --flex-context tutorial3-priceprofile-flex-context.json echo "[TUTORIAL-RUNNER] showing PV and battery schedule ..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 3 --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H -docker exec -it $CONTAINER_NAME flexmeasures delete beliefs --sensor 3 --force -docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor 3 --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam +docker exec -it $CONTAINER_NAME flexmeasures delete beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --force +docker exec -it $CONTAINER_NAME flexmeasures add beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --source 4 /app/solar-tomorrow.csv --timezone Europe/Amsterdam echo "[TUTORIAL-RUNNER] Now running both battery and PV together, with realistic DA prices and larger battery ..." -docker exec -it $CONTAINER_NAME flexmeasures add schedule --asset 2 \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --asset ${FM_TOY_BUILDING_ASSET_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh", "soc-max": "900kWh"}]' + --flex-model '[{"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "consumption-capacity": "0 kW", "production-capacity": {"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "source": 4}}, {"sensor": '"${FM_TOY_BATTERY_SENSOR_ID}"', "soc-at-start": "225 kWh", "soc-min": "50 kWh", "soc-max": "900kWh"}]' echo "[TUTORIAL-RUNNER] showing PV and battery schedule ..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 3 --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H diff --git a/documentation/tut/scripts/run-tutorial4-in-docker.sh b/documentation/tut/scripts/run-tutorial4-in-docker.sh index 9d9058e709..d61e129c41 100755 --- a/documentation/tut/scripts/run-tutorial4-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial4-in-docker.sh @@ -9,22 +9,22 @@ echo "------------------------------------------------------------" TOMORROW=$(date --date="next day" '+%Y-%m-%d') echo "[TUTORIAL-RUNNER] Setting up toy account with reporters..." -docker exec -it $CONTAINER_NAME flexmeasures add toy-account --kind process +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars)" echo "[TUTORIAL-RUNNER] Creating three process schedules ..." -docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 4 --scheduler ProcessScheduler \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_PROCESS_INFLEXIBLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{"consumption-price": {"sensor": 1}}' \ + --flex-context '{"consumption-price": {"sensor": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{"duration": "PT4H", "process-type": "INFLEXIBLE", "power": 0.2, "time-restrictions": [{"start": "'"${TOMORROW}"'T15:00:00+02:00", "duration": "PT1H"}]}' -docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 5 --scheduler ProcessScheduler \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{"consumption-price": {"sensor": 1}}' \ + --flex-context '{"consumption-price": {"sensor": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{"duration": "PT4H", "process-type": "BREAKABLE", "power": 0.2, "time-restrictions": [{"start": "'"${TOMORROW}"'T15:00:00+02:00", "duration": "PT1H"}]}' -docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor 6 --scheduler ProcessScheduler \ +docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_PROCESS_SHIFTABLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{"consumption-price": {"sensor": 1}}' \ + --flex-context '{"consumption-price": {"sensor": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{"duration": "PT4H", "process-type": "SHIFTABLE", "power": 0.2, "time-restrictions": [{"start": "'"${TOMORROW}"'T15:00:00+02:00", "duration": "PT1H"}]}' echo "Now visit http://localhost:5000/assets/6/graphs to see all three schedules." diff --git a/documentation/tut/scripts/run-tutorial5-in-docker.sh b/documentation/tut/scripts/run-tutorial5-in-docker.sh index 84773d18b5..97712e2893 100755 --- a/documentation/tut/scripts/run-tutorial5-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial5-in-docker.sh @@ -8,12 +8,14 @@ echo "------------------------------------------------------------" TOMORROW=$(date --date="next day" '+%Y-%m-%d') -echo "[TUTORIAL-RUNNER] Setting up toy account with reporters..." -docker exec -it $CONTAINER_NAME flexmeasures add toy-account --kind reporter +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind reporter --shell-vars)" +echo "[TUTORIAL-RUNNER] Setting up toy account with reporters..." -echo "[TUTORIAL-RUNNER] Show grid connection capacity (sensor 7)..." -docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor 7 --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --resolution PT1H +echo "[TUTORIAL-RUNNER] Show grid connection capacity ..." +docker exec -it $CONTAINER_NAME flexmeasures show beliefs --sensor ${FM_TOY_GRID_CAPACITY_SENSOR_ID} --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --resolution PT1H docker exec -it $CONTAINER_NAME flexmeasures show data-sources --show-attributes --id 6 @@ -30,9 +32,9 @@ docker cp headroom-config.json $CONTAINER_NAME:/app echo " { - 'input': [{'name': 'grid connection capacity', 'sensor': 7}, - {'name': 'PV', 'sensor': 3, 'sources': [4]}], - 'output': [{'sensor': 8}] + 'input': [{'name': 'grid connection capacity', 'sensor': ${FM_TOY_GRID_CAPACITY_SENSOR_ID}}, + {'name': 'PV', 'sensor': ${FM_TOY_SOLAR_SENSOR_ID}, 'sources': [4]}], + 'output': [{'sensor': ${FM_TOY_HEADROOM_SENSOR_ID}}] }" > headroom-parameters.json docker cp headroom-parameters.json $CONTAINER_NAME:/app @@ -46,14 +48,14 @@ docker exec -it $CONTAINER_NAME flexmeasures add report --reporter AggregatorRep echo "[TUTORIAL-RUNNER] showing reported data ..." -docker exec -it $CONTAINER_NAME bash -c "flexmeasures show beliefs --sensor 8 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H" +docker exec -it $CONTAINER_NAME bash -c "flexmeasures show beliefs --sensor ${FM_TOY_HEADROOM_SENSOR_ID} --start ${TOMORROW}T00:00:00+01:00 --duration PT24H" echo "[TUTORIAL-RUNNER] now the inflexible process ..." echo " { - 'input': [{'sensor': 4}], + 'input': [{'sensor': ${FM_TOY_PROCESS_INFLEXIBLE_SENSOR_ID}}], 'output': [{'sensor': 9}] }" > inflexible-parameters.json @@ -71,7 +73,7 @@ echo "[TUTORIAL-RUNNER] now the breakable process ..." echo " { - 'input': [{'sensor': 5}], + 'input': [{'sensor': ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID}}], 'output': [{'sensor': 10}] }" > breakable-parameters.json @@ -90,7 +92,7 @@ echo "[TUTORIAL-RUNNER] now the breakable process ..." echo " { - 'input' : [{'sensor': 6}], + 'input' : [{'sensor': ${FM_TOY_PROCESS_SHIFTABLE_SENSOR_ID}}], 'output' : [{'sensor': 11}] }" > shiftable-parameters.json @@ -102,4 +104,3 @@ docker exec -it $CONTAINER_NAME flexmeasures add report --source 6 \ echo "[TUTORIAL-RUNNER] showing reported data ..." docker exec -it $CONTAINER_NAME bash -c "flexmeasures show beliefs --sensor 11 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H" - diff --git a/documentation/tut/toy-example-expanded.rst b/documentation/tut/toy-example-expanded.rst index 3922ad9a3e..08ec37556f 100644 --- a/documentation/tut/toy-example-expanded.rst +++ b/documentation/tut/toy-example-expanded.rst @@ -66,7 +66,7 @@ Setting the data source type to "forecaster" helps FlexMeasures to visually dist $ flexmeasures add source --name "toy-forecaster" --type forecaster Added source - $ flexmeasures add beliefs --sensor 3 --source 4 solar-tomorrow.csv --timezone Europe/Amsterdam + $ flexmeasures add beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --source 4 solar-tomorrow.csv --timezone Europe/Amsterdam Successfully created beliefs The one-hour CSV data is automatically resampled to the 15-minute resolution of the sensor that is recording solar production. We can see solar production in the `FlexMeasures UI `_: @@ -92,7 +92,7 @@ This will have an effect on the available headroom for the battery, given the `` :emphasize-lines: 6 $ flexmeasures add schedule \ - --sensor 2 \ + --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 \ --duration PT12H \ --soc-at-start 50% \ @@ -115,7 +115,7 @@ This will have an effect on the available headroom for the battery, given the `` "soc-min": "50 kWh" }, "flex-context": { - "inflexible-device-sensors": [3] + "inflexible-device-sensors": [${FM_TOY_SOLAR_SENSOR_ID}] } } diff --git a/documentation/tut/toy-example-from-scratch.rst b/documentation/tut/toy-example-from-scratch.rst index e4a9d40d04..9e70320018 100644 --- a/documentation/tut/toy-example-from-scratch.rst +++ b/documentation/tut/toy-example-from-scratch.rst @@ -19,7 +19,7 @@ Make a schedule After going through the setup, we can finally create the schedule, which is the main benefit of FlexMeasures (smart real-time control). -We'll ask FlexMeasures for a schedule for our battery, specifically to store it on the (dis)charging sensor (ID 2). +We'll ask FlexMeasures for a schedule for our battery, specifically to store it on the battery power sensor we created in :ref:`tut_load_data`. To keep this short, we'll only ask for a 12-hour window starting at 7am. Finally, the scheduler should know what the state of charge of the battery is when the schedule starts (50%) and also that the SoC should never fall below 50 kWh. @@ -39,7 +39,7 @@ There is more information being used by the scheduler, such as the battery's cap .. code-block:: bash $ flexmeasures add schedule \ - --sensor 2 \ + --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 \ --duration PT12H \ --soc-at-start 50% \ @@ -66,7 +66,7 @@ There is more information being used by the scheduler, such as the battery's cap } $ flexmeasures add schedule \ - --sensor 2 \ + --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start 2024-02-04T07:00+01:00 \ --duration PT24H \ --soc-at-start 50% \ diff --git a/documentation/tut/toy-example-multiasset-curtailment.rst b/documentation/tut/toy-example-multiasset-curtailment.rst index 3fb281986a..913edcd0b5 100644 --- a/documentation/tut/toy-example-multiasset-curtailment.rst +++ b/documentation/tut/toy-example-multiasset-curtailment.rst @@ -9,7 +9,7 @@ What if the solar production is curtailable? We could turn it off when prices ar This is useful, but also an exciting next step for our modeling: Curtailing its output makes the PV inverter a flexible control, so with the battery, there are now two flexible assets. -We are now moving to multi-asset scheduling. We'll officially be scheduling the building (asset 2). +We are now moving to multi-asset scheduling. We'll officially be scheduling the building referenced by ``FM_TOY_BUILDING_ASSET_ID``. We will do this step by step. First, we demonstrate PV curtailment by itself. @@ -22,11 +22,11 @@ PV curtailment --------------------------------------- We start by curtailing the PV asset only. -To make the PV asset curtailable, we tell FlexMeasures that the PV (represented by sensor 3) can only pick production values between 0 and the production forecast recorded on sensor 3. -We store the resulting schedule on sensor 3, as well (the FlexMeasures UI will still be able to distinguish forecasts from schedules). +To make the PV asset curtailable, we tell FlexMeasures that the PV (represented by ``FM_TOY_SOLAR_SENSOR_ID``) can only pick production values between 0 and the production forecast recorded on that same sensor. +We store the resulting schedule on that solar sensor as well (the FlexMeasures UI will still be able to distinguish forecasts from schedules). -Since sensor 3 will end up holding both the forecast and the schedule, ``production-capacity`` needs to reference only the forecast, not the schedule this very run is about to write. -We do this with a source filter: ``{"sensor": 3, "source-types": ["forecaster"]}`` scopes the reference to data recorded by a "forecaster"-type source, so it keeps pointing at the forecast even after the schedule lands on the same sensor. +Since the solar sensor will end up holding both the forecast and the schedule, ``production-capacity`` needs to reference only the forecast, not the schedule this very run is about to write. +We do this with a source filter: ``{"sensor": ${FM_TOY_SOLAR_SENSOR_ID}, "source-types": ["forecaster"]}`` scopes the reference to data recorded by a "forecaster"-type source, so it keeps pointing at the forecast even after the schedule lands on the same sensor. We filter by source *type* rather than a specific source ID because forecasters and schedulers are versioned — a version bump gives new data a new source ID, but the source-type stays the same, so this reference doesn't need updating when that happens. See :ref:`variable_quantities` for the full set of source filter options. @@ -53,9 +53,9 @@ Also, we want to create a situation with negative prices, so curtailment makes s $ docker cp tutorial3-priceprofile-flex-context.json flexmeasures-server-1:/app/ $ # Scheduling only the PV sensor - $ docker exec -it flexmeasures-server-1 flexmeasures add schedule --sensor 3 \ + $ docker exec -it flexmeasures-server-1 flexmeasures add schedule --sensor ${FM_TOY_SOLAR_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source-types": ["forecaster"]}}'\ + --flex-model '{"consumption-capacity": "0 kW", "production-capacity": {"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "source-types": ["forecaster"]}}'\ --flex-context tutorial3-priceprofile-flex-context.json .. tab:: API @@ -131,7 +131,7 @@ Great. Let's see what we made: .. code-block:: bash echo "[TUTORIAL-RUNNER] showing PV schedule ..." - docker exec -it flexmeasures-server-1 flexmeasures show beliefs --sensor 3 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H + docker exec -it flexmeasures-server-1 flexmeasures show beliefs --sensor ${FM_TOY_SOLAR_SENSOR_ID} --start ${TOMORROW}T07:00:00+01:00 --duration PT12H Beliefs for Sensor 'production' (ID 3). Data spans 12 hours and starts at 2025-11-29 07:00:00+01:00. @@ -168,7 +168,7 @@ Multi-asset (building-level) Scheduling Now - we want to schedule the complete building, including two flexible assets: the battery and the PV inverter. -This means we schedule on the building level (asset 2) and include both the target sensors for both flexible assets in the flex-model. +This means we schedule on the building level and include both the target sensors for both flexible assets in the flex-model. Note that we are still passing in the flex-context with block price profiles here, as we did in the previous example - with one block of negative prices. @@ -180,10 +180,10 @@ Note that we are still passing in the flex-context with block price profiles her :emphasize-lines: 2,6 $ flexmeasures add schedule \ - --asset 2 \ + --asset ${FM_TOY_BUILDING_ASSET_ID} \ --start ${TOMORROW}T07:00+01:00 \ --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source-types": ["forecaster"]}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ + --flex-model '[{"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "consumption-capacity": "0 kW", "production-capacity": {"sensor": '"${FM_TOY_SOLAR_SENSOR_ID}"', "source-types": ["forecaster"]}}, {"sensor": '"${FM_TOY_BATTERY_SENSOR_ID}"', "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ --flex-context tutorial3-priceprofile-flex-context.json New schedule is stored. diff --git a/documentation/tut/toy-example-process.rst b/documentation/tut/toy-example-process.rst index 0a2b2b01d7..76f7f3f3bc 100644 --- a/documentation/tut/toy-example-process.rst +++ b/documentation/tut/toy-example-process.rst @@ -34,7 +34,7 @@ Before moving forward, we'll add the `process` asset and three sensors to store .. code-block:: bash - $ flexmeasures add toy-account --kind process + $ eval "$(flexmeasures add toy-account --kind process --shell-vars)" User with email toy-user@flexmeasures.io already exists in account Docker Toy Account. The sensor recording day-ahead prices is day-ahead prices (ID: 1). @@ -59,9 +59,9 @@ Now we are ready to schedule a process. Let's start with the INFLEXIBLE policy, .. code-block:: bash - $ flexmeasures add schedule --sensor 4 --scheduler ProcessScheduler \ + $ flexmeasures add schedule --sensor ${FM_TOY_PROCESS_INFLEXIBLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{\"consumption-price\": {\"sensor\": 1}}' \ + --flex-context '{\"consumption-price\": {\"sensor\": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{\"duration\": \"PT4H\", \"process-type\": \"INFLEXIBLE\", \"power\": 0.2, \"time-restrictions\": [{\"start\": \"${TOMORROW}T15:00:00+02:00\", \"duration\": \"PT1H\"}]}' \ Under the INFLEXIBLE policy, the process starts as soon as possible, in this case, coinciding with the start of the planning window. @@ -70,9 +70,9 @@ Following the INFLEXIBLE policy, we'll schedule the same 4h consumption requirem .. code-block:: bash - $ flexmeasures add schedule --sensor 5 --scheduler ProcessScheduler \ + $ flexmeasures add schedule --sensor ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{\"consumption-price\": {\"sensor\": 1}}' \ + --flex-context '{\"consumption-price\": {\"sensor\": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{\"duration\": \"PT4H\", \"process-type\": \"BREAKABLE\", \"power\": 0.2, \"time-restrictions\": [{\"start\": \"${TOMORROW}T15:00:00+02:00\", \"duration\": \"PT1H\"}]}' \ The BREAKABLE policy splits or breaks the process into blocks that can be scheduled discontinuously. The smallest possible unit is (currently) determined by the sensor's resolution. @@ -81,9 +81,9 @@ Finally, we'll schedule the process using the SHIFTABLE policy. The 4h block can .. code-block:: bash - $ flexmeasures add schedule --sensor 6 --scheduler ProcessScheduler \ + $ flexmeasures add schedule --sensor ${FM_TOY_PROCESS_SHIFTABLE_SENSOR_ID} --scheduler ProcessScheduler \ --start ${TOMORROW}T00:00:00+02:00 --duration PT24H \ - --flex-context '{\"consumption-price\": {\"sensor\": 1}}' \ + --flex-context '{\"consumption-price\": {\"sensor\": '"${FM_TOY_PRICE_SENSOR_ID}"'}}' \ --flex-model '{\"duration\": \"PT4H\", \"process-type\": \"SHIFTABLE\", \"power\": 0.2, \"time-restrictions\": [{\"start\": \"${TOMORROW}T15:00:00+02:00\", \"duration\": \"PT1H\"}]}' \ diff --git a/documentation/tut/toy-example-reporter.rst b/documentation/tut/toy-example-reporter.rst index 7457c200f3..c577172988 100644 --- a/documentation/tut/toy-example-reporter.rst +++ b/documentation/tut/toy-example-reporter.rst @@ -28,7 +28,7 @@ Just as in previous sections, we need to run the command ``flexmeasures add toy- .. code-block:: bash - $ flexmeasures add toy-account --kind reporter + $ eval "$(flexmeasures add toy-account --kind reporter --shell-vars)" Under the hood, this command is adding the following entities: - A sensor that stores the capacity of the grid connection (with a resolution of one year, so storing just one value:) ). @@ -36,7 +36,7 @@ Under the hood, this command is adding the following entities: - A `ProfitOrLossReporter` configured to use the prices that we set up in Tut. Part II. - Three sensors to register the profits/losses from running the three different processes of Tut. Part III. -.. note:: The above command should also print out the IDs of these sensors. We will use these IDs verbatim in this tutorial. +.. note:: The command above sets shell variables for the created IDs, so the next commands stay copy-pasteable on fresh installations. Let's check it out! @@ -45,7 +45,7 @@ Run the command below to show the values for our newly-created `grid connection .. code-block:: bash $ TOMORROW=$(date --date="next day" '+%Y-%m-%d') - $ flexmeasures show beliefs --sensor 7 --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --resolution PT1H + $ flexmeasures show beliefs --sensor ${FM_TOY_GRID_CAPACITY_SENSOR_ID} --start ${TOMORROW}T00:00:00+02:00 --duration PT24H --resolution PT1H Beliefs for Sensor 'grid connection capacity' (ID 7). Data spans a day and starts at 2025-06-13 00:00:00+02:00. @@ -119,9 +119,9 @@ In practice, we need to create the `config` and `parameters`: $ echo " $ { - $ 'input': [{'name': 'grid connection capacity', 'sensor': 7}, - $ {'name': 'PV', 'sensor': 3, 'sources': [4]}], - $ 'output': [{'sensor': 8}] + $ 'input': [{'name': 'grid connection capacity', 'sensor': ${FM_TOY_GRID_CAPACITY_SENSOR_ID}}, + $ {'name': 'PV', 'sensor': ${FM_TOY_SOLAR_SENSOR_ID}, 'sources': [4]}], + $ 'output': [{'sensor': ${FM_TOY_HEADROOM_SENSOR_ID}}] $ }" > headroom-parameters.json The output sensor (ID: 8) is actually the one created just to store that information - the headroom our battery has when considering solar production. @@ -179,7 +179,7 @@ Define parameters in a JSON file: $ echo " $ { - $ 'input': [{'sensor': 4}], + $ 'input': [{'sensor': ${FM_TOY_PROCESS_INFLEXIBLE_SENSOR_ID}}], $ 'output': [{'sensor': 9}] $ }" > inflexible-parameters.json @@ -207,7 +207,7 @@ Define parameters in a JSON file: $ echo " $ { - $ 'input': [{'sensor': 5}], + $ 'input': [{'sensor': ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID}}], $ 'output': [{'sensor': 10}] $ }" > breakable-parameters.json @@ -235,7 +235,7 @@ Define parameters in a JSON file: $ echo " $ { - $ 'input': [{'sensor': 6}], + $ 'input': [{'sensor': ${FM_TOY_PROCESS_SHIFTABLE_SENSOR_ID}}], $ 'output': [{'sensor': 11}] $ }" > shiftable-parameters.json diff --git a/documentation/tut/toy-example-setup.rst b/documentation/tut/toy-example-setup.rst index ddcee09284..d2f2d34ee2 100644 --- a/documentation/tut/toy-example-setup.rst +++ b/documentation/tut/toy-example-setup.rst @@ -17,10 +17,10 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s .. code-block:: bash - # setup an account with a user, assets for battery & solar and an energy market (ID 1) - $ flexmeasures add toy-account + # setup an account with a user, assets for battery & solar and an energy market + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" # load prices to optimize schedules against - $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam + $ flexmeasures add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Okay, let's get started! @@ -141,11 +141,11 @@ The data we need for our example is both structural (e.g. a company account, a u Let's create the structural data first. -FlexMeasures offers a command to create a toy account with a battery: +FlexMeasures offers a command to create a toy account with a battery and expose the relevant IDs as shell variables: .. code-block:: bash - $ flexmeasures add toy-account --kind battery + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" Generic asset type `solar` created successfully. Generic asset type `wind` created successfully. @@ -169,6 +169,8 @@ FlexMeasures offers a command to create a toy account with a battery: +This sets variables such as ``FM_TOY_PRICE_SENSOR_ID``, ``FM_TOY_BATTERY_SENSOR_ID``, ``FM_TOY_SOLAR_SENSOR_ID`` and ``FM_TOY_BUILDING_ASSET_ID`` in your current shell. + And with that, we're done with the structural data for this tutorial! If you want, you can inspect what you created in the CLI (we'll also show the UI later): @@ -348,7 +350,7 @@ This is time series data, in FlexMeasures we call *"beliefs"*. Beliefs can also .. code-block:: bash - $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam + $ flexmeasures add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Successfully created beliefs In FlexMeasures, all beliefs have a data source. Here, we use the username of the user we created earlier. We could also pass a user ID, or the name of a new data source we want to use for CLI scripts. @@ -359,7 +361,7 @@ Let's look at the price data we just loaded: .. code-block:: bash - $ flexmeasures show beliefs --sensor 1 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H + $ flexmeasures show beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --start ${TOMORROW}T00:00:00+01:00 --duration PT24H Beliefs for Sensor 'day-ahead prices' (ID 1). Data spans a day and starts at 2025-11-11 00:00:00+01:00. @@ -396,4 +398,3 @@ Again, we can also view these prices in the `FlexMeasures UI Date: Mon, 6 Jul 2026 15:25:00 +0200 Subject: [PATCH 2/3] more robustness in only evaluating output which sets our variables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Nicolas Höning --- .github/workflows/docker-build.yml | 4 +- documentation/index.rst | 2 +- .../tut/scripts/run-tutorial-in-docker.sh | 2 +- .../tut/scripts/run-tutorial2-in-docker.sh | 2 +- .../tut/scripts/run-tutorial3-in-docker.sh | 2 +- .../tut/scripts/run-tutorial4-in-docker.sh | 2 +- .../tut/scripts/run-tutorial5-in-docker.sh | 6 +- documentation/tut/toy-example-process.rst | 2 +- documentation/tut/toy-example-reporter.rst | 2 +- documentation/tut/toy-example-setup.rst | 5 +- flexmeasures/cli/data_add.py | 561 +++++++++--------- .../cli/tests/test_data_add_fresh_db.py | 5 - 12 files changed, 289 insertions(+), 306 deletions(-) diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index e2bb55af4b..46799cd0e8 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -47,7 +47,7 @@ jobs: db upgrade - name: Add toy user run: docker exec --env-file .env fm-container flexmeasures - add toy-account --kind battery --shell-vars >> $GITHUB_ENV + add toy-account --kind battery --shell-vars | grep '^FM_TOY_' >> $GITHUB_ENV - name: Generate prices dummy data run: ci/generate-dummy-price.sh - name: Copy prices dummy data @@ -65,7 +65,7 @@ jobs: add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} --start ${TOMORROW}T07:00+01:00 \ --duration PT12H --soc-at-start 50% --flex-model '{"roundtrip-efficiency": "90%"}' - name: Add Toy Account for process - run: docker exec --env-file .env fm-container flexmeasures add toy-account --kind process --shell-vars >> $GITHUB_ENV + run: docker exec --env-file .env fm-container flexmeasures add toy-account --kind process --shell-vars | grep '^FM_TOY_' >> $GITHUB_ENV - name: Add Process schedule run: | docker exec --env-file .env fm-container flexmeasures add schedule --sensor ${FM_TOY_PROCESS_BREAKABLE_SENSOR_ID} --scheduler ProcessScheduler \ diff --git a/documentation/index.rst b/documentation/index.rst index 6919aa067c..6f31d9498d 100644 --- a/documentation/index.rst +++ b/documentation/index.rst @@ -53,7 +53,7 @@ The main purpose of FlexMeasures is to create optimized schedules. Let's have a $ docker pull postgres; docker run --name pg-docker -e POSTGRES_PASSWORD=docker -e POSTGRES_DB=flexmeasures-db -d -p 5433:5432 postgres:latest $ export SQLALCHEMY_DATABASE_URI="postgresql://postgres:docker@127.0.0.1:5433/flexmeasures-db" && export SECRET_KEY=notsecret && export SECURITY_TOTP_SECRETS={"1":"something-secret"} $ flexmeasures db upgrade # create tables - $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" # setup account incl. a user, battery, solar and market + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" # setup account incl. a user, battery, solar and market $ flexmeasures add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone utc # load prices, also possible per API $ flexmeasures add schedule --sensor ${FM_TOY_BATTERY_SENSOR_ID} \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ diff --git a/documentation/tut/scripts/run-tutorial-in-docker.sh b/documentation/tut/scripts/run-tutorial-in-docker.sh index c143cdaab6..9151806406 100755 --- a/documentation/tut/scripts/run-tutorial-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial-in-docker.sh @@ -6,7 +6,7 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 1 (SIMPLE BATTERY SCHEDULE)..." echo "-----------------------------------------------------------------" -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" echo "[TUTORIAL-RUNNER] loading prices..." TOMORROW=$(date --date="next day" '+%Y-%m-%d') diff --git a/documentation/tut/scripts/run-tutorial2-in-docker.sh b/documentation/tut/scripts/run-tutorial2-in-docker.sh index 6f77baf16e..792f73ef02 100755 --- a/documentation/tut/scripts/run-tutorial2-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial2-in-docker.sh @@ -6,7 +6,7 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 2 (ADDING SOLAR FORECAST) ..." echo "------------------------------------------------------------" -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" echo "[TUTORIAL-RUNNER] loading solar production data..." diff --git a/documentation/tut/scripts/run-tutorial3-in-docker.sh b/documentation/tut/scripts/run-tutorial3-in-docker.sh index 152a6daa69..5343443654 100755 --- a/documentation/tut/scripts/run-tutorial3-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial3-in-docker.sh @@ -7,7 +7,7 @@ CONTAINER_NAME="${1:-$(basename $(pwd))-server-1}" echo "[TUTORIAL-RUNNER] RUNNING TUTORIAL 3 (PV CURTAILMENT / MORE THAN ONE FLEXIBLE ASSET) ..." echo "----------------------------------------------------------------------------------------" -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" TOMORROW=$(date --date="next day" '+%Y-%m-%d') diff --git a/documentation/tut/scripts/run-tutorial4-in-docker.sh b/documentation/tut/scripts/run-tutorial4-in-docker.sh index d61e129c41..b76ed35faa 100755 --- a/documentation/tut/scripts/run-tutorial4-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial4-in-docker.sh @@ -9,7 +9,7 @@ echo "------------------------------------------------------------" TOMORROW=$(date --date="next day" '+%Y-%m-%d') echo "[TUTORIAL-RUNNER] Setting up toy account with reporters..." -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars | grep '^FM_TOY_')" echo "[TUTORIAL-RUNNER] Creating three process schedules ..." docker exec -it $CONTAINER_NAME flexmeasures add schedule --sensor ${FM_TOY_PROCESS_INFLEXIBLE_SENSOR_ID} --scheduler ProcessScheduler \ diff --git a/documentation/tut/scripts/run-tutorial5-in-docker.sh b/documentation/tut/scripts/run-tutorial5-in-docker.sh index 97712e2893..6f34c0ff13 100755 --- a/documentation/tut/scripts/run-tutorial5-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial5-in-docker.sh @@ -8,9 +8,9 @@ echo "------------------------------------------------------------" TOMORROW=$(date --date="next day" '+%Y-%m-%d') -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars)" -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars)" -eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind reporter --shell-vars)" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind process --shell-vars | grep '^FM_TOY_')" +eval "$(docker exec -i $CONTAINER_NAME flexmeasures add toy-account --kind reporter --shell-vars | grep '^FM_TOY_')" echo "[TUTORIAL-RUNNER] Setting up toy account with reporters..." diff --git a/documentation/tut/toy-example-process.rst b/documentation/tut/toy-example-process.rst index 76f7f3f3bc..19fd43ba4b 100644 --- a/documentation/tut/toy-example-process.rst +++ b/documentation/tut/toy-example-process.rst @@ -34,7 +34,7 @@ Before moving forward, we'll add the `process` asset and three sensors to store .. code-block:: bash - $ eval "$(flexmeasures add toy-account --kind process --shell-vars)" + $ eval "$(flexmeasures add toy-account --kind process --shell-vars | grep '^FM_TOY_')" User with email toy-user@flexmeasures.io already exists in account Docker Toy Account. The sensor recording day-ahead prices is day-ahead prices (ID: 1). diff --git a/documentation/tut/toy-example-reporter.rst b/documentation/tut/toy-example-reporter.rst index c577172988..5dbc0b9ad1 100644 --- a/documentation/tut/toy-example-reporter.rst +++ b/documentation/tut/toy-example-reporter.rst @@ -28,7 +28,7 @@ Just as in previous sections, we need to run the command ``flexmeasures add toy- .. code-block:: bash - $ eval "$(flexmeasures add toy-account --kind reporter --shell-vars)" + $ eval "$(flexmeasures add toy-account --kind reporter --shell-vars | grep '^FM_TOY_')" Under the hood, this command is adding the following entities: - A sensor that stores the capacity of the grid connection (with a resolution of one year, so storing just one value:) ). diff --git a/documentation/tut/toy-example-setup.rst b/documentation/tut/toy-example-setup.rst index d2f2d34ee2..a608cabc24 100644 --- a/documentation/tut/toy-example-setup.rst +++ b/documentation/tut/toy-example-setup.rst @@ -18,7 +18,7 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s .. code-block:: bash # setup an account with a user, assets for battery & solar and an energy market - $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" # load prices to optimize schedules against $ flexmeasures add beliefs --sensor ${FM_TOY_PRICE_SENSOR_ID} --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam @@ -145,7 +145,7 @@ FlexMeasures offers a command to create a toy account with a battery and expose .. code-block:: bash - $ eval "$(flexmeasures add toy-account --kind battery --shell-vars)" + $ eval "$(flexmeasures add toy-account --kind battery --shell-vars | grep '^FM_TOY_')" Generic asset type `solar` created successfully. Generic asset type `wind` created successfully. @@ -397,4 +397,3 @@ Again, we can also view these prices in the `FlexMeasures UI