oeds: update mastr to english table names from open-mastr

[maurerle]

User description

Description

When setting up the OEDS using the latest version of the mastr crawler: https://github.com/open-energy-data-server/open-energy-data-server/blob/main/oeds/crawler/mastr.py
we are now using open-mastr which uses english table names instead of the German default table names.
We therefore need to adjust the crawler as well.

For this, we did rerun the crawler and validated its usage to evaluate the compared renewable timeseries data as well as total generation capacities.

Checklist

• Documentation updated (docstrings, READMEs, user guides, inline comments, docs folder updates, etc.)
• New unit/integration tests added (if applicable)
• Changes noted in release notes (if any)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

Additional Notes (optional)

---

PR Type

Enhancement, Bug fix

---

Description

• Migrates MaStR queries to open-mastr tables

• Adds time-aware asset filtering in loaders

• Fixes postcode and solar parsing issues

• Updates demand, weather, and static mappings

---

Diagram Walkthrough

flowchart LR
  loader["OEDS loader"]
  filters["Time-aware asset filters"]
  infra["Infrastructure interface"]
  mastr["open-mastr tables"]
  mappings["String-based MaStR mappings"]
  series["Renewable and storage series"]

  loader -- "passes simulation window" --> filters
  filters -- "constrains queries" --> infra
  infra -- "reads" --> mastr
  mappings -- "translate labels" --> infra
  infra -- "returns" --> series

Loading

File Walkthrough

	Relevant files
Enhancement	loader_oeds.py Add time-aware OEDS asset loading                                                assume/scenario/loader_oeds.py Passes created_before and stopped_after into biomass, hydro, storage, and conventional plant queries. Enables water storage loading unconditionally by removing the redundant if True wrapper. Builds conventional plant availability from both startDate and endDate. +47/-31  infrastructure.py Migrate infrastructure interface to open-mastr                      assume/scenario/oeds/infrastructure.py Migrates MaStR SQL queries to open-mastr extended tables such as solar_extended, wind_extended, combustion_extended, and storage_extended. Formats postcode filters as zero-padded five-digit strings to preserve leading zeros. Adds localized postcode centroid fallbacks for missing solar, wind, and solar-storage coordinates. Updates solar parsing for string codes, power limits, own consumption, EEG joins, and battery-linked PV systems. Switches weather access from weather to ecmwf and demand access from demand to ego_demand. +182/-191
Configuration changes	static.py Update MaStR static string mappings                                            assume/scenario/oeds/static.py Adds mastr_wind_type mappings for onshore and offshore string labels. Converts mastr_solar_codes from numeric MaStR codes to open-mastr German labels. Updates azimuth and tilt mappings to string-based MaStR values. Adds mastr_solar_power_limit factors for PV output limitation handling. +37/-27  pyproject.toml Ignore German MaStR spelling terms                                              pyproject.toml Adds vertikal and unter to the codespell ignored words list. Prevents false positives from German MaStR label strings. +1/-1

---

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit dff9853)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Issue setup() no longer initializes the MaStR translation attributes, but get_power_plant_in_area still dereferences self.energietraeger_translated. Loading conventional non-nuclear plants will raise AttributeError before executing the query. AND ev."Energietraeger" = \'{self.energietraeger_translated[fuel_type]}\' SQL Bug The storage query appends optional time filters directly after AND "Nettonennleistung" > 500 without a separating space. Because the loader now passes created_before and stopped_after, the SQL becomes 500AND ..., which will fail to parse when storage systems are loaded. f'AND "Nettonennleistung" > 500' ) if created_before:

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to dff9853
Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix active-date filtering The stopped_after filter is inverted here: it currently selects solar-storage systems that stopped before the simulation period and excludes systems still active after it. Use the same > comparison pattern as the other asset queries. assume/scenario/oeds/infrastructure.py [731-732] if stopped_after: - query += f'AND (so."DatumEndgueltigeStilllegung" IS NULL OR so."DatumEndgueltigeStilllegung" <= \'{stopped_after.isoformat()}\')' + query += f'AND (so."DatumEndgueltigeStilllegung" IS NULL OR so."DatumEndgueltigeStilllegung" > \'{stopped_after.isoformat()}\')' Suggestion importance[1-10]: 9 __ Why: The stopped_after condition in get_solar_storage_systems_in_area is indeed inverted compared with the other asset queries and would include assets stopped before the requested period. Changing <= to > is a correctness fix for active asset filtering.	High
General	Preserve utility PV behavior The new generic ownConsumption inference also applies to free_area and other solar plants, which previously were forced to 0 because their local demand is unknown. Restore that override to avoid creating phantom self-consumption demand for utility-scale PV. assume/scenario/oeds/infrastructure.py [352-364] df["ownConsumption"] = df["ownConsumption"].fillna( (df["startDate"].dt.year > 2013).astype(int) ) df["ownConsumption"] = df["ownConsumption"].apply( lambda x: 1 if "Teileinspeisung" in str(x) or "Eigenverbrauch" in str(x) or str(x) == "689" or str(x) == "1" or x is True or x == 1 else 0 ) +if solar_type in ("free_area", "other"): + df["ownConsumption"] = 0 Suggestion importance[1-10]: 6 __ Why: The suggestion accurately targets the new generic ownConsumption inference, which changes prior behavior for free_area and other PV systems. Restoring the override is a reasonable behavioral consistency fix, though the actual impact depends on how downstream code uses ownConsumption.	Low
	Avoid north-facing PV fallback 360 is treated as north-facing by pvlib when passed as a single surface_azimuth, so Ost-West systems will be simulated with the wrong orientation. Until get_solar_series explicitly splits these systems into east and west components, use a safer fallback that does not model them as north-facing. assume/scenario/oeds/static.py [48] -"Ost-West": "360", # 704 half ost, half west +"Ost-West": "180", # fallback until east/west split-orientation systems are modeled Suggestion importance[1-10]: 6 __ Why: Mapping Ost-West to 360 can cause pvlib to model these systems effectively as north-facing when used as a single surface_azimuth. Using 180 as a temporary fallback is a reasonable approximation until split-orientation modeling is implemented.	Low

---

Previous suggestions

Suggestions up to commit 08f2322

Category	Suggestion	Impact
Possible issue	Restore fuel name mapping The PR removes the setup logic that populated self.energietraeger_translated, but the new query still dereferences it, which will raise an AttributeError when loading conventional plants. Use an explicit static mapping for the supported fuel_type values before building the SQL. assume/scenario/oeds/infrastructure.py [220-231] if fuel_type != "nuclear": + fuel_name = { + "lignite": "Braunkohle", + "hard coal": "Steinkohle", + "oil": "Mineralölprodukte", + "gas": "Erdgas", + }[fuel_type] query += f""" , kwk."ThermischeNutzleistung" as "kwkPowerTherm", kwk."ElektrischeKwkLeistung" as "kwkPowerElec", ev."AnlageIstImKombibetrieb" as "combination" FROM "combustion_extended" ev LEFT JOIN "kwk" kwk ON kwk."KwkMastrNummer" = ev."KwkMastrNummer" WHERE ev."Postleitzahl" in {plz_codes_str} - AND ev."Energietraeger" = \'{self.energietraeger_translated[fuel_type]}\' + AND ev."Energietraeger" = '{fuel_name}' AND ev."Nettonennleistung" > 5000 """ Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that self.energietraeger_translated was removed from setup() but is still dereferenced in get_power_plant_in_area, causing a runtime AttributeError. The proposed static mapping is directly relevant and fixes conventional plant loading for the supported fuel_type values.	Medium
	Fix malformed SQL query The concatenated SQL fragments omit whitespace between FROM "biomass_extended" and WHERE, producing invalid SQL such as "biomass_extended"WHERE. Add trailing spaces to the fragments so the generated query remains valid when filters are appended. assume/scenario/oeds/infrastructure.py [504-513] query = ( f'SELECT "EinheitMastrNummer" as "unitID", ' f'COALESCE("Inbetriebnahmedatum", \'2018-01-01\') as "startDate", ' f'COALESCE("DatumEndgueltigeStilllegung", \'2050-01-01\') as "endDate", ' f'"Nettonennleistung" as "maxPower", ' f'COALESCE("Laengengrad", {longitude}) as "lon", ' f'COALESCE("Breitengrad", {latitude}) as "lat" ' - f'FROM "biomass_extended"' - f'WHERE "Postleitzahl" in {plz_codes_str}' + f'FROM "biomass_extended" ' + f'WHERE "Postleitzahl" in {plz_codes_str} ' ) Suggestion importance[1-10]: 8 __ Why: The SQL fragments for biomass_extended are concatenated without spaces, producing invalid SQL like "biomass_extended"WHERE and potentially )AND when filters are appended. The suggested whitespace fix is accurate and prevents a concrete query failure.	Medium
General	Normalize catalog value lookups The new open-MaStR string values are looked up with exact dictionary keys, so harmless whitespace differences like 90 Grad (vertikal) versus 90 Grad (vertikal) can crash the loader with a KeyError. Normalize both the lookup tables and incoming values before indexing. assume/scenario/oeds/infrastructure.py [366-386] -df["azimuth"] = [mastr_solar_azimuth[str(code)] for code in df["azimuthCode"]] +solar_azimuth_lookup = { + str(key).strip(): value for key, value in mastr_solar_azimuth.items() +} +solar_power_limit_lookup = { + str(key).strip(): value for key, value in mastr_solar_power_limit.items() +} + +df["azimuth"] = [ + solar_azimuth_lookup[str(code).strip()] for code in df["azimuthCode"] +] del df["azimuthCode"] # all PVs with nan have a tilt angle of 30° -df["tilt"] = [mastr_solar_azimuth[str(code)] for code in df["tiltCode"]] +df["tilt"] = [ + solar_azimuth_lookup[str(code).strip()] for code in df["tiltCode"] +] del df["tiltCode"] ... df["limit_factor"] = [ - mastr_solar_power_limit[str(code)] for code in df["limited"] + solar_power_limit_lookup[str(code).strip()] for code in df["limited"] ] Suggestion importance[1-10]: 6 __ Why: The suggestion is valid because exact string lookups in mastr_solar_azimuth and mastr_solar_power_limit can fail on harmless whitespace differences, especially given keys like 90 Grad (vertikal) . Its impact is moderate since it improves robustness against data inconsistencies, though the provided snippet is partially elided rather than a complete patch.	Low

[github-actions]

Persistent review updated to latest commit dff9853