Add PLEXOS custom-constraint extraction script#105
Merged
Conversation
The v7.5 IASR workbook no longer carries the information needed to build ISPyPSA's custom constraints; their full definitions only exist in AEMO's published PLEXOS model. This script extracts them into intermediate CSVs for the templater to consume, as a repeatable step that can be re-run when AEMO publishes a new model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
EllieKallmier
approved these changes
May 26, 2026
Member
EllieKallmier
left a comment
There was a problem hiding this comment.
yeah looks solid to me, I'm just getting into playing with plexosdb and less familiar with the plexos data structures and stuff so perhaps some of the detail is lost on me lol. But the tests and assertions throughout the script make lots of sense!
Testing my understanding here - this script doesn't get distributed with the package, but the CSV files output by the script will. So the script is like a dev tool to make a (probably) repeated process quicker, testable, and generally less prone to human error re: translating XML constraints to a CSV file (in this case).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
scripts/extract_plexos_constraints.py, which extracts the custom constraint definitions ISPyPSA needs — the REZ export-limit constraints and one gas-generation constraint — from an AEMO PLEXOS XML model into intermediate CSVs undersrc/ispypsa/templater/plexos/<version>/.Why: earlier versions of the IASR workbook contained the information needed to build these constraints; the v7.5 workbook does not. The full definitions (LHS coefficients, RHS values, etc.) still exist in AEMO's published PLEXOS model, so this script extracts them from the PLEXOS XML. It is a repeatable step that can be re-run when AEMO publishes a new model.
How:
plexosdbloads the XML into an in-memory SQLite database, and a five-step query pipeline (constraint objects → memberships → data values → dates → tags) is merged into three long-format tables:constraints,lhs_terms,rhs_values. PLEXOS names, property names and the sense encoding are preserved verbatim; translation to ISPyPSA conventions is left to a follow-up templater PR. The module docstring documents the PLEXOS data model, which is otherwise difficult to review.Validation: the PLEXOS XML is an undocumented relational-database dump, and the script's correctness depends on assumptions about its schema that cannot be fully verified from within the script. As a safeguard, before writing any output the script asserts a set of structural invariants against the real model — properties that must hold if those assumptions are correct (each constraint has exactly one Sense, an RHS and LHS terms; no bands; no unresolved references; and similar). These are not synthetic unit tests, which would only re-encode the assumptions being tested. A further cross-check against the simpler constraints still present in the IASR workbook is planned for the templater PR.
Includes a workaround for a
plexosdbbug: AEMO'st_data.uidvalues overflow SQLite's 64-bit integers (reported upstream as issue #135).Unit tests for the pure helpers are in
tests/test_scripts/. The committed CSVs are the current 7.5 extract.