Skip to content

Add PLEXOS custom-constraint extraction script#105

Merged
nick-gorman merged 1 commit into
mainfrom
extract-plexos-constraints
May 27, 2026
Merged

Add PLEXOS custom-constraint extraction script#105
nick-gorman merged 1 commit into
mainfrom
extract-plexos-constraints

Conversation

@nick-gorman
Copy link
Copy Markdown
Member

Adds scripts/extract_plexos_constraints.py, which extracts the custom constraint definitions ISPyPSA needs — the REZ export-limit constraints and one gas-generation constraint — from an AEMO PLEXOS XML model into intermediate CSVs under src/ispypsa/templater/plexos/<version>/.

Why: earlier versions of the IASR workbook contained the information needed to build these constraints; the v7.5 workbook does not. The full definitions (LHS coefficients, RHS values, etc.) still exist in AEMO's published PLEXOS model, so this script extracts them from the PLEXOS XML. It is a repeatable step that can be re-run when AEMO publishes a new model.

How: plexosdb loads the XML into an in-memory SQLite database, and a five-step query pipeline (constraint objects → memberships → data values → dates → tags) is merged into three long-format tables: constraints, lhs_terms, rhs_values. PLEXOS names, property names and the sense encoding are preserved verbatim; translation to ISPyPSA conventions is left to a follow-up templater PR. The module docstring documents the PLEXOS data model, which is otherwise difficult to review.

Validation: the PLEXOS XML is an undocumented relational-database dump, and the script's correctness depends on assumptions about its schema that cannot be fully verified from within the script. As a safeguard, before writing any output the script asserts a set of structural invariants against the real model — properties that must hold if those assumptions are correct (each constraint has exactly one Sense, an RHS and LHS terms; no bands; no unresolved references; and similar). These are not synthetic unit tests, which would only re-encode the assumptions being tested. A further cross-check against the simpler constraints still present in the IASR workbook is planned for the templater PR.

Includes a workaround for a plexosdb bug: AEMO's t_data.uid values overflow SQLite's 64-bit integers (reported upstream as issue #135).

Unit tests for the pure helpers are in tests/test_scripts/. The committed CSVs are the current 7.5 extract.

The v7.5 IASR workbook no longer carries the information needed to build ISPyPSA's custom constraints; their full definitions only exist in AEMO's published PLEXOS model. This script extracts them into intermediate CSVs for the templater to consume, as a repeatable step that can be re-run when AEMO publishes a new model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member

@EllieKallmier EllieKallmier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah looks solid to me, I'm just getting into playing with plexosdb and less familiar with the plexos data structures and stuff so perhaps some of the detail is lost on me lol. But the tests and assertions throughout the script make lots of sense!

Testing my understanding here - this script doesn't get distributed with the package, but the CSV files output by the script will. So the script is like a dev tool to make a (probably) repeated process quicker, testable, and generally less prone to human error re: translating XML constraints to a CSV file (in this case).

@nick-gorman nick-gorman merged commit 77dff41 into main May 27, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants