Add mxspec.py specification script#2997
Conversation
This changelist adds an `mxspec.py` script for processing MaterialX specification documents, replacing the `comparenodedefs.py` script with a more general tool that supports three subcommands: - `compare`: Compare node definitions between a specification document and a data library document, reporting differences in node sets, typed signatures, and default values. - `normalize`: Normalize table formatting and default value notation, aligning columns and converting defaults to canonical placeholders such as `__zero__` and `__one__`. - `expand`: Expand polymorphic tables into concrete per-signature tables, with an optional `--nodedefs` flag that adds nodedef name subheaders from the standard libraries. GitHub CI additionally compares the NPR Nodes specification against its data library on each commit, extending the trackable alignment history already maintained for the Standard and PBR Nodes. This work was developed as a collaboration between the AOUSD Materials Working Group and the ASWF.
jstone-lucasfilm
requested review from
anderslanglands,
kwokcb and
niklasharrysson
ashwinbhat
left a comment
ashwinbhat
left a comment
There was a problem hiding this comment.
LGTM, I was wondering if we should add some notes about markdown layout requirements since we are parsing table cells.
Following up on review notes from Ashwin Bhat, this changelist adds reporting of specification parsing warnings, such as column count mismatches, when running the `normalize` and `expand` subcommands of `mxspec.py`, matching the diagnostics already reported by `compare`.
|
This is a great point, @ashwinbhat, and I've now added clearer reporting of parser warnings across all of the commands, so that any mismatches in conventions are surfaced to the user. We could add explicit documentation of our Markdown conventions as well, but my concern would be that we're then creating two distinct homes for these conventions (the specification documents and our supporting Python scripts), which would then need to be kept in sync over time. |
There was a problem hiding this comment.
This looks good to me.
Some minor comments in matching.
A general observation:
- I'm curious if instead of markdown formatted tables it would be easier to use
html instead. - Then it's much easier to parse (especially if you tag the tables with metadata), though a bit more trouble to write.
| signatures = {} | ||
| specDefaults = {} | ||
| for nodeName, tables in nodeSections.items(): | ||
| sigs = signatures.setdefault(nodeName, set()) |
There was a problem hiding this comment.
Not sure but I think you'd want to get defaults based on the per table item's defaults ?
There was a problem hiding this comment.
Good catch, and fixed!
|
|
||
| extraInLib = tuple(sorted(libInputs - specInputs)) | ||
| extraInSpec = tuple(sorted(specInputs - libInputs)) | ||
| return libSig, extraInLib, extraInSpec |
There was a problem hiding this comment.
Doesn't this just return the first one vs comparing all matches ?
There was a problem hiding this comment.
Yes, the exact match pass above handles most cases, and this fallback pass uses fuzzy matching and returns the first one. So far, I don't believe we have any real-world cases where there's more than one fuzzy match, so this works well, but indeed we could revisit this if such a case appears in the future.
Following up on review feedback from Bernard Kwok, this changelist keys specification defaults by node and signature, mirroring the structure of data library defaults, so that each expanded signature is compared against the defaults from its own specification table.
Agreed that HTML would be easier to parse, though I'd lean towards Markdown here since the specifications are hand-authored and reviewed in pull requests, where Markdown tables remain readable in raw text and diffs. The |
jstone-lucasfilm
merged commit 38368ee
into
AcademySoftwareFoundation:main
3 participants
This changelist adds an
mxspec.pyscript for processing MaterialX specification documents, replacing thecomparenodedefs.pyscript with a more general tool that supports three subcommands:compare: Compare node definitions between a specification document and a data library document, reporting differences in node sets, typed signatures, and default values.normalize: Normalize table formatting and default value notation, aligning columns and converting defaults to canonical placeholders such as__zero__and__one__.expand: Expand polymorphic tables into concrete per-signature tables, with an optional--nodedefsflag that adds nodedef name subheaders from the standard libraries.GitHub CI additionally compares the NPR Nodes specification against its data library on each commit, extending the trackable alignment history already maintained for the Standard and PBR Nodes.
This work was developed as a collaboration between the AOUSD Materials Working Group and the ASWF.