From ef9d6ea210b23e772d7af100676d9959a058fb45 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Wed, 27 May 2026 21:45:47 -0500 Subject: [PATCH 01/15] Add reverse-engineered PRD for MetaEd 4.x Documents the MetaEd product through version 4.7, covering: - VS Code extension capabilities - Build and Deploy commands - Generated artifact outputs (SQL, XSD, API metadata, handbook, dictionaries, API catalog) - Configuration system - CI/CD support - Non-functional requirements Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/PRD-4.x.md | 350 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 350 insertions(+) create mode 100644 docs/PRD-4.x.md diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md new file mode 100644 index 000000000..4f142e54d --- /dev/null +++ b/docs/PRD-4.x.md @@ -0,0 +1,350 @@ +# MetaEd Product Requirements Document — Version 4.x + +| Field | Value | +| ------------------ | -------------------------------------------------------------- | +| Status | Draft | +| Owner | Ed-Fi Alliance | +| Repository | Ed-Fi-Alliance-OSS/MetaEd-Js | +| Current Version | 4.7 | +| Platform | Visual Studio Code Extension + Node.js CLI | +| Source References | cli.md, automated-tests.md, artifact-specific-configuration.md, api-schema-catalog.md, MetaEd IDE User Guide, MetaEd for Continuous Integration | + +--- + +## 1. Product Overview + +MetaEd is a technology framework that uses an Ed-Fi-aligned domain-specific language (DSL) to auto-generate software, database, and data standard artifacts from a single source definition. The term "MetaEd" encompasses: + +- **MetaEd Language** — a lightweight DSL describing Ed-Fi data elements, their properties, and their organization into entities. Files use the `.metaed` extension. +- **MetaEd Generator** — a command-line application that compiles MetaEd language files into technical artifacts. +- **MetaEd IDE** — a Visual Studio Code extension providing an integrated development environment for authoring `.metaed` files and invoking the generator. +- **MetaEd Packages** — a collection of Node.js packages that implement the parsing, validation, enhancement, and generation pipeline. + +The Ed-Fi Alliance uses MetaEd internally to produce core components such as the Ed-Fi Standard Interchange Schema (XSD files), the Ed-Fi ODS database scripts (SQL), and technical documentation from a single source definition written in the MetaEd DSL. + +A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community members for the generation of Ed-Fi Extensions and all related artifacts. + +### 1.1 Strategic Alignment + +- **Single source of truth for data models.** All Ed-Fi technical artifacts (SQL, XSD, API metadata, documentation) are derived from one set of `.metaed` files, eliminating drift between artifacts. +- **Accelerate extension development.** Community implementers can rapidly create, validate, and build extensions to the Ed-Fi Data Standard without manual artifact authoring. +- **Support continuous integration.** MetaEd can be run headlessly in CI/CD pipelines to validate data models and produce deployment-ready artifacts on every commit. +- **Multi-version support.** A single MetaEd installation can target multiple Data Standard versions and ODS/API versions through configuration. + +### 1.2 Target Users and Personas + +- **Ed-Fi Alliance Technical Team** — maintains the core Ed-Fi Data Standard using MetaEd in Alliance Mode. Requires full control over core model files and uses advanced features like plugin configuration and CI builds. +- **Education Agency Developers** — extend the Ed-Fi Data Standard for local requirements. Primarily uses the IDE for authoring extensions and deploying artifacts to their ODS/API environment. +- **Ed-Fi Integration Partners** — build and maintain extensions on behalf of education agencies. Uses both the IDE and CI workflows to produce and validate extension artifacts. +- **Business Analysts** — review generated documentation (Data Handbook, API Catalog, Data Dictionaries) to understand the data model without reading source files. + +### 1.3 Jobs to Be Done + +- **When** I need to create or modify an Ed-Fi extension, **I want** to author `.metaed` files in a guided IDE with syntax highlighting and validation, **so that** I can quickly define my extension without learning the internals of XSD, SQL, or API metadata formats. +- **When** I am ready to test my model changes, **I want** to run a Build command that compiles all artifacts, **so that** I can verify correctness before deploying. +- **When** I need to integrate my extension with the ODS/API, **I want** to run a Deploy command that copies artifacts to the correct locations, **so that** the ODS/API build process picks them up automatically. +- **When** I maintain a CI pipeline for data model changes, **I want** to run MetaEd headlessly via Node.js, **so that** I can validate and produce artifacts on every commit without a graphical environment. +- **When** I need to understand the API shape of a data model, **I want** a generated API Catalog spreadsheet, **so that** I can catalog resources and their properties for comparison and documentation purposes. + +--- + +## 2. Enterprise and System Context + +### 2.1 External Systems and Integrations + +- **Ed-Fi ODS/API** — the primary consumer of MetaEd-generated artifacts (SQL scripts, XSD, API metadata). MetaEd's Deploy command copies artifacts into the ODS/API source tree for its build process. +- **Ed-Fi Data Standard (Model packages)** — versioned npm packages (e.g., `@edfi/ed-fi-model-4.0`) containing the core `.metaed` files. Extensions reference these as dependencies. +- **Visual Studio Code** — host environment for the MetaEd IDE extension. +- **Node.js** — runtime environment for the MetaEd generator and all packages. +- **Ed-Fi npm Registry** — Azure DevOps-hosted npm feed for distributing MetaEd packages. +- **ANTLR 4.6** — parser generator framework used for the MetaEd grammar. + +### 2.2 Deployment Boundaries + +- MetaEd runs entirely on the developer's local machine (or CI agent). It does not communicate with external services at runtime. +- Generated artifacts are files on disk. MetaEd does not directly deploy to running servers or databases. +- The Deploy command writes to a local clone of the ODS/API source repositories. + +--- + +## 3. Functional Requirements + +### 3.1 VS Code Extension (MetaEd IDE) + +- **FR-IDE-1**: The extension SHALL provide syntax highlighting for `.metaed` files. +- **FR-IDE-2**: The extension SHALL provide real-time semantic validation, reporting errors and warnings in the VS Code Problems panel. +- **FR-IDE-3**: Errors SHALL be displayed with red indicators on files/folders in the Explorer tree; warnings SHALL be displayed with gold indicators. +- **FR-IDE-4**: The extension SHALL provide a **Build** command (accessible from the editor toolbar) that compiles all `.metaed` files in the workspace into output artifacts. +- **FR-IDE-5**: The extension SHALL provide a **Deploy** command that copies build artifacts to the configured ODS/API source directory. +- **FR-IDE-6**: The extension SHALL expose workspace-level settings for: + - Accepted License (boolean, required) + - Target ODS/API Version (string) + - ODS/API Deployment Directory (path) + - Suppress Delete on Deploy (boolean) + - Alliance Mode (boolean) +- **FR-IDE-7**: The extension SHALL require acceptance of the Ed-Fi License Agreement before Build or Deploy can execute. +- **FR-IDE-8**: The extension SHALL support multi-folder workspaces, allowing users to open both the core data model and one or more extension projects simultaneously. +- **FR-IDE-9**: Build output SHALL be written to a `MetaEdOutput` directory in the last workspace folder. +- **FR-IDE-10**: The extension SHALL display a notification on build success or failure, with error details available in the output panel. + +### 3.2 Build Command + +- **FR-BUILD-1**: The Build command SHALL compile all `.metaed` files from configured project paths into output artifacts. +- **FR-BUILD-2**: The Build command SHALL accept configuration via a JSON config file (`-c` / `--config` flag) or VS Code workspace settings. +- **FR-BUILD-3**: The Build command SHALL require the `-a` / `--accept-license` flag (or equivalent workspace setting) to acknowledge the Ed-Fi License Agreement. +- **FR-BUILD-4**: The Build command SHALL support the `--defaultPluginTechVersion` flag to specify the target ODS/API version in semver format. +- **FR-BUILD-5**: The Build command SHALL support the `--suppressPrereleaseVersion` flag (default: true) to suppress prerelease identifiers in version output. +- **FR-BUILD-6**: The Build command SHALL exit with code `1` on failure, enabling CI pipelines to detect build errors. +- **FR-BUILD-7**: The Build command SHALL produce output in the following directory structure under the configured `artifactDirectory`: + - `Documentation/DataDictionary/` + - `Documentation/Ed-Fi-Handbook/` + - `Documentation/UDM/` + - `EdFi/ApiMetadata/` + - `EdFi/Database/` + - `EdFi/Interchange/` + - `EdFi/XSD/` + +### 3.3 Deploy Command + +- **FR-DEPLOY-1**: The Deploy command SHALL copy built artifacts from the artifact directory into the correct locations within the ODS/API source tree. +- **FR-DEPLOY-2**: The Deploy command SHALL accept configuration via a JSON config file or command-line switches (`-s` source, `-t` target). +- **FR-DEPLOY-3**: The Deploy command SHALL require the `-a` / `--accept-license` flag. +- **FR-DEPLOY-4**: The Deploy command SHALL support the `--core` flag to deploy core artifacts in addition to extensions. +- **FR-DEPLOY-5**: The Deploy command SHALL support the `--suppressDelete` flag to prevent deletion of the SupportingArtifacts deployment folder. +- **FR-DEPLOY-6**: The Deploy command SHALL produce output in the following structure under the configured `deployDirectory`: + - `Ed-Fi-ODS/Standard/Metadata/` + - `Ed-Fi-ODS/Database/Data/EdFi/` + - `Ed-Fi-ODS/Database/Structure/EdFi/` + - `Ed-Fi-ODS/Database/Schemas/Interchange/` + - `Ed-Fi-ODS/Database/Schemas/XSD/` + - `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Standard/SupportingArtifacts/Metadata/ApiModel.json` +- **FR-DEPLOY-7**: The `deployDirectory` SHALL point to the parent directory containing `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folders. + +### 3.4 Configuration + +- **FR-CFG-1**: The system SHALL support a JSON configuration file with the following top-level structure: + - `metaEdConfiguration.artifactDirectory` — output directory for build artifacts + - `metaEdConfiguration.deployDirectory` — target for deploy + - `metaEdConfiguration.projects[]` — array of project definitions (projectName, namespaceName, projectExtension, projectVersion, description) + - `metaEdConfiguration.projectPaths[]` — array of filesystem paths to `.metaed` source directories + - `metaEdConfiguration.pluginConfigDirectories[]` — array of paths to plugin configuration files + - `metaEdConfiguration.defaultPluginTechVersion` — target ODS/API version + - `metaEdConfiguration.allianceMode` — enables editing of core model files + - `metaEdConfiguration.suppressPrereleaseVersion` — suppresses prerelease version identifiers +- **FR-CFG-2**: A configuration file named `metaed.conf.json` SHALL be ignored by git (convention). +- **FR-CFG-3**: The system SHALL support artifact-specific configuration files (`{pluginShortName}.config.json`) placed in plugin configuration directories. +- **FR-CFG-4**: Artifact-specific configuration files SHALL support matching rules based on entity type, namespace, core/extensions flag, and entity name. +- **FR-CFG-5**: Artifact-specific configuration data SHALL be validated using Joi schemas registered by each plugin. +- **FR-CFG-6**: Configuration matching options (`namespace`, `core`, `extensions`) SHALL be mutually exclusive within a single match rule. + +### 3.5 Build Outputs + +#### 3.5.1 SQL Scripts + +- **FR-SQL-1**: The Build SHALL generate SQL scripts compatible with both Microsoft SQL Server and PostgreSQL for creating an operational data store (ODS). +- **FR-SQL-2**: Generated SQL files SHALL include: + - `0001-Schemas.sql` — core database schema definitions + - `0001-EXTENSION-{namespace}-Schemas.sql` — extension schema definitions + - `0004-Tables.sql` — core database tables + - `0004-EXTENSION-{namespace}.sql` — extension database tables + - `0009-IdColumnUniqueIndexes.sql` — indexing for API support on core tables + - `0009-EXTENSION-{namespace}-IdColumnUniqueIndexes.sql` — indexing for extension tables +- **FR-SQL-3**: Files prefixed with "EXTENSION" SHALL be additive to the standard ODS/API script set; files without "EXTENSION" SHALL replace the corresponding shipped files. + +#### 3.5.2 XSD Files + +- **FR-XSD-1**: The Build SHALL generate XSD files for defining the structure of bulk data exchange and bulk loading of Ed-Fi data. +- **FR-XSD-2**: Generated XSD files SHALL include: + - `Ed-Fi-Core.xsd` — the core Ed-Fi XSD + - `EXTENSION-Ed-Fi-Extended-Core.xsd` — the extension XSD + - `SchemaAnnotation.xsd` — annotation definitions + - `Interchange-*.xsd` — core interchange schemas + - `EXTENSION-Interchange-*.xsd` — extension interchange schemas + +#### 3.5.3 API Metadata + +- **FR-API-1**: The Build SHALL generate API metadata files used by the ODS/API build process, including: + - `DomainMetadata.xml` + - `DomainMetadata-Extension.xml` + - `EdOrgReferenceMetadata.xml` + - `ApiModel.json` + +#### 3.5.4 Data Handbook + +- **FR-HBK-1**: The Build SHALL generate a comprehensive data handbook combining core and extension definitions. +- **FR-HBK-2**: The handbook SHALL be produced in multiple formats: + - `Ed-Fi-Handbook.xlsx` — Excel workbook + - `Ed-Fi-Data-Handbook-Index.html` — interactive, searchable HTML file (self-contained, can be hosted on a web server or run locally) + +#### 3.5.5 Data Dictionaries + +- **FR-DICT-1**: The Build SHALL generate data dictionaries documenting generated structures: + - `SqlDataDictionary.xlsx` — documentation on core and extension database tables + - `XmlDataDictionary.xlsx` — documentation on core and extension XSD elements + +#### 3.5.6 API Catalog + +- **FR-CAT-1**: The Build SHALL generate an API Catalog spreadsheet (`Ed-Fi-API-Catalog.xlsx`) describing the data model as it will be represented in the Open API specification. +- **FR-CAT-2**: The API Catalog SHALL contain a **Resources** worksheet with columns: Project, Version, Resource Name, Resource Description, Domains. +- **FR-CAT-3**: The API Catalog SHALL contain a **Properties** worksheet with columns: Project, Version, Resource Name, Property Name, Property Description, Data Type, Min Length, Max Length, Validation RegEx, Is Identity Key, Is Nullable, Is Required. +- **FR-CAT-4**: Properties belonging to embedded Common types or References SHALL be prefixed with a dot-separated path indicating their origin (e.g., `address.streetNumberName`). +- **FR-CAT-5**: The `id` property SHALL be omitted from the Properties worksheet. +- **FR-CAT-6**: The API Catalog output SHALL appear in an `Ed-Fi-API-Catalog` folder within the artifact directory. + +### 3.6 Extension Project Structure + +- **FR-EXT-1**: An extension project SHALL contain a `package.json` with a `metaEdProject` field specifying `projectName` and `projectVersion`. +- **FR-EXT-2**: The `projectName` SHALL start with an uppercase alphabetic character and contain at least two alphabetic characters. +- **FR-EXT-3**: The `projectVersion` SHALL follow SemVer 2.0.0 format. +- **FR-EXT-4**: Extension projects SHALL use the standard MetaEd directory structure: + - `Association/`, `Choice/`, `Common/`, `Descriptor/`, `Domain/`, `DomainEntity/`, `Enumeration/`, `Interchange/`, `Shared/` + +### 3.7 Continuous Integration Support + +- **FR-CI-1**: MetaEd SHALL be operable directly through Node.js without requiring VS Code or a graphical environment. +- **FR-CI-2**: The `@edfi/metaed-console` package SHALL be installable from the Ed-Fi npm registry for use in CI pipelines. +- **FR-CI-3**: The `@edfi/metaed-odsapi-deploy` package SHALL be installable from the Ed-Fi npm registry for deploy in CI pipelines. +- **FR-CI-4**: Both packages SHALL support the same configuration file format used by the IDE. + +--- + +## 4. Non-Functional Requirements + +### Compatibility + +- **NFR-COMPAT-1**: MetaEd SHALL run on Windows, macOS, and Linux (any platform supported by VS Code and Node.js). +- **NFR-COMPAT-2**: MetaEd SHALL support Node.js LTS versions current at the time of release. +- **NFR-COMPAT-3**: The MetaEd IDE extension SHALL be compatible with the current stable release of Visual Studio Code. +- **NFR-COMPAT-4**: Generated SQL scripts SHALL be compatible with both Microsoft SQL Server and PostgreSQL. +- **NFR-COMPAT-5**: MetaEd SHALL support multiple Ed-Fi Data Standard versions (configurable via `projectVersion` and `defaultPluginTechVersion`). + +### Licensing + +- **NFR-LIC-1**: Usage of the Ed-Fi Unifying Data Model SHALL require explicit acceptance of the Ed-Fi License Agreement. +- **NFR-LIC-2**: The system SHALL not execute Build or Deploy commands until the license is accepted. + +### Reliability + +- **NFR-REL-1**: MetaEd runs continuously in the IDE environment. The system SHALL NOT crash or throw unhandled exceptions during normal operation; failures SHALL be reported as validation messages. +- **NFR-REL-2**: Build failures SHALL be reported with clear, actionable error messages and a non-zero exit code. + +### Performance + +- **NFR-PERF-1**: Real-time validation in the IDE SHOULD complete within 2 seconds of a file save for typical extension projects. +- **NFR-PERF-2**: Full build of a standard-sized project SHOULD complete within 60 seconds on typical developer hardware. + +### Security + +- **NFR-SEC-1**: MetaEd SHALL NOT transmit data to external services during operation. +- **NFR-SEC-2**: MetaEd SHALL NOT store or require credentials for normal operation. + +### Observability + +- **NFR-OBS-1**: Build and deploy operations SHALL produce log output with sufficient detail to diagnose failures. +- **NFR-OBS-2**: The IDE SHALL display build progress notifications and final status. + +### SDLC + +- **NFR-SDLC-1**: The repository SHALL maintain automated CI checks including TypeScript type checking, ESLint, and unit tests on every pull request. +- **NFR-SDLC-2**: Test coverage SHALL include both unit tests and database integration tests (SQL Server and PostgreSQL). + +--- + +## 5. System Architecture + +### 5.1 Component Packages + +| Package | Description | +| ------- | ----------- | +| `metaed-core` | _(to be described in a future iteration)_ | +| `metaed-console` | _(to be described in a future iteration)_ | +| `metaed-default-plugins` | _(to be described in a future iteration)_ | +| `metaed-odsapi-deploy` | _(to be described in a future iteration)_ | +| `metaed-odsapi-deploy-console` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-api-catalog` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-api-schema` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-data-catalog` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-handbook` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-changequery` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-changequery-postgresql` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-changequery-sqlserver` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-postgresql` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-recordownership` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-recordownership-postgresql` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-recordownership-sqlserver` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-relational` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-ods-sqlserver` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-odsapi` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-sql-dictionary` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-unified` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-unified-advanced` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-xml-dictionary` | _(to be described in a future iteration)_ | +| `metaed-plugin-edfi-xsd` | _(to be described in a future iteration)_ | + +### 5.2 Runtime and Deployment + +- **Runtime**: Node.js (with TypeScript compiled to ES2017/CommonJS) +- **Package Management**: npm workspaces managed by Lerna +- **Grammar**: ANTLR 4.6 (JavaScript target) +- **Distribution**: npm packages published to the Ed-Fi Azure DevOps npm registry; VS Code extension published to the Visual Studio Marketplace + +### 5.3 Processing Pipeline + +The system follows a sequential pipeline: **Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write**. The DSL and pipeline internals are documented separately and are out of scope for this PRD. + +--- + +## 6. Out of Scope and Known Limitations + +### Explicit Exclusions + +- **The MetaEd DSL syntax and semantics** — documented in the MetaEd Language Specification. +- **Internal compilation pipeline details** — the Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write pipeline is documented in developer-facing architecture docs, not in this product requirements document. +- **Individual package descriptions** — package purposes will be documented in a future iteration of this PRD. +- **ODS/API runtime behavior** — MetaEd generates artifacts for the ODS/API; it does not define ODS/API runtime requirements. +- **Data Standard content** — the actual entities, descriptors, and associations in any Data Standard version are authored in `.metaed` files and are not part of this product's requirements. + +### Known Limitations + +- MetaEd does not validate that the ODS/API deployment directory is correctly structured before deploying; it assumes the target follows the expected repository layout. +- The IDE's workspace file includes user-specific paths (e.g., to `node_modules` inside the extension directory) that are not portable across machines and should not be committed to version control. +- Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent and costly errors. +- The Deploy command deletes the `SupportingArtifacts` folder by default unless `--suppressDelete` is specified. + +### Assumptions + +- Users have Node.js and Visual Studio Code installed. +- Users with CI workflows have access to the Ed-Fi npm registry. +- The Ed-Fi License Agreement must be accepted before any artifact generation. + +--- + +## 7. Open Questions and Decision Log + +| ID | Question | Status | Decision | +| -- | -------- | ------ | -------- | +| OQ-1 | Should the PRD specify minimum supported Node.js version? | Open | Currently deferred to release notes. | +| OQ-2 | Should the Deploy command validate the target directory structure before writing? | Open | — | +| OQ-3 | Should Alliance Mode be configurable per-project rather than globally? | Open | — | + +--- + +## 8. Glossary + +| Term | Definition | +| ---- | ---------- | +| **MetaEd** | Umbrella term for the DSL, the compiler, the IDE, and the supporting packages. | +| **MetaEd DSL** | The domain-specific language used to define Ed-Fi data models. | +| **MetaEd IDE** | The Visual Studio Code extension providing authoring and build/deploy capabilities. | +| **MetaEd Generator** | The command-line application that compiles `.metaed` files into artifacts. | +| **ODS** | Operational Data Store — the database backing an Ed-Fi API. | +| **ODS/API** | The Ed-Fi ODS / API platform that serves the Ed-Fi REST API. | +| **Data Standard** | The Ed-Fi Unifying Data Model, versioned and expressed in `.metaed` files. | +| **Extension** | A user-defined addition to the core Ed-Fi Data Standard. | +| **Alliance Mode** | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | +| **Plugin** | A MetaEd package that provides validators, enhancers, and/or generators for a specific output format. | +| **Artifact** | Any file generated by the MetaEd build process (SQL, XSD, JSON, HTML, Excel). | +| **Deploy** | The act of copying generated artifacts into the ODS/API source tree for its build process to consume. | +| **Namespace** | A scoping mechanism for MetaEd projects (e.g., `EdFi` for core, custom names for extensions). | +| **Tech Version** | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | +| **SemVer** | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | From a236d898c8534ea6e27fb95ed54b6bf13bd7d217 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Wed, 27 May 2026 21:49:43 -0500 Subject: [PATCH 02/15] docs: add package descriptions to PRD architecture table Reverse-engineered purpose of each package from source code and package.json metadata. Removed metaed-plugin-edfi-data-catalog (does not exist in the repository). Updated out-of-scope section to reflect that package descriptions are now documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/PRD-4.x.md | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index 4f142e54d..8c08b5408 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -256,30 +256,29 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community | Package | Description | | ------- | ----------- | -| `metaed-core` | _(to be described in a future iteration)_ | -| `metaed-console` | _(to be described in a future iteration)_ | -| `metaed-default-plugins` | _(to be described in a future iteration)_ | -| `metaed-odsapi-deploy` | _(to be described in a future iteration)_ | -| `metaed-odsapi-deploy-console` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-api-catalog` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-api-schema` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-data-catalog` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-handbook` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-changequery` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-changequery-postgresql` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-changequery-sqlserver` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-postgresql` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-recordownership` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-recordownership-postgresql` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-recordownership-sqlserver` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-relational` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-ods-sqlserver` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-odsapi` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-sql-dictionary` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-unified` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-unified-advanced` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-xml-dictionary` | _(to be described in a future iteration)_ | -| `metaed-plugin-edfi-xsd` | _(to be described in a future iteration)_ | +| `metaed-core` | Core engine providing parsing, model, pipeline, validation, enhancement, and generation APIs. | +| `metaed-console` | CLI entry point for running the MetaEd build process from the command line. | +| `metaed-default-plugins` | Returns the default ordered set of MetaEd plugins used for a standard build. | +| `metaed-odsapi-deploy` | Provides deployment tasks for copying generated artifacts into an Ed-Fi ODS/API target directory. | +| `metaed-odsapi-deploy-console` | CLI entry point for running the ODS/API deployment workflow. | +| `metaed-plugin-edfi-api-catalog` | Generates an Excel catalog of API resources and their properties from the MetaEd API schema. | +| `metaed-plugin-edfi-api-schema` | Builds the enhanced Ed-Fi API schema model used by downstream plugins and generators. | +| `metaed-plugin-edfi-handbook` | Generates the Ed-Fi Data Handbook outputs (HTML and Excel) from the MetaEd model. | +| `metaed-plugin-edfi-ods-changequery` | Provides the common change-query model logic for Ed-Fi ODS/API change tracking. | +| `metaed-plugin-edfi-ods-changequery-postgresql` | Generates PostgreSQL-specific change-query database artifacts. | +| `metaed-plugin-edfi-ods-changequery-sqlserver` | Generates SQL Server-specific change-query database artifacts. | +| `metaed-plugin-edfi-ods-postgresql` | Generates Ed-Fi ODS database artifacts for PostgreSQL. | +| `metaed-plugin-edfi-ods-recordownership` | Generates the record-ownership database artifacts used by the ODS/API. | +| `metaed-plugin-edfi-ods-recordownership-postgresql` | Generates record-ownership database artifacts for PostgreSQL. | +| `metaed-plugin-edfi-ods-recordownership-sqlserver` | Generates record-ownership database artifacts for SQL Server. | +| `metaed-plugin-edfi-ods-relational` | Provides the shared relational ODS generation logic used by database-specific plugins. | +| `metaed-plugin-edfi-ods-sqlserver` | Generates Ed-Fi ODS database artifacts for SQL Server. | +| `metaed-plugin-edfi-odsapi` | Generates the Ed-Fi ODS/API metadata artifacts and coordinates overall API build outputs. | +| `metaed-plugin-edfi-sql-dictionary` | Generates the Ed-Fi SQL data dictionary output. | +| `metaed-plugin-edfi-unified` | Provides the shared unified model foundation (validators, enhancers) used by multiple plugins. | +| `metaed-plugin-edfi-unified-advanced` | Provides the advanced unified model layer for richer cross-plugin generation. | +| `metaed-plugin-edfi-xml-dictionary` | Generates the Ed-Fi XML data dictionary output. | +| `metaed-plugin-edfi-xsd` | Generates Ed-Fi XSD schema artifacts. | ### 5.2 Runtime and Deployment @@ -300,7 +299,7 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu - **The MetaEd DSL syntax and semantics** — documented in the MetaEd Language Specification. - **Internal compilation pipeline details** — the Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write pipeline is documented in developer-facing architecture docs, not in this product requirements document. -- **Individual package descriptions** — package purposes will be documented in a future iteration of this PRD. +- **Internal package implementation details** — individual package internals are documented in developer-facing architecture docs. - **ODS/API runtime behavior** — MetaEd generates artifacts for the ODS/API; it does not define ODS/API runtime requirements. - **Data Standard content** — the actual entities, descriptors, and associations in any Data Standard version are authored in `.metaed` files and are not part of this product's requirements. From 9045bc43587cef460b54fcb7d6688324eed1657a Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 15:36:39 -0500 Subject: [PATCH 03/15] Draft PRD for MetaEd from user perspective --- docs/PRD-4.x.md | 198 +++++++++++++++++++++++++----------------------- 1 file changed, 102 insertions(+), 96 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index 8c08b5408..9beb30878 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -1,28 +1,26 @@ # MetaEd Product Requirements Document — Version 4.x -| Field | Value | -| ------------------ | -------------------------------------------------------------- | -| Status | Draft | -| Owner | Ed-Fi Alliance | -| Repository | Ed-Fi-Alliance-OSS/MetaEd-Js | -| Current Version | 4.7 | -| Platform | Visual Studio Code Extension + Node.js CLI | -| Source References | cli.md, automated-tests.md, artifact-specific-configuration.md, api-schema-catalog.md, MetaEd IDE User Guide, MetaEd for Continuous Integration | +- Status: Done +- Owner: Stephen Fuqua +- Repository: Ed-Fi-Alliance-OSS/MetaEd-Js +- Current Version: 4.7 +- Platform: Visual Studio Code Extension + Node.js CLI ---- +> [!TIP] Document Scope +> This PRD defines the product requirements for MetaEd version 4.x primarily from the end-user's perspective, which includes the new JavaScript/TypeScript implementation of the MetaEd generator and the accompanying VS Code extension. It covers the core features, functional and non-functional requirements, system architecture, and known limitations of this major release. The document is intended to guide development and ensure alignment with user needs and strategic goals. +> +> It does _not cover_ details about the MetaEd DSL syntax and semantics, the internal compilation pipeline, or details on how to construct the API model files used by ODS/API code generation and by the Ed-Fi API v8 deployment and runtime processes. ## 1. Product Overview MetaEd is a technology framework that uses an Ed-Fi-aligned domain-specific language (DSL) to auto-generate software, database, and data standard artifacts from a single source definition. The term "MetaEd" encompasses: -- **MetaEd Language** — a lightweight DSL describing Ed-Fi data elements, their properties, and their organization into entities. Files use the `.metaed` extension. +- **MetaEd Language** — a lightweight DSL describing data elements, their properties, and their organization into entities and domains (the Ed-Fi Unified Data Model). Files use the `.metaed` extension. - **MetaEd Generator** — a command-line application that compiles MetaEd language files into technical artifacts. - **MetaEd IDE** — a Visual Studio Code extension providing an integrated development environment for authoring `.metaed` files and invoking the generator. - **MetaEd Packages** — a collection of Node.js packages that implement the parsing, validation, enhancement, and generation pipeline. -The Ed-Fi Alliance uses MetaEd internally to produce core components such as the Ed-Fi Standard Interchange Schema (XSD files), the Ed-Fi ODS database scripts (SQL), and technical documentation from a single source definition written in the MetaEd DSL. - -A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community members for the generation of Ed-Fi Extensions and all related artifacts. +The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSON API metadata, SQL scripts, and technical documentation from a single source definition written in the MetaEd DSL. Externally, Ed-Fi Community members use MetaEd to create and maintain Ed-Fi Extensions, which are additive data models that extend the core Ed-Fi Data Standard to meet local needs. ### 1.1 Strategic Alignment @@ -40,19 +38,52 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community ### 1.3 Jobs to Be Done -- **When** I need to create or modify an Ed-Fi extension, **I want** to author `.metaed` files in a guided IDE with syntax highlighting and validation, **so that** I can quickly define my extension without learning the internals of XSD, SQL, or API metadata formats. -- **When** I am ready to test my model changes, **I want** to run a Build command that compiles all artifacts, **so that** I can verify correctness before deploying. -- **When** I need to integrate my extension with the ODS/API, **I want** to run a Deploy command that copies artifacts to the correct locations, **so that** the ODS/API build process picks them up automatically. -- **When** I maintain a CI pipeline for data model changes, **I want** to run MetaEd headlessly via Node.js, **so that** I can validate and produce artifacts on every commit without a graphical environment. -- **When** I need to understand the API shape of a data model, **I want** a generated API Catalog spreadsheet, **so that** I can catalog resources and their properties for comparison and documentation purposes. +#### JTBD 1.1. Author data model extensions in a guided IDE + +**Personas**: Education Agency Developers, Ed-Fi Integration Partners, Ed-Fi Alliance Technical Team + +**When** I need to create or modify an Ed-Fi extension, **I want** to author `.metaed` files in a guided IDE with syntax highlighting and validation, **so that** I can quickly define my extension without learning the internals of XSD, SQL, or API metadata formats. + +**How MetaEd Helps**: The VS Code extension provides syntax highlighting, real-time semantic validation, and inline error reporting so authors get immediate feedback on model correctness. + +#### JTBD 1.2. Compile model changes into deployable artifacts + +**Personas**: Education Agency Developers, Ed-Fi Integration Partners, Ed-Fi Alliance Technical Team + +**When** I am ready to test my model changes, **I want** to run a Build command that compiles all artifacts, **so that** I can verify correctness before deploying. + +**How MetaEd Helps**: The Build command transforms `.metaed` source files into SQL scripts, XSD schemas, API metadata, interchange definitions, and documentation in a single operation. + +#### JTBD 1.3. Deploy extension artifacts to the ODS/API + +**Personas**: Education Agency Developers, Ed-Fi Integration Partners + +**When** I need to integrate my extension with the ODS/API, **I want** to run a Deploy command that copies artifacts to the correct locations, **so that** the ODS/API build process picks them up automatically. + +**How MetaEd Helps**: The Deploy command knows the expected directory structure of the ODS/API source tree and copies each artifact type to its correct destination. + +#### JTBD 1.4. Validate and produce artifacts in CI + +**Personas**: Ed-Fi Alliance Technical Team, Ed-Fi Integration Partners + +**When** I maintain a CI pipeline for data model changes, **I want** to run MetaEd headlessly via Node.js, **so that** I can validate and produce artifacts on every commit without a graphical environment. ---- +**How MetaEd Helps**: The `metaed-console` CLI accepts a JSON configuration file and returns a non-zero exit code on failure, integrating cleanly with CI/CD pipelines. + +#### JTBD 1.5. Understand the API shape of a data model + +**Personas**: Business Analysts, Education Agency Developers, Ed-Fi Integration Partners + +**When** I need to understand the API shape of a data model, **I want** a generated API Catalog spreadsheet, **so that** I can catalog resources and their properties for comparison and documentation purposes. + +**How MetaEd Helps**: The API Catalog plugin generates an Excel workbook with Resources and Properties tabs that describe every endpoint and field derived from the model. ## 2. Enterprise and System Context ### 2.1 External Systems and Integrations - **Ed-Fi ODS/API** — the primary consumer of MetaEd-generated artifacts (SQL scripts, XSD, API metadata). MetaEd's Deploy command copies artifacts into the ODS/API source tree for its build process. +- **Ed-Fi API** - successor to the ODS/API, which also consumes MetaEd-generated artifacts, particularly the API metadata. - **Ed-Fi Data Standard (Model packages)** — versioned npm packages (e.g., `@edfi/ed-fi-model-4.0`) containing the core `.metaed` files. Extensions reference these as dependencies. - **Visual Studio Code** — host environment for the MetaEd IDE extension. - **Node.js** — runtime environment for the MetaEd generator and all packages. @@ -65,8 +96,6 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - Generated artifacts are files on disk. MetaEd does not directly deploy to running servers or databases. - The Deploy command writes to a local clone of the ODS/API source repositories. ---- - ## 3. Functional Requirements ### 3.1 VS Code Extension (MetaEd IDE) @@ -131,11 +160,10 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - `metaEdConfiguration.defaultPluginTechVersion` — target ODS/API version - `metaEdConfiguration.allianceMode` — enables editing of core model files - `metaEdConfiguration.suppressPrereleaseVersion` — suppresses prerelease version identifiers -- **FR-CFG-2**: A configuration file named `metaed.conf.json` SHALL be ignored by git (convention). -- **FR-CFG-3**: The system SHALL support artifact-specific configuration files (`{pluginShortName}.config.json`) placed in plugin configuration directories. -- **FR-CFG-4**: Artifact-specific configuration files SHALL support matching rules based on entity type, namespace, core/extensions flag, and entity name. -- **FR-CFG-5**: Artifact-specific configuration data SHALL be validated using Joi schemas registered by each plugin. -- **FR-CFG-6**: Configuration matching options (`namespace`, `core`, `extensions`) SHALL be mutually exclusive within a single match rule. +- **FR-CFG-2**: The system MAY support artifact-specific configuration files (`{pluginShortName}.config.json`) placed in plugin configuration directories. +- **FR-CFG-3**: Artifact-specific configuration files SHALL support matching rules based on entity type, namespace, core/extensions flag, and entity name. +- **FR-CFG-4**: Artifact-specific configuration data SHALL be validated using Joi schemas registered by each plugin. +- **FR-CFG-5**: Configuration matching options (`namespace`, `core`, `extensions`) SHALL be mutually exclusive within a single match rule. ### 3.5 Build Outputs @@ -168,17 +196,18 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - `DomainMetadata-Extension.xml` - `EdOrgReferenceMetadata.xml` - `ApiModel.json` +- **FR-API-2**: The Build SHALL generate API metadata files used by the Ed-Fi API v8 build and runtime processes, including: + - `ApiSchema{prefix}.json` #### 3.5.4 Data Handbook - **FR-HBK-1**: The Build SHALL generate a comprehensive data handbook combining core and extension definitions. -- **FR-HBK-2**: The handbook SHALL be produced in multiple formats: - - `Ed-Fi-Handbook.xlsx` — Excel workbook - - `Ed-Fi-Data-Handbook-Index.html` — interactive, searchable HTML file (self-contained, can be hosted on a web server or run locally) +- **FR-HBK-2**: The handbook SHALL be produced in an HTML format: `Ed-Fi-Data-Handbook-Index.html` — interactive, searchable HTML file (self-contained, can be hosted on a web server or run locally) +- **FR-HBK-3**: The handbook MAY be produced in an Excel format: `Ed-Fi-Handbook.xlsx` #### 3.5.5 Data Dictionaries -- **FR-DICT-1**: The Build SHALL generate data dictionaries documenting generated structures: +- **FR-DICT-1**: The Build MAY generate data dictionaries documenting generated structures: - `SqlDataDictionary.xlsx` — documentation on core and extension database tables - `XmlDataDictionary.xlsx` — documentation on core and extension XSD elements @@ -206,8 +235,6 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - **FR-CI-3**: The `@edfi/metaed-odsapi-deploy` package SHALL be installable from the Ed-Fi npm registry for deploy in CI pipelines. - **FR-CI-4**: Both packages SHALL support the same configuration file format used by the IDE. ---- - ## 4. Non-Functional Requirements ### Compatibility @@ -222,6 +249,7 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - **NFR-LIC-1**: Usage of the Ed-Fi Unifying Data Model SHALL require explicit acceptance of the Ed-Fi License Agreement. - **NFR-LIC-2**: The system SHALL not execute Build or Deploy commands until the license is accepted. +- **NFR-LIC-3**: The IDE itself SHALL be freely available on the Visual Studio Marketplace under the Apache-2.0 license. ### Reliability @@ -248,37 +276,35 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community - **NFR-SDLC-1**: The repository SHALL maintain automated CI checks including TypeScript type checking, ESLint, and unit tests on every pull request. - **NFR-SDLC-2**: Test coverage SHALL include both unit tests and database integration tests (SQL Server and PostgreSQL). ---- - ## 5. System Architecture ### 5.1 Component Packages -| Package | Description | -| ------- | ----------- | -| `metaed-core` | Core engine providing parsing, model, pipeline, validation, enhancement, and generation APIs. | -| `metaed-console` | CLI entry point for running the MetaEd build process from the command line. | -| `metaed-default-plugins` | Returns the default ordered set of MetaEd plugins used for a standard build. | -| `metaed-odsapi-deploy` | Provides deployment tasks for copying generated artifacts into an Ed-Fi ODS/API target directory. | -| `metaed-odsapi-deploy-console` | CLI entry point for running the ODS/API deployment workflow. | -| `metaed-plugin-edfi-api-catalog` | Generates an Excel catalog of API resources and their properties from the MetaEd API schema. | -| `metaed-plugin-edfi-api-schema` | Builds the enhanced Ed-Fi API schema model used by downstream plugins and generators. | -| `metaed-plugin-edfi-handbook` | Generates the Ed-Fi Data Handbook outputs (HTML and Excel) from the MetaEd model. | -| `metaed-plugin-edfi-ods-changequery` | Provides the common change-query model logic for Ed-Fi ODS/API change tracking. | -| `metaed-plugin-edfi-ods-changequery-postgresql` | Generates PostgreSQL-specific change-query database artifacts. | -| `metaed-plugin-edfi-ods-changequery-sqlserver` | Generates SQL Server-specific change-query database artifacts. | -| `metaed-plugin-edfi-ods-postgresql` | Generates Ed-Fi ODS database artifacts for PostgreSQL. | -| `metaed-plugin-edfi-ods-recordownership` | Generates the record-ownership database artifacts used by the ODS/API. | -| `metaed-plugin-edfi-ods-recordownership-postgresql` | Generates record-ownership database artifacts for PostgreSQL. | -| `metaed-plugin-edfi-ods-recordownership-sqlserver` | Generates record-ownership database artifacts for SQL Server. | -| `metaed-plugin-edfi-ods-relational` | Provides the shared relational ODS generation logic used by database-specific plugins. | -| `metaed-plugin-edfi-ods-sqlserver` | Generates Ed-Fi ODS database artifacts for SQL Server. | -| `metaed-plugin-edfi-odsapi` | Generates the Ed-Fi ODS/API metadata artifacts and coordinates overall API build outputs. | -| `metaed-plugin-edfi-sql-dictionary` | Generates the Ed-Fi SQL data dictionary output. | -| `metaed-plugin-edfi-unified` | Provides the shared unified model foundation (validators, enhancers) used by multiple plugins. | -| `metaed-plugin-edfi-unified-advanced` | Provides the advanced unified model layer for richer cross-plugin generation. | -| `metaed-plugin-edfi-xml-dictionary` | Generates the Ed-Fi XML data dictionary output. | -| `metaed-plugin-edfi-xsd` | Generates Ed-Fi XSD schema artifacts. | +| Package | Description | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| `metaed-core` | Core engine providing parsing, model, pipeline, validation, enhancement, and generation APIs. | +| `metaed-console` | CLI entry point for running the MetaEd build process from the command line. | +| `metaed-default-plugins` | Returns the default ordered set of MetaEd plugins used for a standard build. | +| `metaed-odsapi-deploy` | Provides deployment tasks for copying generated artifacts into an Ed-Fi ODS/API target directory. | +| `metaed-odsapi-deploy-console` | CLI entry point for running the ODS/API deployment workflow. | +| `metaed-plugin-edfi-api-catalog` | Generates an Excel catalog of API resources and their properties from the MetaEd API schema. | +| `metaed-plugin-edfi-api-schema` | Builds the enhanced Ed-Fi API schema model used by downstream plugins and generators. | +| `metaed-plugin-edfi-handbook` | Generates the Ed-Fi Data Handbook outputs (HTML and Excel) from the MetaEd model. | +| `metaed-plugin-edfi-ods-changequery` | Provides the common change-query model logic for Ed-Fi ODS/API change tracking. | +| `metaed-plugin-edfi-ods-changequery-postgresql` | Generates PostgreSQL-specific change-query database artifacts. | +| `metaed-plugin-edfi-ods-changequery-sqlserver` | Generates SQL Server-specific change-query database artifacts. | +| `metaed-plugin-edfi-ods-postgresql` | Generates Ed-Fi ODS database artifacts for PostgreSQL. | +| `metaed-plugin-edfi-ods-recordownership` | Generates the record-ownership database artifacts used by the ODS/API. | +| `metaed-plugin-edfi-ods-recordownership-postgresql` | Generates record-ownership database artifacts for PostgreSQL. | +| `metaed-plugin-edfi-ods-recordownership-sqlserver` | Generates record-ownership database artifacts for SQL Server. | +| `metaed-plugin-edfi-ods-relational` | Provides the shared relational ODS generation logic used by database-specific plugins. | +| `metaed-plugin-edfi-ods-sqlserver` | Generates Ed-Fi ODS database artifacts for SQL Server. | +| `metaed-plugin-edfi-odsapi` | Generates the Ed-Fi ODS/API metadata artifacts and coordinates overall API build outputs. | +| `metaed-plugin-edfi-sql-dictionary` | Generates the Ed-Fi SQL data dictionary output. | +| `metaed-plugin-edfi-unified` | Provides the shared unified model foundation (validators, enhancers) used by multiple plugins. | +| `metaed-plugin-edfi-unified-advanced` | Provides the advanced unified model layer for richer cross-plugin generation. | +| `metaed-plugin-edfi-xml-dictionary` | Generates the Ed-Fi XML data dictionary output. | +| `metaed-plugin-edfi-xsd` | Generates Ed-Fi XSD schema artifacts. | ### 5.2 Runtime and Deployment @@ -291,8 +317,6 @@ A version of MetaEd called the MetaEd IDE is freely available to Ed-Fi community The system follows a sequential pipeline: **Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write**. The DSL and pipeline internals are documented separately and are out of scope for this PRD. ---- - ## 6. Out of Scope and Known Limitations ### Explicit Exclusions @@ -310,40 +334,22 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu - Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent and costly errors. - The Deploy command deletes the `SupportingArtifacts` folder by default unless `--suppressDelete` is specified. -### Assumptions - -- Users have Node.js and Visual Studio Code installed. -- Users with CI workflows have access to the Ed-Fi npm registry. -- The Ed-Fi License Agreement must be accepted before any artifact generation. - ---- - -## 7. Open Questions and Decision Log - -| ID | Question | Status | Decision | -| -- | -------- | ------ | -------- | -| OQ-1 | Should the PRD specify minimum supported Node.js version? | Open | Currently deferred to release notes. | -| OQ-2 | Should the Deploy command validate the target directory structure before writing? | Open | — | -| OQ-3 | Should Alliance Mode be configurable per-project rather than globally? | Open | — | - ---- - ## 8. Glossary -| Term | Definition | -| ---- | ---------- | -| **MetaEd** | Umbrella term for the DSL, the compiler, the IDE, and the supporting packages. | -| **MetaEd DSL** | The domain-specific language used to define Ed-Fi data models. | -| **MetaEd IDE** | The Visual Studio Code extension providing authoring and build/deploy capabilities. | -| **MetaEd Generator** | The command-line application that compiles `.metaed` files into artifacts. | -| **ODS** | Operational Data Store — the database backing an Ed-Fi API. | -| **ODS/API** | The Ed-Fi ODS / API platform that serves the Ed-Fi REST API. | -| **Data Standard** | The Ed-Fi Unifying Data Model, versioned and expressed in `.metaed` files. | -| **Extension** | A user-defined addition to the core Ed-Fi Data Standard. | -| **Alliance Mode** | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | -| **Plugin** | A MetaEd package that provides validators, enhancers, and/or generators for a specific output format. | -| **Artifact** | Any file generated by the MetaEd build process (SQL, XSD, JSON, HTML, Excel). | -| **Deploy** | The act of copying generated artifacts into the ODS/API source tree for its build process to consume. | -| **Namespace** | A scoping mechanism for MetaEd projects (e.g., `EdFi` for core, custom names for extensions). | -| **Tech Version** | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | -| **SemVer** | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | +| Term | Definition | +| -------------------- | ----------------------------------------------------------------------------------------------------------------- | +| **MetaEd** | Umbrella term for the DSL, the compiler, the IDE, and the supporting packages. | +| **MetaEd DSL** | The domain-specific language used to define Ed-Fi data models. | +| **MetaEd IDE** | The Visual Studio Code extension providing authoring and build/deploy capabilities. | +| **MetaEd Generator** | The command-line application that compiles `.metaed` files into artifacts. | +| **ODS** | Operational Data Store — the database backing an Ed-Fi API. | +| **ODS/API** | The Ed-Fi ODS / API platform that serves the Ed-Fi REST API. | +| **Data Standard** | The Ed-Fi Unifying Data Model, versioned and expressed in `.metaed` files. | +| **Extension** | A user-defined addition to the core Ed-Fi Data Standard. | +| **Alliance Mode** | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | +| **Plugin** | A MetaEd package that provides validators, enhancers, and/or generators for a specific output format. | +| **Artifact** | Any file generated by the MetaEd build process (SQL, XSD, JSON, HTML, Excel). | +| **Deploy** | The act of copying generated artifacts into the ODS/API source tree for its build process to consume. | +| **Namespace** | A scoping mechanism for MetaEd projects (e.g., `EdFi` for core, custom names for extensions). | +| **Tech Version** | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | +| **SemVer** | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | From f6d4ec9165a388330755bb4e4340993a23c68111 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 16:16:48 -0500 Subject: [PATCH 04/15] Fix some hallucinations --- docs/PRD-4.x.md | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index 9beb30878..e0f78632a 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -124,14 +124,7 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO - **FR-BUILD-4**: The Build command SHALL support the `--defaultPluginTechVersion` flag to specify the target ODS/API version in semver format. - **FR-BUILD-5**: The Build command SHALL support the `--suppressPrereleaseVersion` flag (default: true) to suppress prerelease identifiers in version output. - **FR-BUILD-6**: The Build command SHALL exit with code `1` on failure, enabling CI pipelines to detect build errors. -- **FR-BUILD-7**: The Build command SHALL produce output in the following directory structure under the configured `artifactDirectory`: - - `Documentation/DataDictionary/` - - `Documentation/Ed-Fi-Handbook/` - - `Documentation/UDM/` - - `EdFi/ApiMetadata/` - - `EdFi/Database/` - - `EdFi/Interchange/` - - `EdFi/XSD/` +- **FR-BUILD-7**: The Build command SHALL produce output in a configured output directory; details about the output files depend on the plugins used at build time. ### 3.3 Deploy Command @@ -139,15 +132,18 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO - **FR-DEPLOY-2**: The Deploy command SHALL accept configuration via a JSON config file or command-line switches (`-s` source, `-t` target). - **FR-DEPLOY-3**: The Deploy command SHALL require the `-a` / `--accept-license` flag. - **FR-DEPLOY-4**: The Deploy command SHALL support the `--core` flag to deploy core artifacts in addition to extensions. -- **FR-DEPLOY-5**: The Deploy command SHALL support the `--suppressDelete` flag to prevent deletion of the SupportingArtifacts deployment folder. -- **FR-DEPLOY-6**: The Deploy command SHALL produce output in the following structure under the configured `deployDirectory`: - - `Ed-Fi-ODS/Standard/Metadata/` - - `Ed-Fi-ODS/Database/Data/EdFi/` - - `Ed-Fi-ODS/Database/Structure/EdFi/` - - `Ed-Fi-ODS/Database/Schemas/Interchange/` - - `Ed-Fi-ODS/Database/Schemas/XSD/` - - `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Standard/SupportingArtifacts/Metadata/ApiModel.json` -- **FR-DEPLOY-7**: The `deployDirectory` SHALL point to the parent directory containing `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folders. +- **FR-DEPLOY-5**: The Deploy command SHALL support the `--suppressDelete` flag to prevent deletion of existing extension artifact directories before deploying. +- **FR-DEPLOY-6**: When the `--core` flag is specified (ODS/API version ≥ 7.0), the Deploy command SHALL copy core artifacts into the configured `deployDirectory` under `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{dataStandardVersion}/Artifacts/`: + - `ApiMetadata/` → `Metadata/` + - `Database/SQLServer/ODS/Data/` → `MsSql/Data/Ods/` + - `Database/SQLServer/ODS/Structure/` → `MsSql/Structure/Ods/` + - `Database/PostgreSQL/ODS/Data/` → `PgSql/Data/Ods/` + - `Database/PostgreSQL/ODS/Structure/` → `PgSql/Structure/Ods/` + - `Interchange/` → `Schemas/` + - `XSD/` → `Schemas/` +- **FR-DEPLOY-7**: The Deploy command SHALL copy extension artifacts into the configured `deployDirectory` under `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Versions/{projectVersion}/Standard/{dataStandardVersion}/Artifacts/` using the same mapping as FR-DEPLOY-6. +- **FR-DEPLOY-8**: The `deployDirectory` SHALL point to the parent directory containing `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folders. +- **FR-DEPLOY-9**: Unless `--suppressDelete` is specified, the Deploy command SHALL remove existing extension artifact directories (`Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Artifacts`) before deploying new artifacts. ### 3.4 Configuration @@ -191,13 +187,8 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO #### 3.5.3 API Metadata -- **FR-API-1**: The Build SHALL generate API metadata files used by the ODS/API build process, including: - - `DomainMetadata.xml` - - `DomainMetadata-Extension.xml` - - `EdOrgReferenceMetadata.xml` - - `ApiModel.json` -- **FR-API-2**: The Build SHALL generate API metadata files used by the Ed-Fi API v8 build and runtime processes, including: - - `ApiSchema{prefix}.json` +- **FR-API-1**: The Build SHALL generate API metadata files used by the ODS/API build process, including: `ApiModel{prefix}.json` +- **FR-API-2**: The Build SHALL generate API metadata files used by the Ed-Fi API v8+ build and runtime processes, including: `ApiSchema{prefix}.json` #### 3.5.4 Data Handbook @@ -332,7 +323,7 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu - MetaEd does not validate that the ODS/API deployment directory is correctly structured before deploying; it assumes the target follows the expected repository layout. - The IDE's workspace file includes user-specific paths (e.g., to `node_modules` inside the extension directory) that are not portable across machines and should not be committed to version control. - Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent and costly errors. -- The Deploy command deletes the `SupportingArtifacts` folder by default unless `--suppressDelete` is specified. +- The Deploy command deletes existing extension artifact directories by default unless `--suppressDelete` is specified. ## 8. Glossary From 06a0cf7096e891af07869e67e486b0f0b3277026 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 16:22:44 -0500 Subject: [PATCH 05/15] Some manual clarifications --- docs/PRD-4.x.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index e0f78632a..1827a091d 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -310,7 +310,7 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu ## 6. Out of Scope and Known Limitations -### Explicit Exclusions +### Explicit Exclusions from the Document - **The MetaEd DSL syntax and semantics** — documented in the MetaEd Language Specification. - **Internal compilation pipeline details** — the Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write pipeline is documented in developer-facing architecture docs, not in this product requirements document. @@ -318,7 +318,7 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu - **ODS/API runtime behavior** — MetaEd generates artifacts for the ODS/API; it does not define ODS/API runtime requirements. - **Data Standard content** — the actual entities, descriptors, and associations in any Data Standard version are authored in `.metaed` files and are not part of this product's requirements. -### Known Limitations +### Known Limitations of the Current Implementation - MetaEd does not validate that the ODS/API deployment directory is correctly structured before deploying; it assumes the target follows the expected repository layout. - The IDE's workspace file includes user-specific paths (e.g., to `node_modules` inside the extension directory) that are not portable across machines and should not be committed to version control. @@ -334,7 +334,8 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu | **MetaEd IDE** | The Visual Studio Code extension providing authoring and build/deploy capabilities. | | **MetaEd Generator** | The command-line application that compiles `.metaed` files into artifacts. | | **ODS** | Operational Data Store — the database backing an Ed-Fi API. | -| **ODS/API** | The Ed-Fi ODS / API platform that serves the Ed-Fi REST API. | +| **ODS/API** | The Ed-Fi ODS / API platform is the legacy reference implementation of REST API based on the Ed-Fi Data Standard. | +| **Ed-Fi API** | The successor to the ODS/API, which also consumes MetaEd-generated artifacts, particularly the API metadata. | | **Data Standard** | The Ed-Fi Unifying Data Model, versioned and expressed in `.metaed` files. | | **Extension** | A user-defined addition to the core Ed-Fi Data Standard. | | **Alliance Mode** | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | From 6ad31e33b7dfba43b2dbd34245b07d801c4e9efc Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 16:34:52 -0500 Subject: [PATCH 06/15] Add or update readmes for each package --- packages/metaed-console/README.md | 28 +++++++++++++- packages/metaed-core/README.md | 32 ++++++++++++++++ packages/metaed-default-plugins/README.md | 19 ++++++++++ .../metaed-odsapi-deploy-console/README.md | 31 +++++++++++++++ packages/metaed-odsapi-deploy/README.md | 38 +++++++++++++++++++ .../metaed-plugin-edfi-api-catalog/README.md | 24 ++++++------ .../metaed-plugin-edfi-api-schema/README.md | 26 +++++++++++++ .../metaed-plugin-edfi-data-catalog/README.md | 21 ++++++++++ .../metaed-plugin-edfi-handbook/README.md | 22 +++++++++++ .../README.md | 25 ++++++++++++ .../README.md | 25 ++++++++++++ .../README.md | 31 +++++++++++++++ .../README.md | 27 +++++++++++++ .../README.md | 20 ++++++++++ .../README.md | 20 ++++++++++ .../README.md | 19 ++++++++++ .../README.md | 23 +++++++++++ .../README.md | 27 +++++++++++++ packages/metaed-plugin-edfi-odsapi/README.md | 21 ++++++++++ .../README.md | 23 +++++++++++ .../README.md | 18 +++++++++ packages/metaed-plugin-edfi-unified/README.md | 20 ++++++++++ .../README.md | 22 +++++++++++ packages/metaed-plugin-edfi-xsd/README.md | 24 ++++++++++++ 24 files changed, 574 insertions(+), 12 deletions(-) create mode 100644 packages/metaed-core/README.md create mode 100644 packages/metaed-default-plugins/README.md create mode 100644 packages/metaed-odsapi-deploy-console/README.md create mode 100644 packages/metaed-odsapi-deploy/README.md create mode 100644 packages/metaed-plugin-edfi-api-schema/README.md create mode 100644 packages/metaed-plugin-edfi-data-catalog/README.md create mode 100644 packages/metaed-plugin-edfi-handbook/README.md create mode 100644 packages/metaed-plugin-edfi-ods-changequery-postgresql/README.md create mode 100644 packages/metaed-plugin-edfi-ods-changequery-sqlserver/README.md create mode 100644 packages/metaed-plugin-edfi-ods-changequery/README.md create mode 100644 packages/metaed-plugin-edfi-ods-postgresql/README.md create mode 100644 packages/metaed-plugin-edfi-ods-recordownership-postgresql/README.md create mode 100644 packages/metaed-plugin-edfi-ods-recordownership-sqlserver/README.md create mode 100644 packages/metaed-plugin-edfi-ods-recordownership/README.md create mode 100644 packages/metaed-plugin-edfi-ods-relational/README.md create mode 100644 packages/metaed-plugin-edfi-ods-sqlserver/README.md create mode 100644 packages/metaed-plugin-edfi-odsapi/README.md create mode 100644 packages/metaed-plugin-edfi-sql-dictionary/README.md create mode 100644 packages/metaed-plugin-edfi-unified-advanced/README.md create mode 100644 packages/metaed-plugin-edfi-unified/README.md create mode 100644 packages/metaed-plugin-edfi-xml-dictionary/README.md create mode 100644 packages/metaed-plugin-edfi-xsd/README.md diff --git a/packages/metaed-console/README.md b/packages/metaed-console/README.md index b2476f73d..863033255 100644 --- a/packages/metaed-console/README.md +++ b/packages/metaed-console/README.md @@ -1,6 +1,32 @@ # metaed-console -This package can be used for running the MetaEd build process from the command line. +Command-line interface for running the MetaEd build pipeline. + +## Input Configuration + +CLI arguments via `yargs`: + +- `--config / -c` — Path to a JSON configuration file (relative to working directory) +- `--defaultPluginTechVersion / -x` — Default plugin technology version +- `--accept-license / -a` — Required flag to accept the license agreement +- `--suppressPrereleaseVersion` — Suppress pre-release version in output paths (default: `true`) + +The config file supplies a `metaEdConfiguration` object with project paths, artifact +directories, and plugin settings. See `metaed-edfi-5.2.json` for a fully-worked sample. + +## Output + +Executes the full MetaEd pipeline (load → parse → build → validate → enhance → +generate → write) and writes all generated artifacts to the configured artifact +directory. Exits with code 0 on success, 1 on failure. + +## Business Logic + +Assembles the default plugin set, resolves the data standard version from project +metadata, builds the pipeline state, and runs all registered validators, enhancers, +and generators in sequence. Logs timing information for each phase. + +## Usage 1. Build the project from the parent directory with `npm run build`. 2. To confirm it is functional, try `node ./dist/index.js -h`. diff --git a/packages/metaed-core/README.md b/packages/metaed-core/README.md new file mode 100644 index 000000000..56f8f1af2 --- /dev/null +++ b/packages/metaed-core/README.md @@ -0,0 +1,32 @@ +# metaed-core + +Core engine for the MetaEd DSL pipeline. Provides the shared model, plugin +abstractions, and pipeline orchestration used by all MetaEd packages. + +## Input Configuration + +`MetaEdConfiguration` type with key fields: + +- `artifactDirectory` — Where generated artifacts are written +- `deployDirectory` — Target for deployment operations +- `pluginTechVersion` — Technology version for plugin selection +- `defaultPluginTechVersion` — Fallback technology version +- `projects` — Array of project definitions (name, version, namespace, paths) +- `projectPaths` — File system paths for MetaEd source files +- `pluginConfigDirectories` — Directories containing plugin configuration +- `allianceMode` — Whether running in Ed-Fi Alliance mode +- `suppressPrereleaseVersion` — Controls version formatting in output paths + +## Output + +- Parsed MetaEd model (domain entities, associations, descriptors, etc.) +- Pipeline execution results (validation failures, generated output) +- `GeneratedOutput` objects containing file name, folder, namespace, and content + (as string or stream) + +## Business Logic + +Orchestrates the sequential pipeline: initialize → load → parse → build → validate → +enhance → generate → write. Defines the plugin contract (validators, enhancers, +generators), the domain model types, and the shared infrastructure for file I/O, +logging, and configuration resolution. diff --git a/packages/metaed-default-plugins/README.md b/packages/metaed-default-plugins/README.md new file mode 100644 index 000000000..ae5bcdc36 --- /dev/null +++ b/packages/metaed-default-plugins/README.md @@ -0,0 +1,19 @@ +# metaed-default-plugins + +Convenience package that assembles the standard MetaEd plugin stack in the correct +dependency order. + +## Input Configuration + +None. This package has no runtime configuration of its own. + +## Output + +Exports `defaultPlugins(): MetaEdPlugin[]` which returns the canonical ordered list of +all standard MetaEd plugins, cached after first initialization. + +## Business Logic + +Imports and initializes all standard Ed-Fi plugins (unified, ODS relational, SQL Server, +PostgreSQL, change query, record ownership, XSD, API schema, handbook, dictionaries, +etc.) and returns them in the dependency order required by the pipeline. diff --git a/packages/metaed-odsapi-deploy-console/README.md b/packages/metaed-odsapi-deploy-console/README.md new file mode 100644 index 000000000..4955bfb05 --- /dev/null +++ b/packages/metaed-odsapi-deploy-console/README.md @@ -0,0 +1,31 @@ +# metaed-odsapi-deploy-console + +Command-line interface for deploying MetaEd artifacts into an ODS/API repository. + +## Input Configuration + +CLI arguments via `yargs`: + +- `--config / -c` — Path to JSON configuration file +- `--source / -s` — Source project directories (array) +- `--target / -t` — Target ODS/API repository path +- `--projectNames / -p` — Project names to deploy (array) +- `--defaultPluginTechVersion / -x` — Default plugin technology version +- `--core` — Deploy core artifacts +- `--suppressDelete` — Skip removal of existing extension artifacts +- `--accept-license / -a` — Required license acceptance flag +- `--suppressPrereleaseVersion` — Suppress pre-release version in paths +- `--additionalMssqlScriptsDirectory` — Extra SQL Server scripts directory +- `--additionalPostgresScriptsDirectory` — Extra PostgreSQL scripts directory + +## Output + +Runs the MetaEd build pipeline for the specified projects, then deploys generated +artifacts into the target ODS/API repository. Exits with code 0 on success, 1 on +failure, and logs duration. + +## Business Logic + +Scans source directories for MetaEd projects, builds `MetaEdConfiguration`, runs the +generation pipeline, then delegates to `metaed-odsapi-deploy` to copy artifacts into +the destination repository structure. diff --git a/packages/metaed-odsapi-deploy/README.md b/packages/metaed-odsapi-deploy/README.md new file mode 100644 index 000000000..e8a1ed4a8 --- /dev/null +++ b/packages/metaed-odsapi-deploy/README.md @@ -0,0 +1,38 @@ +# metaed-odsapi-deploy + +Library for deploying MetaEd-generated artifacts into the ODS/API repository layout. + +## Input Configuration + +Accepts `MetaEdConfiguration` plus deploy-specific options: + +- `dataStandardVersion` — Determines output path structure +- `deployCore` — Whether to deploy core (standard) artifacts +- `suppressDelete` — Skip removal of existing extension artifact directories +- `additionalMssqlScriptsDirectory` — Extra SQL Server scripts to copy +- `additionalPostgresScriptsDirectory` — Extra PostgreSQL scripts to copy + +## Output + +Copies generated artifacts into the ODS/API file system structure: + +- Core artifacts → `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{version}/Artifacts/` +- Extension artifacts → `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{name}/Versions/{version}/Standard/{dsVersion}/Artifacts/` + +Sub-path mappings: + +| Source Directory | Deploy Destination | +|---|---| +| ApiMetadata | Metadata | +| Database/SQLServer/ODS/Data | MsSql/Data/Ods | +| Database/SQLServer/ODS/Structure | MsSql/Structure/Ods | +| Database/PostgreSQL/ODS/Data | PgSql/Data/Ods | +| Database/PostgreSQL/ODS/Structure | PgSql/Structure/Ods | +| Interchange | Schemas | +| XSD | Schemas | + +## Business Logic + +Runs a sequence of filesystem tasks: remove old extension artifacts (unless suppressed), +deploy core artifacts (ODS/API ≥ 7.0 only), deploy extension artifacts, copy additional +DB scripts, and refresh csproj timestamps. Stops on the first failure. diff --git a/packages/metaed-plugin-edfi-api-catalog/README.md b/packages/metaed-plugin-edfi-api-catalog/README.md index abf604943..5c25c78b6 100644 --- a/packages/metaed-plugin-edfi-api-catalog/README.md +++ b/packages/metaed-plugin-edfi-api-catalog/README.md @@ -1,14 +1,18 @@ -# MetaEd Plugin: Ed-Fi API Catalog +# metaed-plugin-edfi-api-catalog This plugin generates an Excel spreadsheet catalog of API resources and their properties from the MetaEd API schema. -## Overview - The API Catalog plugin reads the enhanced MetaEd model from the `metaed-plugin-edfi-api-schema` plugin and generates a comprehensive Excel spreadsheet listing all API resources, their properties, and associated metadata. +## Input Configuration + +No plugin-specific configuration. Depends on API schema metadata produced by +`metaed-plugin-edfi-api-schema`. + ## Dependencies This plugin depends on: + - `@edfi/metaed-core` - `@edfi/metaed-plugin-edfi-api-schema` @@ -16,7 +20,7 @@ The plugin must be loaded after the `metaed-plugin-edfi-api-schema` plugin to ac ## Output -The plugin generates a single Excel file (`Ed-Fi-API-Catalog.xlsx`) with two worksheets: +The plugin generates a single Excel file (`Documentation/Ed-Fi-API-Catalog/Ed-Fi-API-Catalog.xlsx`) with two worksheets: ### Resources Worksheet @@ -45,14 +49,12 @@ Contains one row per property within each resource with the following columns: - **Is Nullable**: Boolean indicating if the property can be null - **Is Required**: Boolean indicating if the property is required -## Usage - -This plugin is included in the default plugins list and will automatically generate the API catalog when MetaEd processes a data model. +## Business Logic -The generated Excel file can be found in the output directory under: -``` -Documentation/Ed-Fi-API-Catalog/Ed-Fi-API-Catalog.xlsx -``` +Walks the API schema resource definitions built by the upstream API schema plugin, +extracts resource and property metadata rows, and writes a multi-sheet XLSX workbook +suitable for documentation and review purposes. Both resources and descriptors are +included; reference properties use `dataType = 'reference'`. ## Implementation Details diff --git a/packages/metaed-plugin-edfi-api-schema/README.md b/packages/metaed-plugin-edfi-api-schema/README.md new file mode 100644 index 000000000..a9d81127b --- /dev/null +++ b/packages/metaed-plugin-edfi-api-schema/README.md @@ -0,0 +1,26 @@ +# metaed-plugin-edfi-api-schema + +MetaEd plugin that generates the DMS API schema JSON used by the Ed-Fi API. + +## Input Configuration + +Registers two optional configuration schemas: + +- `educationOrganizationSecurableElements` — Configures security elements for + education organizations +- `educationOrganizationIdentitySecurableElements` — Configures identity security + elements for education organizations + +## Output + +Generates one JSON file per namespace: + +- `ApiSchema/ApiSchema.json` (for core) +- `ApiSchema/ApiSchema-{projectPrefix}.json` (for extensions) + +## Business Logic + +Builds the DMS API schema from MetaEd-enhanced namespace data through a series of +enhancers that derive resource schemas, document paths, and reference structures. +Serializes the result as pretty-printed JSON. Also exports many API-schema types and +enhancers consumed by downstream plugins (api-catalog, odsapi). diff --git a/packages/metaed-plugin-edfi-data-catalog/README.md b/packages/metaed-plugin-edfi-data-catalog/README.md new file mode 100644 index 000000000..897e24f05 --- /dev/null +++ b/packages/metaed-plugin-edfi-data-catalog/README.md @@ -0,0 +1,21 @@ +# metaed-plugin-edfi-data-catalog + +MetaEd plugin that generates an Excel data catalog of the Ed-Fi Data Standard. + +## Input Configuration + +No plugin-specific configuration. Relies on model data enriched by upstream enhancers. + +## Output + +Generates a single Excel workbook: + +- `Data-Catalog/Data-Catalog.xlsx` + +The workbook contains sheets for domains, entities, and elements. + +## Business Logic + +Aggregates domain, entity, and element metadata across namespaces from the enriched +MetaEd model. Sorts entries and writes a three-sheet Excel catalog suitable for +data standard documentation. diff --git a/packages/metaed-plugin-edfi-handbook/README.md b/packages/metaed-plugin-edfi-handbook/README.md new file mode 100644 index 000000000..bd1368b03 --- /dev/null +++ b/packages/metaed-plugin-edfi-handbook/README.md @@ -0,0 +1,22 @@ +# metaed-plugin-edfi-handbook + +MetaEd plugin that generates the Ed-Fi Data Handbook in HTML and Excel formats. + +## Input Configuration + +No plugin-specific configuration. Relies on model data enriched by upstream enhancers. + +## Output + +Generates two artifacts: + +- `Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` — Interactive HTML single-page + application for browsing handbook entries +- `Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx` — Excel workbook with the same content in + spreadsheet form + +## Business Logic + +Enhances model metadata into handbook entries (entity descriptions, properties, +relationships), then emits both an HTML SPA index and an Excel workbook. Entries are +sorted and formatted for documentation and stakeholder review. diff --git a/packages/metaed-plugin-edfi-ods-changequery-postgresql/README.md b/packages/metaed-plugin-edfi-ods-changequery-postgresql/README.md new file mode 100644 index 000000000..b293874aa --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-changequery-postgresql/README.md @@ -0,0 +1,25 @@ +# metaed-plugin-edfi-ods-changequery-postgresql + +MetaEd plugin that generates PostgreSQL-specific SQL for ODS change-query support. + +## Input Configuration + +No plugin-specific configuration. Depends on the shared `metaed-plugin-edfi-ods-changequery` +plugin for model enrichment. + +## Output + +Generates PostgreSQL-flavored change-query SQL scripts under +`Database/PostgreSQL/ODS/Structure/Changes/`, including: + +- Tracked-delete table and schema creation +- Change-version sequence +- Change-tracking triggers +- Change-version indexes +- Indirect update cascade triggers + +## Business Logic + +Applies PostgreSQL-specific enhancers and generators to the change-query model built by +the common change-query plugin. Emits DDL using PostgreSQL syntax for sequences, +triggers, and indexes. diff --git a/packages/metaed-plugin-edfi-ods-changequery-sqlserver/README.md b/packages/metaed-plugin-edfi-ods-changequery-sqlserver/README.md new file mode 100644 index 000000000..4171bc5d4 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-changequery-sqlserver/README.md @@ -0,0 +1,25 @@ +# metaed-plugin-edfi-ods-changequery-sqlserver + +MetaEd plugin that generates SQL Server-specific SQL for ODS change-query support. + +## Input Configuration + +No plugin-specific configuration. Depends on the shared `metaed-plugin-edfi-ods-changequery` +plugin for model enrichment. + +## Output + +Generates SQL Server-flavored change-query SQL scripts under +`Database/SQLServer/ODS/Structure/Changes/`, including: + +- Tracked-delete table and schema creation +- Change-version sequence +- Change-tracking triggers +- Change-version indexes +- Indirect update cascade triggers + +## Business Logic + +Applies SQL Server-specific enhancers and generators to the change-query model built by +the common change-query plugin. Emits DDL using SQL Server syntax for sequences, +triggers, and indexes. diff --git a/packages/metaed-plugin-edfi-ods-changequery/README.md b/packages/metaed-plugin-edfi-ods-changequery/README.md new file mode 100644 index 000000000..234506a14 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-changequery/README.md @@ -0,0 +1,31 @@ +# metaed-plugin-edfi-ods-changequery + +MetaEd plugin providing shared change-query support for the ODS. Generates +database-agnostic SQL DDL for change tracking infrastructure. + +## Input Configuration + +No plugin-specific configuration. Uses the MetaEd model and target technology version. + +## Output + +Generates SQL DDL scripts for change-query support (numbered for execution order): + +- `0010-CreateChangesSchema.sql` +- `0020-CreateChangeVersionSequence.sql` +- `0030-AddColumnChangeVersionForTables.sql` +- `0045-CreateTrackedDeleteSchema.sql` +- `0050-CreateTrackedDeleteTables.sql` +- `0060-CreateDeletedForTrackingTriggers.sql` +- `0070-AddIndexChangeVersionForTables.sql` +- `0130-CreateTriggerUpdateChangeVersion*.sql` +- `0230-CreateIndirectUpdateCascadeTriggers.sql` + +Script names vary by target technology version (style 6.0 uses alternate numbering). + +## Business Logic + +Validates namespaces for change-query compatibility, enriches entities with +change-tracking metadata, and generates the common SQL objects (schemas, sequences, +columns, triggers, indexes) needed for ODS change tracking. Platform-specific SQL +is delegated to the PostgreSQL and SQL Server companion plugins. diff --git a/packages/metaed-plugin-edfi-ods-postgresql/README.md b/packages/metaed-plugin-edfi-ods-postgresql/README.md new file mode 100644 index 000000000..d3e8d80ca --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-postgresql/README.md @@ -0,0 +1,27 @@ +# metaed-plugin-edfi-ods-postgresql + +MetaEd plugin that generates PostgreSQL DDL scripts for the Ed-Fi ODS. + +## Input Configuration + +No plugin-specific configuration. Uses the MetaEd model and `targetTechnologyVersion` +from the plugin environment to control SQL dialect details (e.g., timestamp defaults). + +## Output + +Generates PostgreSQL ODS scripts under `Database/PostgreSQL/ODS/`: + +- `Structure/0010-*-Schemas.sql` — Schema creation +- `Structure/0020-*-Tables.sql` — Table DDL +- `Structure/0030-*-ForeignKeys.sql` — Foreign key constraints +- `Structure/0050-*-ExtendedProperties.sql` — Extended properties/comments +- `Data/0010-*-Enumerations.sql` — Descriptor/enumeration seed data +- `Data/0020-*-SchoolYears.sql` — School year seed data + +Additional scripts for indexes and authorization views are also generated. + +## Business Logic + +Consumes the relational ODS model produced by `metaed-plugin-edfi-ods-relational` and +emits PostgreSQL-specific DDL for schemas, tables, foreign keys, constraints, seed data, +indexes, and authorization scripts. diff --git a/packages/metaed-plugin-edfi-ods-recordownership-postgresql/README.md b/packages/metaed-plugin-edfi-ods-recordownership-postgresql/README.md new file mode 100644 index 000000000..cc4d8fc81 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-recordownership-postgresql/README.md @@ -0,0 +1,20 @@ +# metaed-plugin-edfi-ods-recordownership-postgresql + +MetaEd plugin that generates PostgreSQL DDL for ODS record-ownership columns. + +## Input Configuration + +No plugin-specific configuration. Depends on `metaed-plugin-edfi-ods-recordownership` +for table-level ownership metadata. + +## Output + +Generates PostgreSQL DDL under `Database/PostgreSQL/ODS/Structure/RecordOwnership/`: + +- `0010-AddColumnOwnershipTokenForTable.sql` — Adds ownership-token columns to flagged + tables + +## Business Logic + +Finds tables flagged for record-ownership support by the common record-ownership plugin +and generates PostgreSQL `ALTER TABLE` statements to add the ownership-token column. diff --git a/packages/metaed-plugin-edfi-ods-recordownership-sqlserver/README.md b/packages/metaed-plugin-edfi-ods-recordownership-sqlserver/README.md new file mode 100644 index 000000000..a8eebd438 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-recordownership-sqlserver/README.md @@ -0,0 +1,20 @@ +# metaed-plugin-edfi-ods-recordownership-sqlserver + +MetaEd plugin that generates SQL Server DDL for ODS record-ownership columns. + +## Input Configuration + +No plugin-specific configuration. Depends on `metaed-plugin-edfi-ods-recordownership` +for table-level ownership metadata. + +## Output + +Generates SQL Server DDL under `Database/SQLServer/ODS/Structure/RecordOwnership/`: + +- `0010-AddColumnOwnershipTokenForTable.sql` — Adds ownership-token columns to flagged + tables + +## Business Logic + +Finds tables flagged for record-ownership support by the common record-ownership plugin +and generates SQL Server `ALTER TABLE` statements to add the ownership-token column. diff --git a/packages/metaed-plugin-edfi-ods-recordownership/README.md b/packages/metaed-plugin-edfi-ods-recordownership/README.md new file mode 100644 index 000000000..58ce42e17 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-recordownership/README.md @@ -0,0 +1,19 @@ +# metaed-plugin-edfi-ods-recordownership + +MetaEd plugin that enriches ODS table metadata with record-ownership awareness. + +## Input Configuration + +No plugin-specific configuration. Uses `targetTechnologyVersion` from the plugin +environment (record ownership is enabled for versions ≥ 3.3.0). + +## Output + +No file artifacts. This is a metadata-only plugin that enriches the in-memory model +with `edfiOdsRecordOwnership` data on each table. + +## Business Logic + +Marks every ODS table with a record-ownership flag so that downstream platform-specific +generators (PostgreSQL, SQL Server) can add ownership-token columns when the feature is +enabled. Acts as the shared foundation for the record-ownership companion plugins. diff --git a/packages/metaed-plugin-edfi-ods-relational/README.md b/packages/metaed-plugin-edfi-ods-relational/README.md new file mode 100644 index 000000000..3ee514d52 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-relational/README.md @@ -0,0 +1,23 @@ +# metaed-plugin-edfi-ods-relational + +MetaEd plugin that builds the shared relational ODS model used by database-specific +generators. + +## Input Configuration + +No plugin-specific configuration. Operates on the core MetaEd model produced by +upstream enhancers. + +## Output + +No file artifacts. This is a model-enrichment plugin that produces the in-memory +relational representation (tables, columns, foreign keys, rows, repositories, naming) +consumed by the PostgreSQL and SQL Server ODS generator plugins. + +## Business Logic + +Transforms the semantic MetaEd model into a relational database model through a large +series of enhancers. Constructs table definitions, column mappings, primary/foreign key +relationships, enumeration rows, and naming conventions. Also provides validators for +relational integrity. Serves as the shared foundation that both platform-specific ODS +plugins build upon. diff --git a/packages/metaed-plugin-edfi-ods-sqlserver/README.md b/packages/metaed-plugin-edfi-ods-sqlserver/README.md new file mode 100644 index 000000000..50f4ff837 --- /dev/null +++ b/packages/metaed-plugin-edfi-ods-sqlserver/README.md @@ -0,0 +1,27 @@ +# metaed-plugin-edfi-ods-sqlserver + +MetaEd plugin that generates SQL Server DDL scripts for the Ed-Fi ODS. + +## Input Configuration + +No plugin-specific configuration. Uses the MetaEd model and `targetTechnologyVersion` +from the plugin environment to control SQL Server-specific behavior. + +## Output + +Generates SQL Server ODS scripts under `Database/SQLServer/ODS/`: + +- `Structure/0010-*-Schemas.sql` — Schema creation +- `Structure/0020-*-Tables.sql` — Table DDL +- `Structure/0030-*-ForeignKeys.sql` — Foreign key constraints +- `Structure/0050-*-ExtendedProperties.sql` — Extended properties +- `Data/0010-*-Enumerations.sql` — Descriptor/enumeration seed data +- `Data/0020-*-SchoolYears.sql` — School year seed data + +Additional scripts for indexes and authorization views are also generated. + +## Business Logic + +Consumes the relational ODS model produced by `metaed-plugin-edfi-ods-relational` and +emits SQL Server-specific DDL for schemas, tables, foreign keys, constraints, seed data, +indexes, and authorization scripts. diff --git a/packages/metaed-plugin-edfi-odsapi/README.md b/packages/metaed-plugin-edfi-odsapi/README.md new file mode 100644 index 000000000..0e129fda8 --- /dev/null +++ b/packages/metaed-plugin-edfi-odsapi/README.md @@ -0,0 +1,21 @@ +# metaed-plugin-edfi-odsapi + +MetaEd plugin that generates the Ed-Fi ODS/API domain model JSON. + +## Input Configuration + +No plugin-specific configuration. Operates on the enriched MetaEd model. + +## Output + +Generates one JSON file per namespace: + +- `ApiMetadata/ApiModel.json` (for core) +- `ApiMetadata/ApiModel-{projectExtension}.json` (for extensions) + +## Business Logic + +Builds the Ed-Fi ODS/API domain model by running a large set of API-model enhancers +that derive resource definitions, associations, and metadata. Serializes the enriched +model to JSON for consumption by the ODS/API application. Also includes validators for +API-specific modeling rules. diff --git a/packages/metaed-plugin-edfi-sql-dictionary/README.md b/packages/metaed-plugin-edfi-sql-dictionary/README.md new file mode 100644 index 000000000..73e8b42a0 --- /dev/null +++ b/packages/metaed-plugin-edfi-sql-dictionary/README.md @@ -0,0 +1,23 @@ +# metaed-plugin-edfi-sql-dictionary + +MetaEd plugin that generates an Excel SQL data dictionary from ODS table metadata. + +## Input Configuration + +No plugin-specific configuration. Depends on relational ODS model data produced by +`metaed-plugin-edfi-ods-relational`. + +## Output + +Generates a single Excel workbook: + +- `DataDictionary/SqlDataDictionary.xlsx` + +The workbook contains two sheets: Tables (table names and descriptions) and Columns +(column names, types, constraints per table). + +## Business Logic + +Collects ODS relational table and column metadata from all namespaces, sorts entries +alphabetically, and writes a two-sheet Excel data dictionary for documentation and +database review purposes. diff --git a/packages/metaed-plugin-edfi-unified-advanced/README.md b/packages/metaed-plugin-edfi-unified-advanced/README.md new file mode 100644 index 000000000..8ac3fbb55 --- /dev/null +++ b/packages/metaed-plugin-edfi-unified-advanced/README.md @@ -0,0 +1,18 @@ +# metaed-plugin-edfi-unified-advanced + +MetaEd plugin that provides advanced validation rules for the unified model. + +## Input Configuration + +No plugin-specific configuration. This is a validators-only plugin. + +## Output + +No file artifacts. This plugin only reports validation failures when modeling rules +are violated. + +## Business Logic + +Adds advanced validation rules on top of the unified model, checking for correct usage +of merge directives, self-references, common properties, and deprecated patterns. +Reports human-readable validation errors when modeling constraints are violated. diff --git a/packages/metaed-plugin-edfi-unified/README.md b/packages/metaed-plugin-edfi-unified/README.md new file mode 100644 index 000000000..f52454426 --- /dev/null +++ b/packages/metaed-plugin-edfi-unified/README.md @@ -0,0 +1,20 @@ +# metaed-plugin-edfi-unified + +MetaEd plugin that provides core model transformations shared across multiple +downstream plugins. + +## Input Configuration + +No plugin-specific configuration. This is a pure enhancer plugin. + +## Output + +No file artifacts. This plugin only enriches the in-memory MetaEd model with unified +transformations used by other plugins (XSD, XML dictionary, ODS, etc.). + +## Business Logic + +Applies core "unified" model enhancements that shape entities, properties, and +relationships into forms required by downstream generators. Provides the shared model +foundation for interchange schemas, type hierarchies, and cross-entity relationships +that multiple output plugins depend on. diff --git a/packages/metaed-plugin-edfi-xml-dictionary/README.md b/packages/metaed-plugin-edfi-xml-dictionary/README.md new file mode 100644 index 000000000..dc13e1e3c --- /dev/null +++ b/packages/metaed-plugin-edfi-xml-dictionary/README.md @@ -0,0 +1,22 @@ +# metaed-plugin-edfi-xml-dictionary + +MetaEd plugin that generates an Excel XML data dictionary from XSD schema structures. + +## Input Configuration + +No plugin-specific configuration. Depends on XSD model data produced by +`metaed-plugin-edfi-xsd`. + +## Output + +Generates a single Excel workbook: + +- `Documentation/XmlDataDictionary.xlsx` + +The workbook contains three sheets: Elements, ComplexTypes, and SimpleTypes. + +## Business Logic + +Traverses XSD-derived schema structures and exports a dictionary of XML types and +elements, including names, cardinality, restrictions, and relationships. Provides a +documentation view of the XML schema in spreadsheet form. diff --git a/packages/metaed-plugin-edfi-xsd/README.md b/packages/metaed-plugin-edfi-xsd/README.md new file mode 100644 index 000000000..0794d40a2 --- /dev/null +++ b/packages/metaed-plugin-edfi-xsd/README.md @@ -0,0 +1,24 @@ +# metaed-plugin-edfi-xsd + +MetaEd plugin that generates XSD (XML Schema Definition) files for the Ed-Fi data model. + +## Input Configuration + +No plugin-specific configuration. Operates on the enriched MetaEd model with unified +transformations applied. + +## Output + +Generates XSD artifacts under `XSD/`: + +- `Ed-Fi-Core.xsd` (for core namespace) +- `{Extension}-Ed-Fi-Extended-Core.xsd` (for extension namespaces) +- `SchemaAnnotation.xsd` — Schema annotation definitions +- `Interchange-{Name}.xsd` — Interchange schemas (for core) +- `{Extension}-Interchange-{Name}-Extension.xsd` — Interchange extension schemas + +## Business Logic + +Enhances the MetaEd model into XSD schema objects through a series of enhancers, then +generates core schema files, schema annotations, and interchange schemas. Also validates +for duplicate names across dependency namespaces to prevent XSD conflicts. From a5b5517d08b55095f19dba605a5386bab6238b9b Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 16:46:49 -0500 Subject: [PATCH 07/15] Delete spurious file --- .../metaed-plugin-edfi-data-catalog/README.md | 21 ------------------- 1 file changed, 21 deletions(-) delete mode 100644 packages/metaed-plugin-edfi-data-catalog/README.md diff --git a/packages/metaed-plugin-edfi-data-catalog/README.md b/packages/metaed-plugin-edfi-data-catalog/README.md deleted file mode 100644 index 897e24f05..000000000 --- a/packages/metaed-plugin-edfi-data-catalog/README.md +++ /dev/null @@ -1,21 +0,0 @@ -# metaed-plugin-edfi-data-catalog - -MetaEd plugin that generates an Excel data catalog of the Ed-Fi Data Standard. - -## Input Configuration - -No plugin-specific configuration. Relies on model data enriched by upstream enhancers. - -## Output - -Generates a single Excel workbook: - -- `Data-Catalog/Data-Catalog.xlsx` - -The workbook contains sheets for domains, entities, and elements. - -## Business Logic - -Aggregates domain, entity, and element metadata across namespaces from the enriched -MetaEd model. Sorts entries and writes a three-sheet Excel catalog suitable for -data standard documentation. From 66135c1b084f0d0e988f325fcaacbd204800d702 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 17:02:25 -0500 Subject: [PATCH 08/15] Address PRD review comments from Copilot --- docs/PRD-4.x.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index 1827a091d..d0eaa2926 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -166,14 +166,14 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO #### 3.5.1 SQL Scripts - **FR-SQL-1**: The Build SHALL generate SQL scripts compatible with both Microsoft SQL Server and PostgreSQL for creating an operational data store (ODS). -- **FR-SQL-2**: Generated SQL files SHALL include: - - `0001-Schemas.sql` — core database schema definitions - - `0001-EXTENSION-{namespace}-Schemas.sql` — extension schema definitions - - `0004-Tables.sql` — core database tables - - `0004-EXTENSION-{namespace}.sql` — extension database tables - - `0009-IdColumnUniqueIndexes.sql` — indexing for API support on core tables - - `0009-EXTENSION-{namespace}-IdColumnUniqueIndexes.sql` — indexing for extension tables -- **FR-SQL-3**: Files prefixed with "EXTENSION" SHALL be additive to the standard ODS/API script set; files without "EXTENSION" SHALL replace the corresponding shipped files. +- **FR-SQL-2**: Generated SQL files SHALL include (but are not limited to): + - `xxxx-Schemas.sql` — core database schema definitions + - `xxxx-Tables.sql` — core database tables + - `xxxx-IdColumnUniqueIndexes.sql` — indexing for API support on core tables + - `xxxx-CreateTrackedChangeTables.sql` — core tracked delete tables for Change Queries + - `xxxx-CreateTriggersForDeleteTracking.sql` (MSSQL only) - triggers for tracked delete support +- **FR-SQL-3**: The Build SHALL generate separate SQL files for core and extension artifacts, with extension files prefixed by "EXTENSION" (e.g., `EXTENSION-xxxx-Tables.sql`). +- **FR-SQL-4**: Files prefixed with "EXTENSION" SHALL be additive to the standard ODS/API script set; files without "EXTENSION" SHALL replace the corresponding shipped files. #### 3.5.2 XSD Files @@ -325,7 +325,7 @@ The system follows a sequential pipeline: **Initialize → Load → Parse → Bu - Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent and costly errors. - The Deploy command deletes existing extension artifact directories by default unless `--suppressDelete` is specified. -## 8. Glossary +## 7. Glossary | Term | Definition | | -------------------- | ----------------------------------------------------------------------------------------------------------------- | From e28b7f3c80364c6e335b5e95b97af1f7192244d4 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Thu, 28 May 2026 17:03:46 -0500 Subject: [PATCH 09/15] Additional detail on the odsapi-deploy package --- packages/metaed-odsapi-deploy/README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/packages/metaed-odsapi-deploy/README.md b/packages/metaed-odsapi-deploy/README.md index e8a1ed4a8..2aac3402f 100644 --- a/packages/metaed-odsapi-deploy/README.md +++ b/packages/metaed-odsapi-deploy/README.md @@ -33,6 +33,10 @@ Sub-path mappings: ## Business Logic -Runs a sequence of filesystem tasks: remove old extension artifacts (unless suppressed), -deploy core artifacts (ODS/API ≥ 7.0 only), deploy extension artifacts, copy additional -DB scripts, and refresh csproj timestamps. Stops on the first failure. + Runs a sequence of deployment tasks in order: first verifies that required extension + projects exist, then removes old extension artifacts (unless suppressed), deploys core + artifacts (ODS/API ≥ 7.0 only), and deploys extension artifacts. Additional MSSQL and + PostgreSQL script directories are passed into the core/extension deploy steps rather than + copied as a separate standalone task. After deployment it refreshes csproj timestamps, + runs the legacy-directory check, and stops on the first failure. + From f6dee355ff92f86c7c9d122472d90fbc685ed56a Mon Sep 17 00:00:00 2001 From: Brad Banister Date: Thu, 18 Jun 2026 21:04:51 -0500 Subject: [PATCH 10/15] Iteratively improved PRD (#513) --- docs/Iterative-PRD.md | 350 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 350 insertions(+) create mode 100644 docs/Iterative-PRD.md diff --git a/docs/Iterative-PRD.md b/docs/Iterative-PRD.md new file mode 100644 index 000000000..0ed8145d6 --- /dev/null +++ b/docs/Iterative-PRD.md @@ -0,0 +1,350 @@ +# MetaEd-js Product Requirements Document + +- Status: Draft, reverse-engineered from repository evidence +- Repository: `MetaEd-js` +- Package version source: Lerna version `4.7.1-dev.1` +- Platform and runtime: Node.js packages compiled from TypeScript to ES2017/CommonJS; CI runs Node 22 +- Document scope: Product behavior implemented in this repository only +- Evidence basis: package metadata, source entry points, CLI/deploy/configuration code, plugin generators, tests, authoritative artifacts, and workflow configuration + +> [!NOTE] Document Scope +> This PRD describes the MetaEd-js command-line generator, deployment utility, default plugin set, and generated artifact families as implemented by package metadata, source code, tests, authoritative artifacts, and workflow configuration in this repository. +> +> It does not define the MetaEd DSL language specification, Ed-Fi ODS/API runtime behavior, database execution semantics, or behavior of an external IDE extension. + +## 1. Product Overview + +MetaEd-js is a Node.js monorepo that compiles Ed-Fi MetaEd DSL projects into local filesystem artifacts used by downstream Ed-Fi data platform tooling. A MetaEd project is represented by `.metaed` source files plus project metadata such as project name, namespace, version, optional project extension, and description. When project metadata omits a project extension, generated extension artifact names use `EXTENSION` while the core `EdFi` namespace uses no project extension. The generator reads one Data Standard project and zero or more extension projects, validates and enriches the semantic model, runs the default plugin set, and writes generated outputs grouped by namespace. + +The repository provides two primary command-line workflows: + +- `@edfi/metaed-console`: runs the MetaEd build pipeline and writes generated artifacts. +- `@edfi/metaed-odsapi-deploy-console`: optionally builds from project source directories and then copies generated artifacts into a local Ed-Fi ODS/API source tree layout for supported technology versions. + +Repository automation also includes a manually dispatchable API Schema packaging workflow under `eng/ApiSchema` that runs MetaEd, moves selected generated API Schema/XSD artifacts into .NET package projects, packs NuGet packages, and publishes those packages through GitHub Actions when feed credentials are supplied. + +The product value is artifact consistency: SQL, XSD, API metadata, API schema, documentation, and dictionary outputs are generated from the same MetaEd model rather than authored independently. + +### 1.1 Target Users and Personas + +- **Data Standard maintainers** maintain the core Ed-Fi Data Standard MetaEd project and need repeatable generated artifacts for release and packaging workflows. +- **Extension developers** maintain additive MetaEd extension projects and need validation plus generated artifacts for ODS/API integration. +- **Build and release engineers** run MetaEd headlessly in CI, publish npm packages, package API Schema artifacts, and verify generated outputs against authoritative fixtures. +- **Downstream artifact consumers** use generated spreadsheets, HTML documentation, JSON schema/metadata files, SQL scripts, and XSD files without reading `.metaed` source. + +### 1.2 Jobs To Be Done + +- **JTBD-1: Validate a model.** When a user changes `.metaed` files, they need the pipeline to report syntax, semantic, configuration, and plugin validation failures in a machine-detectable build result. +- **JTBD-2: Generate platform artifacts.** When a user has a valid Data Standard or extension model, they need one build command to produce all configured SQL, XSD, JSON, HTML, and Excel outputs. +- **JTBD-3: Deploy generated artifacts to a local ODS/API source tree.** When a user has generated artifacts for ODS/API technology version 7 or later, they need a deployment command to copy those artifacts into the expected `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` artifact folders. +- **JTBD-4: Produce documentation from the model.** When a user needs to inspect generated model shape, they need handbook, API catalog, SQL dictionary, and XML dictionary outputs that are generated from the same model pipeline. +- **JTBD-5: Run without an IDE.** When a build or release engineer automates MetaEd, they need Node.js command-line packages that can run in CI with deterministic exit codes. + +## 2. Enterprise and System Context + +### 2.1 External Systems and Integrations + +- **Local MetaEd project directories** provide `.metaed` files and optional `package.json` files with `metaEdProject` metadata. +- **Ed-Fi Data Standard npm packages** are package dependencies of `@edfi/metaed-console` for model packages `@edfi/ed-fi-model-4.0`, `@edfi/ed-fi-model-5.0`, `@edfi/ed-fi-model-5.1`, `@edfi/ed-fi-model-5.2`, `@edfi/ed-fi-model-6.0`, and `@edfi/ed-fi-model-6.1`. +- **Ed-Fi ODS/API source trees** consume deployed artifact files under `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` directories. +- **SQL Server and PostgreSQL** are downstream targets for generated database scripts; MetaEd-js generates scripts but does not execute them as part of the build pipeline. +- **API Schema and API metadata consumers** consume `ApiSchema*.json` and `ApiModel*.json` outputs. +- **.NET SDK and NuGet/Azure Artifacts** are used by API Schema packaging automation under `eng/ApiSchema`. +- **GitHub Actions and Azure Artifacts** are used by repository workflows for linting, testing, publishing npm packages, and packaging API Schema artifacts. +- **ANTLR 4.6 generated JavaScript grammar files** are part of the source tree used by the parser. + +### 2.2 Deployment Boundaries + +- MetaEd-js build and deploy commands operate on local files. The build pipeline reads `.metaed` source files and plugin configuration files, then writes artifacts to disk. +- Deploy copies files into a local ODS/API source tree layout. It does not deploy to running services, execute SQL, start databases, or validate ODS/API runtime behavior. +- Publishing workflows communicate with package feeds, but normal build and deploy runtime behavior is filesystem-based. + +## 3. Functional Requirements + +### 3.1 Project Input and Configuration + +- **FR-CFG-1**: The build CLI SHALL accept a JSON configuration file through `-c` / `--config` containing a top-level `metaEdConfiguration` object. +- **FR-CFG-2**: `metaEdConfiguration` SHALL support `artifactDirectory`, `deployDirectory`, `pluginTechVersion`, `projects`, `projectPaths`, `pluginConfigDirectories`, `defaultPluginTechVersion`, `allianceMode`, and `suppressPrereleaseVersion`; it MAY include `externalVariables` for Jsonnet configuration evaluation. +- **FR-CFG-3**: The pipeline SHALL require `projects` and `projectPaths` to have the same length and SHALL map each project metadata entry to the corresponding source path. +- **FR-CFG-4**: The file loader SHALL recursively load files with `.metaed`, `.metaEd`, `.MetaEd`, or `.METAED` extensions from each configured project path. +- **FR-CFG-5**: The build and deploy CLIs SHALL require exactly one Data Standard project, identified by namespace `EdFi`; no Data Standard project or multiple Data Standard projects SHALL cause failure. +- **FR-CFG-6**: The project scanner used by the deploy console source mode SHALL read `package.json` files containing `metaEdProject`, derive `namespaceName` by removing non-alphanumeric characters from `projectName` only when the result starts with an uppercase letter, include the package description when present, sort a project whose `projectName` is exactly `EdFi` ahead of other projects, otherwise sort discovered projects by project name, and de-duplicate discovered projects by project name. +- **FR-CFG-7**: The project scanner MAY accept `projectNames` overrides that update discovered project names and derived namespaces in discovery order. +- **FR-CFG-8**: Plugin configuration files SHALL be discovered from configured `pluginConfigDirectories`, or from input project directories when no plugin config directories are configured, as `{pluginShortName}.config.jsonnet` or `{pluginShortName}.config.json`, with Jsonnet preferred when both exist. +- **FR-CFG-9**: Plugin configuration loading SHALL support `externalVariables` for Jsonnet evaluation. +- **FR-CFG-10**: Plugin configuration rules SHALL support plugin-wide data or entity-matched data using `entity`, `namespace`, `core`, `extensions`, and `entityName` matching fields. +- **FR-CFG-11**: Configuration matching SHALL reject a single match rule that combines `namespace` with `core` or `extensions`. +- **FR-CFG-12**: The API Schema plugin SHALL validate plugin-specific configuration rules for `educationOrganizationSecurableElements` and `educationOrganizationIdentitySecurableElements`. +- **FR-CFG-13**: When project metadata does not provide `projectExtension`, the file loader SHALL use an empty project extension for namespace `EdFi` and `EXTENSION` for non-EdFi namespaces. +- **FR-CFG-14**: File loading SHALL fail the build when no `.metaed` files are found in any configured input directory. + +### 3.2 Build Command + +- **FR-BUILD-1**: `@edfi/metaed-console` SHALL run from Node.js and execute the MetaEd pipeline using the default plugin set. +- **FR-BUILD-2**: The build CLI SHALL require `-a` / `--accept-license` before executing. +- **FR-BUILD-3**: The build CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for all plugins. +- **FR-BUILD-4**: The default plugin technology version SHALL be `6.0.0` when not overridden by configuration or CLI argument. +- **FR-BUILD-5**: The build CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when no configuration object supplies a value. +- **FR-BUILD-6**: The build pipeline SHALL run initialization, plugin setup, file loading, syntax validation, file indexing, parse tree building, model building, namespace initialization, plugin configuration loading, validators, enhancers, generators, output writing, and validation file mapping. +- **FR-BUILD-7**: The build pipeline SHALL stop plugin execution on validation errors when `stopOnValidationFailure` is enabled. +- **FR-BUILD-8**: The build CLI SHALL exit with code `0` only when no validation error and no pipeline failure occurred; otherwise it SHALL exit with code `1`. +- **FR-BUILD-9**: Generated outputs SHALL be written under the configured artifact directory or a `MetaEdOutput` directory under the last input project path. Relative `artifactDirectory` values SHALL be resolved from the last input project path. +- **FR-BUILD-10**: Output writing SHALL group generated files by namespace, with the generator-provided folder path under that namespace. +- **FR-BUILD-11**: Output writing SHALL refuse to delete an existing output directory unless the path contains `MetaEdOutput`. +- **FR-BUILD-12**: Output writing SHALL refuse to write when `.metaed` files are found in the output location. + +### 3.3 Generated Artifact Families + +#### 1. ODS/API Metadata + +- **FR-ODSAPI-METADATA-1**: The build SHALL generate ODS/API metadata JSON files per namespace as `ApiMetadata/ApiModel.json` when `projectExtension` is empty or `ApiMetadata/ApiModel-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-odsapi`. + +#### 2. API Schema + +- **FR-API-SCHEMA-1**: The build SHALL generate API Schema JSON files per namespace as `ApiSchema/ApiSchema.json` when `projectExtension` is empty or `ApiSchema/ApiSchema-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-api-schema`. +- **FR-API-SCHEMA-2**: API Schema output SHALL include `apiSchemaVersion` and a project schema containing project identity, endpoint name, `compatibleDsRange` for extension-project compatibility, description, resource schemas, resource-name mappings, case-insensitive endpoint-name mappings, abstract resources, education organization information, domains, extension-project flag, and core OpenAPI base documents where generated by the API Schema model. Resource schemas SHALL include generated JSON schema-for-insert data, identity JSON paths, document path mappings, equality constraints, type-coercion JSON paths, decimal validation information, securable elements, authorization pathways, array uniqueness constraints, OpenAPI fragments, query-field mappings where applicable, school year enumeration resource schema when present, and optional relational naming metadata where generated. + +#### 3. SQL Server and PostgreSQL ODS SQL + +- **FR-ODS-SQL-1**: The build SHALL generate SQL Server ODS structure and data scripts under `Database/SQLServer/ODS/`. +- **FR-ODS-SQL-2**: The build SHALL generate PostgreSQL ODS structure and data scripts under `Database/PostgreSQL/ODS/`. +- **FR-ODS-SQL-3**: SQL ODS outputs SHALL include schema, table, foreign key, extended property, enumeration, and school year scripts when corresponding model data exists; core file names SHALL use `{prefix}-{suffix}.sql`, and extension file names SHALL include namespace context as `{prefix}-{projectExtension}-{namespaceName}-{suffix}.sql` or `{prefix}-{namespaceName}-{suffix}.sql` when `projectExtension` is empty. +- **FR-ODS-SQL-4**: SQL ODS outputs SHALL include ID column unique index scripts for tables that require generated ID indexes. +- **FR-ODS-SQL-5**: For technology versions `>=7.1.0`, SQL ODS outputs SHALL include education organization authorization index creation scripts when table data exists. +- **FR-ODS-SQL-6**: For technology versions `>=7.3.0`, SQL ODS outputs SHALL include aggregate ID column scripts and education organization authorization index update scripts when table data exists. +- **FR-ODS-SQL-7**: SQL-generating plugins SHALL include Apache-2.0 license headers in supported SQL templates only when `allianceMode` is enabled and the relevant target technology version satisfies `>=5.0.0`. + +#### 4. Change Query SQL + +- **FR-CHANGE-QUERY-SQL-1**: The change query plugins SHALL generate change schema, change version sequence, change version column/index, tracked delete schema, tracked delete table, delete tracking trigger, change version/key change trigger, and indirect update cascade trigger scripts under `Database/{SQLServer|PostgreSQL}/ODS/Structure/Changes/` when required model data exists. For target technology versions `>=7.3.0`, shared change schema and sequence files SHALL be generated for the core namespace and skipped for extension namespaces, and indirect update cascade triggers SHALL be generated when required model data exists. + +#### 5. Record Ownership SQL + +- **FR-RECORD-OWNERSHIP-SQL-1**: The record ownership plugins SHALL generate `0010-AddColumnOwnershipTokenForTable.sql` under `Database/{SQLServer|PostgreSQL}/ODS/Structure/RecordOwnership/` for aggregate root tables when the target technology version satisfies `>=3.3.0`. + +#### 6. XSD and Interchange Schemas + +- **FR-XSD-1**: The XSD plugin SHALL generate `XSD/Ed-Fi-Core.xsd` for the core namespace and `{projectExtension}-Ed-Fi-Extended-Core.xsd` for extension namespaces. +- **FR-XSD-2**: The XSD plugin SHALL generate `XSD/SchemaAnnotation.xsd` for the core namespace. +- **FR-XSD-3**: The XSD plugin SHALL generate interchange schemas under `Interchange/`, using `Interchange-{Name}.xsd` for core interchanges and `{projectExtension}-Interchange-{Name}-Extension.xsd` for extension interchanges. + +#### 7. Data Handbook + +- **FR-HANDBOOK-1**: The handbook plugin SHALL generate `Documentation/Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` and `Documentation/Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx`. + +#### 8. Data Dictionaries + +- **FR-DICTIONARY-1**: The SQL dictionary plugin SHALL generate `Documentation/DataDictionary/SqlDataDictionary.xlsx` with table and column worksheets derived from relational ODS model data. +- **FR-DICTIONARY-2**: The XML dictionary plugin SHALL generate `Documentation/DataDictionary/XmlDataDictionary.xlsx` with elements, complex types, and simple types worksheets derived from XSD model data. + +#### 9. API Catalog + +- **FR-API-CATALOG-1**: The API Catalog plugin SHALL generate `Documentation/Ed-Fi-API-Catalog/Ed-Fi-API-Catalog.xlsx`. +- **FR-API-CATALOG-2**: The API Catalog SHALL include `Resources` and `Properties` worksheets. +- **FR-API-CATALOG-3**: The API Catalog `Resources` worksheet SHALL include project, version, resource name, resource description, and domains. +- **FR-API-CATALOG-4**: The API Catalog `Properties` worksheet SHALL include project, version, resource name, property name, property description, data type, min length, max length, validation regular expression, identity key flag, nullable flag, and required flag. +- **FR-API-CATALOG-5**: The API Catalog SHALL omit the top-level `id` property from the properties worksheet. +- **FR-API-CATALOG-6**: The API Catalog SHALL recurse into common sub-schemas and reference schemas and represent nested property origins with dot-separated property paths. + +### 3.4 Model Validation + +- **FR-VAL-1**: The pipeline SHALL collect syntax validation failures per loaded file before building the aggregate parse tree and SHALL include them in the final validation result; when `stopOnValidationFailure` is enabled, existing error-category failures SHALL prevent plugin enhancers, plugin generators, and output writing. +- **FR-VAL-2**: The unified plugin SHALL validate core model relationships, including unresolved references, duplicate names, illegal extensions and subclasses, redeclared properties, invalid identity declarations and renames, invalid domain/subdomain/interchange membership, invalid namespace casing, and invalid shared/simple property bounds. +- **FR-VAL-3**: The unified advanced plugin SHALL validate merge directive paths and property compatibility, out-reference paths that need a merge directive or role name, common-property identity requirements, identity-name conflicts, and self-referencing property role names; it SHALL also report deprecation warnings for deprecated entities, extensions, subclasses, properties, domain item references, and interchange item references. +- **FR-VAL-4**: The ODS relational plugin SHALL warn for properties named `Discriminator` when the target technology version is `>=5.1.0`. +- **FR-VAL-5**: The change query plugin SHALL reject namespaces named `Changes`, case-insensitively, because that name is reserved by generated change query artifacts. +- **FR-VAL-6**: The ODS/API plugin SHALL warn when required choice properties appear in extensions and SHALL reject unsupported extension subclasses except EducationOrganization domain entity subclasses and GeneralStudentProgramAssociation association subclasses. +- **FR-VAL-7**: The XSD plugin SHALL suppress XSD generation for a namespace with duplicate entity names in dependency namespaces and SHALL report validation through the duplicate-name validator. +- **FR-VAL-8**: Deprecation validators SHALL report warnings for extension namespace elements, and SHALL report warnings for non-extension namespace elements only when `allianceMode` is enabled. + +### 3.5 Deploy Command + +- **FR-DEPLOY-1**: `@edfi/metaed-odsapi-deploy-console` SHALL run from Node.js and execute deploy tasks from `@edfi/metaed-odsapi-deploy`. +- **FR-DEPLOY-2**: The deploy CLI SHALL require `-a` / `--accept-license`. +- **FR-DEPLOY-3**: The deploy CLI SHALL accept configuration through `-c` / `--config`. +- **FR-DEPLOY-4**: When no configuration is supplied, the deploy CLI SHALL use `-s` / `--source`, `-t` / `--target`, and `-p` / `--projectNames` to scan project source directories, build artifacts into `MetaEdOutput` under the last resolved project path, and then deploy. +- **FR-DEPLOY-5**: The deploy CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for build and deploy behavior. +- **FR-DEPLOY-6**: In source-scan mode, the deploy CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when the option is not supplied; in config-based mode, deploy SHALL use the `suppressPrereleaseVersion` value supplied by `metaEdConfiguration`. +- **FR-DEPLOY-7**: The deploy CLI SHALL accept `--core` to deploy core artifacts in addition to extension artifacts. +- **FR-DEPLOY-8**: The deploy CLI SHALL accept `--suppressDelete` to skip deletion of existing extension artifact directories. +- **FR-DEPLOY-9**: The deploy CLI SHALL accept `--additionalMssqlScriptsDirectory` and `--additionalPostgresScriptsDirectory`; when a core or extension artifact copy task runs for a supported `defaultPluginTechVersion`, deploy SHALL copy those scripts into the corresponding deployed ODS data folders. +- **FR-DEPLOY-10**: For `defaultPluginTechVersion` values `>=3.0.0`, deploy SHALL check that extension C# project directories exist under `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/` for each immediate artifact directory except `ApiSchema`, `Documentation`, and `EdFi`. +- **FR-DEPLOY-11**: Unless suppressed, deploy SHALL remove existing `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Artifacts` directories for `defaultPluginTechVersion` values `>=3.3.0`. +- **FR-DEPLOY-12**: Core artifact copy SHALL run only when `--core` is supplied and `defaultPluginTechVersion` satisfies `>=7.0.0`. +- **FR-DEPLOY-13**: Extension artifact copy SHALL run only when `defaultPluginTechVersion` satisfies `>=7.0.0`. +- **FR-DEPLOY-14**: Core deploy SHALL copy from artifact namespace `EdFi` into `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{dataStandardVersion}/Artifacts/`. +- **FR-DEPLOY-15**: Extension deploy SHALL copy each non-EdFi project from an artifact source folder named by `projectName` into `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Versions/{projectVersion}/Standard/{dataStandardVersion}/Artifacts/`. +- **FR-DEPLOY-16**: For ODS/API technology versions satisfying `>=7.1.0`, deploy SHALL format Data Standard versions with prerelease suppression according to `suppressPrereleaseVersion`. +- **FR-DEPLOY-17**: Deploy SHALL refresh existing extension `.csproj` files by updating their filesystem modification timestamp after copy tasks run. +- **FR-DEPLOY-18**: For `defaultPluginTechVersion` values `>=3.3.0`, deploy SHALL warn when legacy `SupportingArtifacts` directories exist for the standard or extension projects. +- **FR-DEPLOY-19**: Deploy SHALL skip a generated source folder when that folder is absent rather than failing the entire deploy. +- **FR-DEPLOY-20**: Deploy SHALL map generated folders to ODS/API artifact folders as follows: + +| Generated Folder | Deployed Folder | +| ------------------------------------ | ---------------------- | +| `ApiMetadata/` | `Metadata/` | +| `Database/SQLServer/ODS/Data/` | `MsSql/Data/Ods/` | +| `Database/SQLServer/ODS/Structure/` | `MsSql/Structure/Ods/` | +| `Database/PostgreSQL/ODS/Data/` | `PgSql/Data/Ods/` | +| `Database/PostgreSQL/ODS/Structure/` | `PgSql/Structure/Ods/` | +| `Interchange/` | `Schemas/` | +| `XSD/` | `Schemas/` | + +- **FR-DEPLOY-21**: Deploy SHALL leave generated `ApiSchema/` and `Documentation/` outputs in the artifact directory because deploy tasks only map API metadata, database, interchange, and XSD artifact families into the ODS/API source tree. + +### 3.6 API Schema Packaging Automation + +- **FR-PKG-1**: The repository MAY create MetaEd build configuration files for API Schema packaging automation using `eng/ApiSchema/CreateMetaEdConfig.ps1`. +- **FR-PKG-2**: API Schema packaging configuration generation SHALL write `eng/ApiSchema/MetaEdConfig.json` with `artifactDirectory` set to the packaging `MetaEdOutput` path, a core namespace `EdFi` project named `Ed-Fi`, an optional extension project, `defaultPluginTechVersion`, `allianceMode` set to `true`, and `suppressPrereleaseVersion` set to `true`. Extension project versions SHALL come from `metaEdProject.projectVersion` in the extension `package.json` when present; otherwise the default SHALL be `1.1.0` for TPDM and `1.0.0` for other extensions. +- **FR-PKG-3**: The API Schema packaging automation MAY run MetaEd through `eng/ApiSchema/build.ps1 -Command RunMetaEd`. +- **FR-PKG-4**: The API Schema packaging automation MAY copy generated `ApiSchema*.json` files from `MetaEdOutput` into `eng/ApiSchema` package projects for Core, TPDM, Homograph, or Sample package variants; it MAY also copy core XSD and interchange files for Core and extension XSD/interchange files for TPDM and Sample, while the Homograph packaging path currently copies only API Schema JSON. +- **FR-PKG-5**: The API Schema packaging automation MAY build, pack, and publish .NET packages through `dotnet` and Azure Artifacts when the corresponding workflow commands and credentials are supplied. + +## 4. Non-Functional Requirements + +### 4.1 Runtime and Compatibility + +- **NFR-RUN-1**: Runtime packages SHALL execute under Node.js. +- **NFR-RUN-2**: TypeScript SHALL compile to CommonJS modules targeting ES2017. +- **NFR-RUN-3**: CI SHALL verify repository lint and test workflows on Node 22. +- **NFR-RUN-4**: Generated SQL artifacts SHALL distinguish SQL Server and PostgreSQL output directories and generator plugins. + +### 4.2 Reliability and Data Safety + +- **NFR-REL-1**: The CLI SHALL return non-zero exit codes for validation errors, pipeline failures, uncaught pipeline exceptions, and unsuccessful deploy tasks. +- **NFR-REL-2**: The output writer SHALL protect source projects by refusing to delete output directories that do not contain `MetaEdOutput` in the path. +- **NFR-REL-3**: The output writer SHALL protect source projects by refusing to write into a directory containing `.metaed` files. +- **NFR-REL-4**: Deploy tasks SHALL stop at the first unsuccessful task and return the failure message. + +### 4.3 Security and Privacy + +- **NFR-SEC-1**: Normal build and deploy runtime behavior SHALL operate on local files and SHALL NOT require credentials. +- **NFR-SEC-2**: Publishing workflows MAY use package-feed credentials, but those credentials are outside normal build and deploy runtime behavior. +- **NFR-SEC-3**: Packages SHALL be licensed as Apache-2.0 as declared in package metadata and repository license files. + +### 4.4 Observability + +- **NFR-OBS-1**: Build and deploy commands SHALL log major stages, plugin execution, artifact directory, deploy copy operations, warnings, and elapsed time. +- **NFR-OBS-2**: Validation failures SHALL carry category, message, and source/file mapping where available. + +### 4.5 SDLC Quality + +- **NFR-SDLC-1**: Pull request CI SHALL run TypeScript and ESLint checks through `npm run test:lint`. +- **NFR-SDLC-2**: Pull request CI SHALL run non-database unit tests, SQL Server database tests, and PostgreSQL database tests. +- **NFR-SDLC-3**: Repository workflows SHOULD run dependency review, CodeQL analysis, and OpenSSF Scorecard checks where configured. +- **NFR-SDLC-4**: Release workflows MAY publish npm packages to Azure Artifacts using Lerna. + +## 5. System Architecture + +### 5.1 Processing Pipeline + +The build pipeline is sequential: + +1. Initialize MetaEd environment. +2. Set up plugins and target technology versions. +3. Load `.metaed` files. +4. Validate syntax. +5. Load file indexes. +6. Build the parse tree. +7. Walk builders to construct the semantic model. +8. Initialize namespaces. +9. Load plugin configuration. +10. For each plugin in dependency order, run validators, enhancers, and generators. +11. Write generated output. +12. Map validation failures to files. + +Plugins are ordered by `@edfi/metaed-default-plugins`, allowing upstream plugins to create semantic data used by downstream artifact generators. + +### 5.2 Component Packages + +| Package | Product Responsibility | +| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `@edfi/metaed-core` | Core pipeline, configuration model, file loading, grammar parsing, model builders, validators/enhancers/generators contracts, plugin configuration loading, and output writing. | +| `@edfi/metaed-console` | Build CLI for running the MetaEd pipeline with default plugins. | +| `@edfi/metaed-default-plugins` | Default ordered plugin list used by the build and deploy-console build flow. | +| `@edfi/metaed-odsapi-deploy` | Deploy task library for copying generated artifacts into an ODS/API source tree. | +| `@edfi/metaed-odsapi-deploy-console` | Deploy CLI for source-scan build-and-deploy mode or config-based deploy mode. | +| `@edfi/metaed-plugin-edfi-unified` | Foundational Ed-Fi semantic validators and enhancers, including reference resolution, base class resolution, domain/subdomain links, merge directives, and shared type enrichment. | +| `@edfi/metaed-plugin-edfi-unified-advanced` | Additional validation for merge scenarios, deprecation warnings, common property identity rules, and self-referencing property role names. | +| `@edfi/metaed-plugin-edfi-ods-relational` | Shared relational ODS model enhancers and relational validators used by SQL Server, PostgreSQL, API Schema, dictionaries, change query, and record ownership plugins. | +| `@edfi/metaed-plugin-edfi-api-schema` | API Schema model enhancers, security-related configuration rules, OpenAPI fragment and relational metadata, and `ApiSchema*.json` generation with API Schema version metadata. | +| `@edfi/metaed-plugin-edfi-ods-sqlserver` | SQL Server ODS naming, schema/table/column enhancements, and SQL Server ODS SQL script generation. | +| `@edfi/metaed-plugin-edfi-ods-postgresql` | PostgreSQL ODS naming, schema/table/column enhancements, and PostgreSQL ODS SQL script generation. | +| `@edfi/metaed-plugin-edfi-xsd` | XSD model enhancers, duplicate-name validation for dependency namespaces, core/extension XSD generation, schema annotation generation, and interchange XSD generation. | +| `@edfi/metaed-plugin-edfi-ods-changequery` | Common change query validator, common model types, and shared generation helpers for change query SQL artifacts. | +| `@edfi/metaed-plugin-edfi-ods-changequery-sqlserver` | SQL Server-specific change query enhancers and SQL script generators. | +| `@edfi/metaed-plugin-edfi-ods-changequery-postgresql` | PostgreSQL-specific change query enhancers and SQL script generators. | +| `@edfi/metaed-plugin-edfi-ods-recordownership` | Common record ownership table enhancement that marks aggregate root tables for ownership token columns for target technology versions `>=3.3.0`. | +| `@edfi/metaed-plugin-edfi-ods-recordownership-sqlserver` | SQL Server record ownership SQL generation. | +| `@edfi/metaed-plugin-edfi-ods-recordownership-postgresql` | PostgreSQL record ownership SQL generation. | +| `@edfi/metaed-plugin-edfi-odsapi` | ODS/API model validators/enhancers and `ApiModel*.json` generation. | +| `@edfi/metaed-plugin-edfi-xml-dictionary` | XML data dictionary Excel generation from XSD model data. | +| `@edfi/metaed-plugin-edfi-sql-dictionary` | SQL data dictionary Excel generation from relational ODS model data. | +| `@edfi/metaed-plugin-edfi-handbook` | Handbook-specific model enhancers plus HTML and Excel Data Handbook generation. | +| `@edfi/metaed-plugin-edfi-api-catalog` | API Catalog Excel generation from API Schema and OpenAPI fragment data. | + +### 5.3 Default Plugin Order + +The default plugin order is: + +`edfiUnified`, `edfiUnifiedAdvanced`, `edfiOdsRelational`, `edfiApiSchema`, `edfiOdsSqlServer`, `edfiOdsPostgresql`, `edfiXsd`, `edfiOdsChangeQuery`, `edfiOdsChangeQuerySqlServer`, `edfiOdsChangeQueryPostgresql`, `edfiOdsRecordOwnership`, `edfiOdsRecordOwnershipSqlServer`, `edfiOdsRecordOwnershipPostgresql`, `edfiOdsApi`, `edfiXmlDictionary`, `edfiSqlDictionary`, `edfiHandbook`, `edfiApiCatalog`. + +### 5.4 Output Model + +Generators return `GeneratedOutput` objects containing a human-readable name, namespace, folder name, file name, and either string or binary stream output. The writer creates directories recursively and writes files beneath the artifact directory using `{namespace}/{folderName}/{fileName}`. + +## 6. Out of Scope and Known Limitations + +### 6.1 Explicit Exclusions + +- The MetaEd DSL language specification and grammar semantics are outside this PRD. +- External IDE behavior is outside this repository and outside this PRD. +- ODS/API runtime behavior, API service hosting, database execution, migrations, and production deployment are outside this PRD. +- Data Standard content itself is outside this PRD; this repository processes model files but does not define every data element requirement here. + +### 6.2 Known Limitations + +- `MetaEdConfiguration` includes `pluginTechVersion`, and some fixtures contain it, but current plugin setup assigns every plugin the single `defaultPluginTechVersion`. +- The license acceptance CLI option is required by yargs, but the source does not validate the option value or persist acceptance. +- The build console's custom config loader resolves relative `-c` / `--config` paths relative to the console module directory; absolute config paths avoid that ambiguity. +- Config-based deploy mode runs deploy tasks against an existing `artifactDirectory`; source-scan mode is the path that builds before deploy. +- Config-based deploy mode copies the supplied `metaEdConfiguration` without merging defaults from `newMetaEdConfiguration`, and it does not apply the `--suppressPrereleaseVersion` CLI option to the supplied configuration. +- Source-scan deploy mode returns without deploying when required source-mode inputs are absent, including `source` or `projectNames`. +- After source-scan deploy mode attempts a build, the deploy console still invokes deploy tasks even when the build has set a non-zero exit code; if an existing artifact directory remains, deploy can attempt to copy those artifacts. +- Source-scan project discovery scans subdirectories only while no projects have been discovered yet, so mixed source inputs can miss nested projects after an earlier project has already been found. +- Source-scan project discovery de-duplicates projects by project name, so two discovered projects with the same project name cannot both be represented in one deploy build. +- Source-scan namespace derivation returns an empty namespace when `projectName` cannot be converted to an alphanumeric string that starts with an uppercase letter. +- Deploy copies extension artifacts from folders named by `projectName`, while generators write output folders using `namespaceName`; source-scan and config-based deployments where those values differ can cause deploy to skip generated extension folders. +- Source-scan project sorting only special-cases a project whose `projectName` is exactly `EdFi`; Data Standard packages with `projectName` values such as `Ed-Fi` are sorted alphabetically with other discovered projects after namespace derivation. +- Core and extension artifact copy tasks are no-ops for `defaultPluginTechVersion` values below `7.0.0`. +- A core-only deploy with no extension artifact folders can fail the extension project existence precheck when `allianceMode` is false, because that precheck runs before core copy and ignores root `EdFi`, `Documentation`, and `ApiSchema` folders. +- Deploy does not copy generated `ApiSchema/` or `Documentation/` artifact folders into the ODS/API source tree. +- Deploy assumes the local ODS/API source tree follows expected `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folder conventions. +- The console packages declare Node.js `main` entry points but no package `bin` commands, so command execution is through Node.js entry points or repository scripts rather than package-installed binary names. +- API Schema packaging `RunMetaEd` copies `packages/metaed-plugin-edfi-api-schema/test/integration/edfiApiSchema.config.json` into the core project path before running MetaEd, so packaging output depends on that checked-in configuration fixture. +- The API Schema packaging workflow matrix uses `TPDM`, but the TPDM checkout step currently checks for lowercase `tpdm`, so that workflow step does not run for the TPDM matrix entries as written. +- The API Catalog generator contains hard-coded handling for EducationOrganization and SchoolYear reference schemas because those reference schemas are not always discoverable from generated OpenAPI fragments. +- The output writer's deletion guard is path-name based: it requires `MetaEdOutput` to appear in the output path before deleting an existing output directory. + +## 7. Glossary + +| Term | Definition | +| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| MetaEd-js | The Node.js monorepo that parses MetaEd DSL projects and generates artifacts through plugins. | +| MetaEd DSL | The `.metaed` source language processed by this repository. | +| Data Standard project | The core Ed-Fi project identified by namespace `EdFi`; exactly one is required by the CLIs. | +| Extension project | A non-EdFi MetaEd project processed alongside the Data Standard project. | +| Namespace | The model and output grouping used by MetaEd to separate core and extension artifacts. | +| Project extension | A string used by generators in some extension artifact file names; non-EdFi projects default to `EXTENSION` when metadata omits it. | +| Default plugin technology version | The version string assigned to plugin environments to select version-specific generator behavior. | +| Plugin | A package that contributes validators, enhancers, generators, and optional configuration schemas. | +| Validator | A plugin function that reports syntax, semantic, relationship, or configuration failures. | +| Enhancer | A plugin function that adds derived semantic data to the model for downstream processing. | +| Generator | A plugin function that produces one or more generated outputs. | +| Artifact | A generated file written by the build pipeline, such as SQL, XSD, JSON, HTML, or Excel. | +| ODS | Operational Data Store, represented here by generated SQL artifacts and ODS/API source tree folders. | +| ODS/API | The downstream Ed-Fi source tree layout targeted by deploy tasks. | +| API Model | ODS/API metadata JSON generated as `ApiModel*.json`. | +| API Schema | JSON schema output generated as `ApiSchema*.json` for API schema consumers. | +| Change Query | SQL artifacts for change version, tracked delete, and related change tracking support. | +| Record Ownership | SQL artifacts that add ownership token support to aggregate root tables when enabled by target version. | From 62e8b04a7ea39434bacf5c0e08f39719b5a0e559 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Mon, 22 Jun 2026 08:38:44 -0500 Subject: [PATCH 11/15] doc: merge Brad's PRD with Stephen's original --- docs/Iterative-PRD.md | 350 --------------------------- docs/PRD-4.x.md | 532 ++++++++++++++++++++++++------------------ 2 files changed, 300 insertions(+), 582 deletions(-) delete mode 100644 docs/Iterative-PRD.md diff --git a/docs/Iterative-PRD.md b/docs/Iterative-PRD.md deleted file mode 100644 index 0ed8145d6..000000000 --- a/docs/Iterative-PRD.md +++ /dev/null @@ -1,350 +0,0 @@ -# MetaEd-js Product Requirements Document - -- Status: Draft, reverse-engineered from repository evidence -- Repository: `MetaEd-js` -- Package version source: Lerna version `4.7.1-dev.1` -- Platform and runtime: Node.js packages compiled from TypeScript to ES2017/CommonJS; CI runs Node 22 -- Document scope: Product behavior implemented in this repository only -- Evidence basis: package metadata, source entry points, CLI/deploy/configuration code, plugin generators, tests, authoritative artifacts, and workflow configuration - -> [!NOTE] Document Scope -> This PRD describes the MetaEd-js command-line generator, deployment utility, default plugin set, and generated artifact families as implemented by package metadata, source code, tests, authoritative artifacts, and workflow configuration in this repository. -> -> It does not define the MetaEd DSL language specification, Ed-Fi ODS/API runtime behavior, database execution semantics, or behavior of an external IDE extension. - -## 1. Product Overview - -MetaEd-js is a Node.js monorepo that compiles Ed-Fi MetaEd DSL projects into local filesystem artifacts used by downstream Ed-Fi data platform tooling. A MetaEd project is represented by `.metaed` source files plus project metadata such as project name, namespace, version, optional project extension, and description. When project metadata omits a project extension, generated extension artifact names use `EXTENSION` while the core `EdFi` namespace uses no project extension. The generator reads one Data Standard project and zero or more extension projects, validates and enriches the semantic model, runs the default plugin set, and writes generated outputs grouped by namespace. - -The repository provides two primary command-line workflows: - -- `@edfi/metaed-console`: runs the MetaEd build pipeline and writes generated artifacts. -- `@edfi/metaed-odsapi-deploy-console`: optionally builds from project source directories and then copies generated artifacts into a local Ed-Fi ODS/API source tree layout for supported technology versions. - -Repository automation also includes a manually dispatchable API Schema packaging workflow under `eng/ApiSchema` that runs MetaEd, moves selected generated API Schema/XSD artifacts into .NET package projects, packs NuGet packages, and publishes those packages through GitHub Actions when feed credentials are supplied. - -The product value is artifact consistency: SQL, XSD, API metadata, API schema, documentation, and dictionary outputs are generated from the same MetaEd model rather than authored independently. - -### 1.1 Target Users and Personas - -- **Data Standard maintainers** maintain the core Ed-Fi Data Standard MetaEd project and need repeatable generated artifacts for release and packaging workflows. -- **Extension developers** maintain additive MetaEd extension projects and need validation plus generated artifacts for ODS/API integration. -- **Build and release engineers** run MetaEd headlessly in CI, publish npm packages, package API Schema artifacts, and verify generated outputs against authoritative fixtures. -- **Downstream artifact consumers** use generated spreadsheets, HTML documentation, JSON schema/metadata files, SQL scripts, and XSD files without reading `.metaed` source. - -### 1.2 Jobs To Be Done - -- **JTBD-1: Validate a model.** When a user changes `.metaed` files, they need the pipeline to report syntax, semantic, configuration, and plugin validation failures in a machine-detectable build result. -- **JTBD-2: Generate platform artifacts.** When a user has a valid Data Standard or extension model, they need one build command to produce all configured SQL, XSD, JSON, HTML, and Excel outputs. -- **JTBD-3: Deploy generated artifacts to a local ODS/API source tree.** When a user has generated artifacts for ODS/API technology version 7 or later, they need a deployment command to copy those artifacts into the expected `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` artifact folders. -- **JTBD-4: Produce documentation from the model.** When a user needs to inspect generated model shape, they need handbook, API catalog, SQL dictionary, and XML dictionary outputs that are generated from the same model pipeline. -- **JTBD-5: Run without an IDE.** When a build or release engineer automates MetaEd, they need Node.js command-line packages that can run in CI with deterministic exit codes. - -## 2. Enterprise and System Context - -### 2.1 External Systems and Integrations - -- **Local MetaEd project directories** provide `.metaed` files and optional `package.json` files with `metaEdProject` metadata. -- **Ed-Fi Data Standard npm packages** are package dependencies of `@edfi/metaed-console` for model packages `@edfi/ed-fi-model-4.0`, `@edfi/ed-fi-model-5.0`, `@edfi/ed-fi-model-5.1`, `@edfi/ed-fi-model-5.2`, `@edfi/ed-fi-model-6.0`, and `@edfi/ed-fi-model-6.1`. -- **Ed-Fi ODS/API source trees** consume deployed artifact files under `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` directories. -- **SQL Server and PostgreSQL** are downstream targets for generated database scripts; MetaEd-js generates scripts but does not execute them as part of the build pipeline. -- **API Schema and API metadata consumers** consume `ApiSchema*.json` and `ApiModel*.json` outputs. -- **.NET SDK and NuGet/Azure Artifacts** are used by API Schema packaging automation under `eng/ApiSchema`. -- **GitHub Actions and Azure Artifacts** are used by repository workflows for linting, testing, publishing npm packages, and packaging API Schema artifacts. -- **ANTLR 4.6 generated JavaScript grammar files** are part of the source tree used by the parser. - -### 2.2 Deployment Boundaries - -- MetaEd-js build and deploy commands operate on local files. The build pipeline reads `.metaed` source files and plugin configuration files, then writes artifacts to disk. -- Deploy copies files into a local ODS/API source tree layout. It does not deploy to running services, execute SQL, start databases, or validate ODS/API runtime behavior. -- Publishing workflows communicate with package feeds, but normal build and deploy runtime behavior is filesystem-based. - -## 3. Functional Requirements - -### 3.1 Project Input and Configuration - -- **FR-CFG-1**: The build CLI SHALL accept a JSON configuration file through `-c` / `--config` containing a top-level `metaEdConfiguration` object. -- **FR-CFG-2**: `metaEdConfiguration` SHALL support `artifactDirectory`, `deployDirectory`, `pluginTechVersion`, `projects`, `projectPaths`, `pluginConfigDirectories`, `defaultPluginTechVersion`, `allianceMode`, and `suppressPrereleaseVersion`; it MAY include `externalVariables` for Jsonnet configuration evaluation. -- **FR-CFG-3**: The pipeline SHALL require `projects` and `projectPaths` to have the same length and SHALL map each project metadata entry to the corresponding source path. -- **FR-CFG-4**: The file loader SHALL recursively load files with `.metaed`, `.metaEd`, `.MetaEd`, or `.METAED` extensions from each configured project path. -- **FR-CFG-5**: The build and deploy CLIs SHALL require exactly one Data Standard project, identified by namespace `EdFi`; no Data Standard project or multiple Data Standard projects SHALL cause failure. -- **FR-CFG-6**: The project scanner used by the deploy console source mode SHALL read `package.json` files containing `metaEdProject`, derive `namespaceName` by removing non-alphanumeric characters from `projectName` only when the result starts with an uppercase letter, include the package description when present, sort a project whose `projectName` is exactly `EdFi` ahead of other projects, otherwise sort discovered projects by project name, and de-duplicate discovered projects by project name. -- **FR-CFG-7**: The project scanner MAY accept `projectNames` overrides that update discovered project names and derived namespaces in discovery order. -- **FR-CFG-8**: Plugin configuration files SHALL be discovered from configured `pluginConfigDirectories`, or from input project directories when no plugin config directories are configured, as `{pluginShortName}.config.jsonnet` or `{pluginShortName}.config.json`, with Jsonnet preferred when both exist. -- **FR-CFG-9**: Plugin configuration loading SHALL support `externalVariables` for Jsonnet evaluation. -- **FR-CFG-10**: Plugin configuration rules SHALL support plugin-wide data or entity-matched data using `entity`, `namespace`, `core`, `extensions`, and `entityName` matching fields. -- **FR-CFG-11**: Configuration matching SHALL reject a single match rule that combines `namespace` with `core` or `extensions`. -- **FR-CFG-12**: The API Schema plugin SHALL validate plugin-specific configuration rules for `educationOrganizationSecurableElements` and `educationOrganizationIdentitySecurableElements`. -- **FR-CFG-13**: When project metadata does not provide `projectExtension`, the file loader SHALL use an empty project extension for namespace `EdFi` and `EXTENSION` for non-EdFi namespaces. -- **FR-CFG-14**: File loading SHALL fail the build when no `.metaed` files are found in any configured input directory. - -### 3.2 Build Command - -- **FR-BUILD-1**: `@edfi/metaed-console` SHALL run from Node.js and execute the MetaEd pipeline using the default plugin set. -- **FR-BUILD-2**: The build CLI SHALL require `-a` / `--accept-license` before executing. -- **FR-BUILD-3**: The build CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for all plugins. -- **FR-BUILD-4**: The default plugin technology version SHALL be `6.0.0` when not overridden by configuration or CLI argument. -- **FR-BUILD-5**: The build CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when no configuration object supplies a value. -- **FR-BUILD-6**: The build pipeline SHALL run initialization, plugin setup, file loading, syntax validation, file indexing, parse tree building, model building, namespace initialization, plugin configuration loading, validators, enhancers, generators, output writing, and validation file mapping. -- **FR-BUILD-7**: The build pipeline SHALL stop plugin execution on validation errors when `stopOnValidationFailure` is enabled. -- **FR-BUILD-8**: The build CLI SHALL exit with code `0` only when no validation error and no pipeline failure occurred; otherwise it SHALL exit with code `1`. -- **FR-BUILD-9**: Generated outputs SHALL be written under the configured artifact directory or a `MetaEdOutput` directory under the last input project path. Relative `artifactDirectory` values SHALL be resolved from the last input project path. -- **FR-BUILD-10**: Output writing SHALL group generated files by namespace, with the generator-provided folder path under that namespace. -- **FR-BUILD-11**: Output writing SHALL refuse to delete an existing output directory unless the path contains `MetaEdOutput`. -- **FR-BUILD-12**: Output writing SHALL refuse to write when `.metaed` files are found in the output location. - -### 3.3 Generated Artifact Families - -#### 1. ODS/API Metadata - -- **FR-ODSAPI-METADATA-1**: The build SHALL generate ODS/API metadata JSON files per namespace as `ApiMetadata/ApiModel.json` when `projectExtension` is empty or `ApiMetadata/ApiModel-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-odsapi`. - -#### 2. API Schema - -- **FR-API-SCHEMA-1**: The build SHALL generate API Schema JSON files per namespace as `ApiSchema/ApiSchema.json` when `projectExtension` is empty or `ApiSchema/ApiSchema-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-api-schema`. -- **FR-API-SCHEMA-2**: API Schema output SHALL include `apiSchemaVersion` and a project schema containing project identity, endpoint name, `compatibleDsRange` for extension-project compatibility, description, resource schemas, resource-name mappings, case-insensitive endpoint-name mappings, abstract resources, education organization information, domains, extension-project flag, and core OpenAPI base documents where generated by the API Schema model. Resource schemas SHALL include generated JSON schema-for-insert data, identity JSON paths, document path mappings, equality constraints, type-coercion JSON paths, decimal validation information, securable elements, authorization pathways, array uniqueness constraints, OpenAPI fragments, query-field mappings where applicable, school year enumeration resource schema when present, and optional relational naming metadata where generated. - -#### 3. SQL Server and PostgreSQL ODS SQL - -- **FR-ODS-SQL-1**: The build SHALL generate SQL Server ODS structure and data scripts under `Database/SQLServer/ODS/`. -- **FR-ODS-SQL-2**: The build SHALL generate PostgreSQL ODS structure and data scripts under `Database/PostgreSQL/ODS/`. -- **FR-ODS-SQL-3**: SQL ODS outputs SHALL include schema, table, foreign key, extended property, enumeration, and school year scripts when corresponding model data exists; core file names SHALL use `{prefix}-{suffix}.sql`, and extension file names SHALL include namespace context as `{prefix}-{projectExtension}-{namespaceName}-{suffix}.sql` or `{prefix}-{namespaceName}-{suffix}.sql` when `projectExtension` is empty. -- **FR-ODS-SQL-4**: SQL ODS outputs SHALL include ID column unique index scripts for tables that require generated ID indexes. -- **FR-ODS-SQL-5**: For technology versions `>=7.1.0`, SQL ODS outputs SHALL include education organization authorization index creation scripts when table data exists. -- **FR-ODS-SQL-6**: For technology versions `>=7.3.0`, SQL ODS outputs SHALL include aggregate ID column scripts and education organization authorization index update scripts when table data exists. -- **FR-ODS-SQL-7**: SQL-generating plugins SHALL include Apache-2.0 license headers in supported SQL templates only when `allianceMode` is enabled and the relevant target technology version satisfies `>=5.0.0`. - -#### 4. Change Query SQL - -- **FR-CHANGE-QUERY-SQL-1**: The change query plugins SHALL generate change schema, change version sequence, change version column/index, tracked delete schema, tracked delete table, delete tracking trigger, change version/key change trigger, and indirect update cascade trigger scripts under `Database/{SQLServer|PostgreSQL}/ODS/Structure/Changes/` when required model data exists. For target technology versions `>=7.3.0`, shared change schema and sequence files SHALL be generated for the core namespace and skipped for extension namespaces, and indirect update cascade triggers SHALL be generated when required model data exists. - -#### 5. Record Ownership SQL - -- **FR-RECORD-OWNERSHIP-SQL-1**: The record ownership plugins SHALL generate `0010-AddColumnOwnershipTokenForTable.sql` under `Database/{SQLServer|PostgreSQL}/ODS/Structure/RecordOwnership/` for aggregate root tables when the target technology version satisfies `>=3.3.0`. - -#### 6. XSD and Interchange Schemas - -- **FR-XSD-1**: The XSD plugin SHALL generate `XSD/Ed-Fi-Core.xsd` for the core namespace and `{projectExtension}-Ed-Fi-Extended-Core.xsd` for extension namespaces. -- **FR-XSD-2**: The XSD plugin SHALL generate `XSD/SchemaAnnotation.xsd` for the core namespace. -- **FR-XSD-3**: The XSD plugin SHALL generate interchange schemas under `Interchange/`, using `Interchange-{Name}.xsd` for core interchanges and `{projectExtension}-Interchange-{Name}-Extension.xsd` for extension interchanges. - -#### 7. Data Handbook - -- **FR-HANDBOOK-1**: The handbook plugin SHALL generate `Documentation/Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` and `Documentation/Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx`. - -#### 8. Data Dictionaries - -- **FR-DICTIONARY-1**: The SQL dictionary plugin SHALL generate `Documentation/DataDictionary/SqlDataDictionary.xlsx` with table and column worksheets derived from relational ODS model data. -- **FR-DICTIONARY-2**: The XML dictionary plugin SHALL generate `Documentation/DataDictionary/XmlDataDictionary.xlsx` with elements, complex types, and simple types worksheets derived from XSD model data. - -#### 9. API Catalog - -- **FR-API-CATALOG-1**: The API Catalog plugin SHALL generate `Documentation/Ed-Fi-API-Catalog/Ed-Fi-API-Catalog.xlsx`. -- **FR-API-CATALOG-2**: The API Catalog SHALL include `Resources` and `Properties` worksheets. -- **FR-API-CATALOG-3**: The API Catalog `Resources` worksheet SHALL include project, version, resource name, resource description, and domains. -- **FR-API-CATALOG-4**: The API Catalog `Properties` worksheet SHALL include project, version, resource name, property name, property description, data type, min length, max length, validation regular expression, identity key flag, nullable flag, and required flag. -- **FR-API-CATALOG-5**: The API Catalog SHALL omit the top-level `id` property from the properties worksheet. -- **FR-API-CATALOG-6**: The API Catalog SHALL recurse into common sub-schemas and reference schemas and represent nested property origins with dot-separated property paths. - -### 3.4 Model Validation - -- **FR-VAL-1**: The pipeline SHALL collect syntax validation failures per loaded file before building the aggregate parse tree and SHALL include them in the final validation result; when `stopOnValidationFailure` is enabled, existing error-category failures SHALL prevent plugin enhancers, plugin generators, and output writing. -- **FR-VAL-2**: The unified plugin SHALL validate core model relationships, including unresolved references, duplicate names, illegal extensions and subclasses, redeclared properties, invalid identity declarations and renames, invalid domain/subdomain/interchange membership, invalid namespace casing, and invalid shared/simple property bounds. -- **FR-VAL-3**: The unified advanced plugin SHALL validate merge directive paths and property compatibility, out-reference paths that need a merge directive or role name, common-property identity requirements, identity-name conflicts, and self-referencing property role names; it SHALL also report deprecation warnings for deprecated entities, extensions, subclasses, properties, domain item references, and interchange item references. -- **FR-VAL-4**: The ODS relational plugin SHALL warn for properties named `Discriminator` when the target technology version is `>=5.1.0`. -- **FR-VAL-5**: The change query plugin SHALL reject namespaces named `Changes`, case-insensitively, because that name is reserved by generated change query artifacts. -- **FR-VAL-6**: The ODS/API plugin SHALL warn when required choice properties appear in extensions and SHALL reject unsupported extension subclasses except EducationOrganization domain entity subclasses and GeneralStudentProgramAssociation association subclasses. -- **FR-VAL-7**: The XSD plugin SHALL suppress XSD generation for a namespace with duplicate entity names in dependency namespaces and SHALL report validation through the duplicate-name validator. -- **FR-VAL-8**: Deprecation validators SHALL report warnings for extension namespace elements, and SHALL report warnings for non-extension namespace elements only when `allianceMode` is enabled. - -### 3.5 Deploy Command - -- **FR-DEPLOY-1**: `@edfi/metaed-odsapi-deploy-console` SHALL run from Node.js and execute deploy tasks from `@edfi/metaed-odsapi-deploy`. -- **FR-DEPLOY-2**: The deploy CLI SHALL require `-a` / `--accept-license`. -- **FR-DEPLOY-3**: The deploy CLI SHALL accept configuration through `-c` / `--config`. -- **FR-DEPLOY-4**: When no configuration is supplied, the deploy CLI SHALL use `-s` / `--source`, `-t` / `--target`, and `-p` / `--projectNames` to scan project source directories, build artifacts into `MetaEdOutput` under the last resolved project path, and then deploy. -- **FR-DEPLOY-5**: The deploy CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for build and deploy behavior. -- **FR-DEPLOY-6**: In source-scan mode, the deploy CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when the option is not supplied; in config-based mode, deploy SHALL use the `suppressPrereleaseVersion` value supplied by `metaEdConfiguration`. -- **FR-DEPLOY-7**: The deploy CLI SHALL accept `--core` to deploy core artifacts in addition to extension artifacts. -- **FR-DEPLOY-8**: The deploy CLI SHALL accept `--suppressDelete` to skip deletion of existing extension artifact directories. -- **FR-DEPLOY-9**: The deploy CLI SHALL accept `--additionalMssqlScriptsDirectory` and `--additionalPostgresScriptsDirectory`; when a core or extension artifact copy task runs for a supported `defaultPluginTechVersion`, deploy SHALL copy those scripts into the corresponding deployed ODS data folders. -- **FR-DEPLOY-10**: For `defaultPluginTechVersion` values `>=3.0.0`, deploy SHALL check that extension C# project directories exist under `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/` for each immediate artifact directory except `ApiSchema`, `Documentation`, and `EdFi`. -- **FR-DEPLOY-11**: Unless suppressed, deploy SHALL remove existing `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Artifacts` directories for `defaultPluginTechVersion` values `>=3.3.0`. -- **FR-DEPLOY-12**: Core artifact copy SHALL run only when `--core` is supplied and `defaultPluginTechVersion` satisfies `>=7.0.0`. -- **FR-DEPLOY-13**: Extension artifact copy SHALL run only when `defaultPluginTechVersion` satisfies `>=7.0.0`. -- **FR-DEPLOY-14**: Core deploy SHALL copy from artifact namespace `EdFi` into `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{dataStandardVersion}/Artifacts/`. -- **FR-DEPLOY-15**: Extension deploy SHALL copy each non-EdFi project from an artifact source folder named by `projectName` into `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Versions/{projectVersion}/Standard/{dataStandardVersion}/Artifacts/`. -- **FR-DEPLOY-16**: For ODS/API technology versions satisfying `>=7.1.0`, deploy SHALL format Data Standard versions with prerelease suppression according to `suppressPrereleaseVersion`. -- **FR-DEPLOY-17**: Deploy SHALL refresh existing extension `.csproj` files by updating their filesystem modification timestamp after copy tasks run. -- **FR-DEPLOY-18**: For `defaultPluginTechVersion` values `>=3.3.0`, deploy SHALL warn when legacy `SupportingArtifacts` directories exist for the standard or extension projects. -- **FR-DEPLOY-19**: Deploy SHALL skip a generated source folder when that folder is absent rather than failing the entire deploy. -- **FR-DEPLOY-20**: Deploy SHALL map generated folders to ODS/API artifact folders as follows: - -| Generated Folder | Deployed Folder | -| ------------------------------------ | ---------------------- | -| `ApiMetadata/` | `Metadata/` | -| `Database/SQLServer/ODS/Data/` | `MsSql/Data/Ods/` | -| `Database/SQLServer/ODS/Structure/` | `MsSql/Structure/Ods/` | -| `Database/PostgreSQL/ODS/Data/` | `PgSql/Data/Ods/` | -| `Database/PostgreSQL/ODS/Structure/` | `PgSql/Structure/Ods/` | -| `Interchange/` | `Schemas/` | -| `XSD/` | `Schemas/` | - -- **FR-DEPLOY-21**: Deploy SHALL leave generated `ApiSchema/` and `Documentation/` outputs in the artifact directory because deploy tasks only map API metadata, database, interchange, and XSD artifact families into the ODS/API source tree. - -### 3.6 API Schema Packaging Automation - -- **FR-PKG-1**: The repository MAY create MetaEd build configuration files for API Schema packaging automation using `eng/ApiSchema/CreateMetaEdConfig.ps1`. -- **FR-PKG-2**: API Schema packaging configuration generation SHALL write `eng/ApiSchema/MetaEdConfig.json` with `artifactDirectory` set to the packaging `MetaEdOutput` path, a core namespace `EdFi` project named `Ed-Fi`, an optional extension project, `defaultPluginTechVersion`, `allianceMode` set to `true`, and `suppressPrereleaseVersion` set to `true`. Extension project versions SHALL come from `metaEdProject.projectVersion` in the extension `package.json` when present; otherwise the default SHALL be `1.1.0` for TPDM and `1.0.0` for other extensions. -- **FR-PKG-3**: The API Schema packaging automation MAY run MetaEd through `eng/ApiSchema/build.ps1 -Command RunMetaEd`. -- **FR-PKG-4**: The API Schema packaging automation MAY copy generated `ApiSchema*.json` files from `MetaEdOutput` into `eng/ApiSchema` package projects for Core, TPDM, Homograph, or Sample package variants; it MAY also copy core XSD and interchange files for Core and extension XSD/interchange files for TPDM and Sample, while the Homograph packaging path currently copies only API Schema JSON. -- **FR-PKG-5**: The API Schema packaging automation MAY build, pack, and publish .NET packages through `dotnet` and Azure Artifacts when the corresponding workflow commands and credentials are supplied. - -## 4. Non-Functional Requirements - -### 4.1 Runtime and Compatibility - -- **NFR-RUN-1**: Runtime packages SHALL execute under Node.js. -- **NFR-RUN-2**: TypeScript SHALL compile to CommonJS modules targeting ES2017. -- **NFR-RUN-3**: CI SHALL verify repository lint and test workflows on Node 22. -- **NFR-RUN-4**: Generated SQL artifacts SHALL distinguish SQL Server and PostgreSQL output directories and generator plugins. - -### 4.2 Reliability and Data Safety - -- **NFR-REL-1**: The CLI SHALL return non-zero exit codes for validation errors, pipeline failures, uncaught pipeline exceptions, and unsuccessful deploy tasks. -- **NFR-REL-2**: The output writer SHALL protect source projects by refusing to delete output directories that do not contain `MetaEdOutput` in the path. -- **NFR-REL-3**: The output writer SHALL protect source projects by refusing to write into a directory containing `.metaed` files. -- **NFR-REL-4**: Deploy tasks SHALL stop at the first unsuccessful task and return the failure message. - -### 4.3 Security and Privacy - -- **NFR-SEC-1**: Normal build and deploy runtime behavior SHALL operate on local files and SHALL NOT require credentials. -- **NFR-SEC-2**: Publishing workflows MAY use package-feed credentials, but those credentials are outside normal build and deploy runtime behavior. -- **NFR-SEC-3**: Packages SHALL be licensed as Apache-2.0 as declared in package metadata and repository license files. - -### 4.4 Observability - -- **NFR-OBS-1**: Build and deploy commands SHALL log major stages, plugin execution, artifact directory, deploy copy operations, warnings, and elapsed time. -- **NFR-OBS-2**: Validation failures SHALL carry category, message, and source/file mapping where available. - -### 4.5 SDLC Quality - -- **NFR-SDLC-1**: Pull request CI SHALL run TypeScript and ESLint checks through `npm run test:lint`. -- **NFR-SDLC-2**: Pull request CI SHALL run non-database unit tests, SQL Server database tests, and PostgreSQL database tests. -- **NFR-SDLC-3**: Repository workflows SHOULD run dependency review, CodeQL analysis, and OpenSSF Scorecard checks where configured. -- **NFR-SDLC-4**: Release workflows MAY publish npm packages to Azure Artifacts using Lerna. - -## 5. System Architecture - -### 5.1 Processing Pipeline - -The build pipeline is sequential: - -1. Initialize MetaEd environment. -2. Set up plugins and target technology versions. -3. Load `.metaed` files. -4. Validate syntax. -5. Load file indexes. -6. Build the parse tree. -7. Walk builders to construct the semantic model. -8. Initialize namespaces. -9. Load plugin configuration. -10. For each plugin in dependency order, run validators, enhancers, and generators. -11. Write generated output. -12. Map validation failures to files. - -Plugins are ordered by `@edfi/metaed-default-plugins`, allowing upstream plugins to create semantic data used by downstream artifact generators. - -### 5.2 Component Packages - -| Package | Product Responsibility | -| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `@edfi/metaed-core` | Core pipeline, configuration model, file loading, grammar parsing, model builders, validators/enhancers/generators contracts, plugin configuration loading, and output writing. | -| `@edfi/metaed-console` | Build CLI for running the MetaEd pipeline with default plugins. | -| `@edfi/metaed-default-plugins` | Default ordered plugin list used by the build and deploy-console build flow. | -| `@edfi/metaed-odsapi-deploy` | Deploy task library for copying generated artifacts into an ODS/API source tree. | -| `@edfi/metaed-odsapi-deploy-console` | Deploy CLI for source-scan build-and-deploy mode or config-based deploy mode. | -| `@edfi/metaed-plugin-edfi-unified` | Foundational Ed-Fi semantic validators and enhancers, including reference resolution, base class resolution, domain/subdomain links, merge directives, and shared type enrichment. | -| `@edfi/metaed-plugin-edfi-unified-advanced` | Additional validation for merge scenarios, deprecation warnings, common property identity rules, and self-referencing property role names. | -| `@edfi/metaed-plugin-edfi-ods-relational` | Shared relational ODS model enhancers and relational validators used by SQL Server, PostgreSQL, API Schema, dictionaries, change query, and record ownership plugins. | -| `@edfi/metaed-plugin-edfi-api-schema` | API Schema model enhancers, security-related configuration rules, OpenAPI fragment and relational metadata, and `ApiSchema*.json` generation with API Schema version metadata. | -| `@edfi/metaed-plugin-edfi-ods-sqlserver` | SQL Server ODS naming, schema/table/column enhancements, and SQL Server ODS SQL script generation. | -| `@edfi/metaed-plugin-edfi-ods-postgresql` | PostgreSQL ODS naming, schema/table/column enhancements, and PostgreSQL ODS SQL script generation. | -| `@edfi/metaed-plugin-edfi-xsd` | XSD model enhancers, duplicate-name validation for dependency namespaces, core/extension XSD generation, schema annotation generation, and interchange XSD generation. | -| `@edfi/metaed-plugin-edfi-ods-changequery` | Common change query validator, common model types, and shared generation helpers for change query SQL artifacts. | -| `@edfi/metaed-plugin-edfi-ods-changequery-sqlserver` | SQL Server-specific change query enhancers and SQL script generators. | -| `@edfi/metaed-plugin-edfi-ods-changequery-postgresql` | PostgreSQL-specific change query enhancers and SQL script generators. | -| `@edfi/metaed-plugin-edfi-ods-recordownership` | Common record ownership table enhancement that marks aggregate root tables for ownership token columns for target technology versions `>=3.3.0`. | -| `@edfi/metaed-plugin-edfi-ods-recordownership-sqlserver` | SQL Server record ownership SQL generation. | -| `@edfi/metaed-plugin-edfi-ods-recordownership-postgresql` | PostgreSQL record ownership SQL generation. | -| `@edfi/metaed-plugin-edfi-odsapi` | ODS/API model validators/enhancers and `ApiModel*.json` generation. | -| `@edfi/metaed-plugin-edfi-xml-dictionary` | XML data dictionary Excel generation from XSD model data. | -| `@edfi/metaed-plugin-edfi-sql-dictionary` | SQL data dictionary Excel generation from relational ODS model data. | -| `@edfi/metaed-plugin-edfi-handbook` | Handbook-specific model enhancers plus HTML and Excel Data Handbook generation. | -| `@edfi/metaed-plugin-edfi-api-catalog` | API Catalog Excel generation from API Schema and OpenAPI fragment data. | - -### 5.3 Default Plugin Order - -The default plugin order is: - -`edfiUnified`, `edfiUnifiedAdvanced`, `edfiOdsRelational`, `edfiApiSchema`, `edfiOdsSqlServer`, `edfiOdsPostgresql`, `edfiXsd`, `edfiOdsChangeQuery`, `edfiOdsChangeQuerySqlServer`, `edfiOdsChangeQueryPostgresql`, `edfiOdsRecordOwnership`, `edfiOdsRecordOwnershipSqlServer`, `edfiOdsRecordOwnershipPostgresql`, `edfiOdsApi`, `edfiXmlDictionary`, `edfiSqlDictionary`, `edfiHandbook`, `edfiApiCatalog`. - -### 5.4 Output Model - -Generators return `GeneratedOutput` objects containing a human-readable name, namespace, folder name, file name, and either string or binary stream output. The writer creates directories recursively and writes files beneath the artifact directory using `{namespace}/{folderName}/{fileName}`. - -## 6. Out of Scope and Known Limitations - -### 6.1 Explicit Exclusions - -- The MetaEd DSL language specification and grammar semantics are outside this PRD. -- External IDE behavior is outside this repository and outside this PRD. -- ODS/API runtime behavior, API service hosting, database execution, migrations, and production deployment are outside this PRD. -- Data Standard content itself is outside this PRD; this repository processes model files but does not define every data element requirement here. - -### 6.2 Known Limitations - -- `MetaEdConfiguration` includes `pluginTechVersion`, and some fixtures contain it, but current plugin setup assigns every plugin the single `defaultPluginTechVersion`. -- The license acceptance CLI option is required by yargs, but the source does not validate the option value or persist acceptance. -- The build console's custom config loader resolves relative `-c` / `--config` paths relative to the console module directory; absolute config paths avoid that ambiguity. -- Config-based deploy mode runs deploy tasks against an existing `artifactDirectory`; source-scan mode is the path that builds before deploy. -- Config-based deploy mode copies the supplied `metaEdConfiguration` without merging defaults from `newMetaEdConfiguration`, and it does not apply the `--suppressPrereleaseVersion` CLI option to the supplied configuration. -- Source-scan deploy mode returns without deploying when required source-mode inputs are absent, including `source` or `projectNames`. -- After source-scan deploy mode attempts a build, the deploy console still invokes deploy tasks even when the build has set a non-zero exit code; if an existing artifact directory remains, deploy can attempt to copy those artifacts. -- Source-scan project discovery scans subdirectories only while no projects have been discovered yet, so mixed source inputs can miss nested projects after an earlier project has already been found. -- Source-scan project discovery de-duplicates projects by project name, so two discovered projects with the same project name cannot both be represented in one deploy build. -- Source-scan namespace derivation returns an empty namespace when `projectName` cannot be converted to an alphanumeric string that starts with an uppercase letter. -- Deploy copies extension artifacts from folders named by `projectName`, while generators write output folders using `namespaceName`; source-scan and config-based deployments where those values differ can cause deploy to skip generated extension folders. -- Source-scan project sorting only special-cases a project whose `projectName` is exactly `EdFi`; Data Standard packages with `projectName` values such as `Ed-Fi` are sorted alphabetically with other discovered projects after namespace derivation. -- Core and extension artifact copy tasks are no-ops for `defaultPluginTechVersion` values below `7.0.0`. -- A core-only deploy with no extension artifact folders can fail the extension project existence precheck when `allianceMode` is false, because that precheck runs before core copy and ignores root `EdFi`, `Documentation`, and `ApiSchema` folders. -- Deploy does not copy generated `ApiSchema/` or `Documentation/` artifact folders into the ODS/API source tree. -- Deploy assumes the local ODS/API source tree follows expected `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folder conventions. -- The console packages declare Node.js `main` entry points but no package `bin` commands, so command execution is through Node.js entry points or repository scripts rather than package-installed binary names. -- API Schema packaging `RunMetaEd` copies `packages/metaed-plugin-edfi-api-schema/test/integration/edfiApiSchema.config.json` into the core project path before running MetaEd, so packaging output depends on that checked-in configuration fixture. -- The API Schema packaging workflow matrix uses `TPDM`, but the TPDM checkout step currently checks for lowercase `tpdm`, so that workflow step does not run for the TPDM matrix entries as written. -- The API Catalog generator contains hard-coded handling for EducationOrganization and SchoolYear reference schemas because those reference schemas are not always discoverable from generated OpenAPI fragments. -- The output writer's deletion guard is path-name based: it requires `MetaEdOutput` to appear in the output path before deleting an existing output directory. - -## 7. Glossary - -| Term | Definition | -| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| MetaEd-js | The Node.js monorepo that parses MetaEd DSL projects and generates artifacts through plugins. | -| MetaEd DSL | The `.metaed` source language processed by this repository. | -| Data Standard project | The core Ed-Fi project identified by namespace `EdFi`; exactly one is required by the CLIs. | -| Extension project | A non-EdFi MetaEd project processed alongside the Data Standard project. | -| Namespace | The model and output grouping used by MetaEd to separate core and extension artifacts. | -| Project extension | A string used by generators in some extension artifact file names; non-EdFi projects default to `EXTENSION` when metadata omits it. | -| Default plugin technology version | The version string assigned to plugin environments to select version-specific generator behavior. | -| Plugin | A package that contributes validators, enhancers, generators, and optional configuration schemas. | -| Validator | A plugin function that reports syntax, semantic, relationship, or configuration failures. | -| Enhancer | A plugin function that adds derived semantic data to the model for downstream processing. | -| Generator | A plugin function that produces one or more generated outputs. | -| Artifact | A generated file written by the build pipeline, such as SQL, XSD, JSON, HTML, or Excel. | -| ODS | Operational Data Store, represented here by generated SQL artifacts and ODS/API source tree folders. | -| ODS/API | The downstream Ed-Fi source tree layout targeted by deploy tasks. | -| API Model | ODS/API metadata JSON generated as `ApiModel*.json`. | -| API Schema | JSON schema output generated as `ApiSchema*.json` for API schema consumers. | -| Change Query | SQL artifacts for change version, tracked delete, and related change tracking support. | -| Record Ownership | SQL artifacts that add ownership token support to aggregate root tables when enabled by target version. | diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index d0eaa2926..c2602c15f 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -1,15 +1,15 @@ # MetaEd Product Requirements Document — Version 4.x -- Status: Done - Owner: Stephen Fuqua - Repository: Ed-Fi-Alliance-OSS/MetaEd-Js - Current Version: 4.7 -- Platform: Visual Studio Code Extension + Node.js CLI +- Package version source: Lerna version `4.7.1-dev.1` +- Platform and runtime: Node.js packages compiled from TypeScript to ES2017/CommonJS; CI runs Node 22 > [!TIP] Document Scope -> This PRD defines the product requirements for MetaEd version 4.x primarily from the end-user's perspective, which includes the new JavaScript/TypeScript implementation of the MetaEd generator and the accompanying VS Code extension. It covers the core features, functional and non-functional requirements, system architecture, and known limitations of this major release. The document is intended to guide development and ensure alignment with user needs and strategic goals. +> This PRD defines the product requirements for MetaEd version 4.x, covering the Node.js monorepo that implements the MetaEd generator and its command-line interfaces. It covers the core features, functional and non-functional requirements, system architecture, and known limitations. > -> It does _not cover_ details about the MetaEd DSL syntax and semantics, the internal compilation pipeline, or details on how to construct the API model files used by ODS/API code generation and by the Ed-Fi API v8 deployment and runtime processes. +> It does _not cover_ the MetaEd DSL syntax and semantics, the internal compilation pipeline, the MetaEd IDE Visual Studio Code extension (maintained in a separate repository), the Ed-Fi ODS/API runtime behavior, database execution semantics, or the Data Standard content itself. ## 1. Product Overview @@ -17,11 +17,21 @@ MetaEd is a technology framework that uses an Ed-Fi-aligned domain-specific lang - **MetaEd Language** — a lightweight DSL describing data elements, their properties, and their organization into entities and domains (the Ed-Fi Unified Data Model). Files use the `.metaed` extension. - **MetaEd Generator** — a command-line application that compiles MetaEd language files into technical artifacts. -- **MetaEd IDE** — a Visual Studio Code extension providing an integrated development environment for authoring `.metaed` files and invoking the generator. - **MetaEd Packages** — a collection of Node.js packages that implement the parsing, validation, enhancement, and generation pipeline. +MetaEd-js is a Node.js monorepo that compiles Ed-Fi MetaEd DSL projects into local filesystem artifacts used by downstream Ed-Fi data platform tooling. A MetaEd project is represented by `.metaed` source files plus project metadata such as project name, namespace, version, optional project extension, and description. The generator reads one Data Standard project and zero or more extension projects, validates and enriches the semantic model, runs the default plugin set, and writes generated outputs grouped by namespace. + +The repository provides two primary command-line workflows: + +- `@edfi/metaed-console` — runs the MetaEd build pipeline and writes generated artifacts. +- `@edfi/metaed-odsapi-deploy-console` — optionally builds from project source directories and then copies generated artifacts into a local Ed-Fi ODS/API source tree layout for supported technology versions. + +Repository automation also includes a manually dispatchable API Schema packaging workflow under `eng/ApiSchema` that runs MetaEd, moves selected generated API Schema and XSD artifacts into .NET package projects, packs NuGet packages, and publishes those packages through GitHub Actions when feed credentials are supplied. + The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSON API metadata, SQL scripts, and technical documentation from a single source definition written in the MetaEd DSL. Externally, Ed-Fi Community members use MetaEd to create and maintain Ed-Fi Extensions, which are additive data models that extend the core Ed-Fi Data Standard to meet local needs. +The product value is artifact consistency: SQL, XSD, API metadata, API schema, documentation, and dictionary outputs are generated from the same MetaEd model rather than authored independently. + ### 1.1 Strategic Alignment - **Single source of truth for data models.** All Ed-Fi technical artifacts (SQL, XSD, API metadata, documentation) are derived from one set of `.metaed` files, eliminating drift between artifacts. @@ -31,28 +41,28 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO ### 1.2 Target Users and Personas -- **Ed-Fi Alliance Technical Team** — maintains the core Ed-Fi Data Standard using MetaEd in Alliance Mode. Requires full control over core model files and uses advanced features like plugin configuration and CI builds. -- **Education Agency Developers** — extend the Ed-Fi Data Standard for local requirements. Primarily uses the IDE for authoring extensions and deploying artifacts to their ODS/API environment. -- **Ed-Fi Integration Partners** — build and maintain extensions on behalf of education agencies. Uses both the IDE and CI workflows to produce and validate extension artifacts. -- **Business Analysts** — review generated documentation (Data Handbook, API Catalog, Data Dictionaries) to understand the data model without reading source files. +- **Ed-Fi Alliance Technical Team / Data Standard maintainers** — maintain the core Ed-Fi Data Standard MetaEd project and need repeatable generated artifacts for release and packaging workflows. Require full control over core model files and use advanced features like plugin configuration, Alliance Mode, and CI builds. +- **Education Agency Developers and Ed-Fi Integration Partners / Extension developers** — maintain additive MetaEd extension projects and need validation plus generated artifacts for ODS/API integration. Build and maintain extensions on behalf of education agencies using both CLI workflows and CI pipelines. +- **Build and release engineers** — run MetaEd headlessly in CI, publish npm packages, package API Schema artifacts, and verify generated outputs against authoritative fixtures. +- **Business Analysts / Downstream artifact consumers** — use generated spreadsheets, HTML documentation, JSON schema and metadata files, SQL scripts, and XSD files without reading `.metaed` source. Review generated documentation to understand the data model. ### 1.3 Jobs to Be Done -#### JTBD 1.1. Author data model extensions in a guided IDE +#### JTBD 1.1. Compile model changes into deployable artifacts **Personas**: Education Agency Developers, Ed-Fi Integration Partners, Ed-Fi Alliance Technical Team -**When** I need to create or modify an Ed-Fi extension, **I want** to author `.metaed` files in a guided IDE with syntax highlighting and validation, **so that** I can quickly define my extension without learning the internals of XSD, SQL, or API metadata formats. +**When** I am ready to test my model changes, **I want** to run a Build command that compiles all artifacts, **so that** I can verify correctness before deploying. -**How MetaEd Helps**: The VS Code extension provides syntax highlighting, real-time semantic validation, and inline error reporting so authors get immediate feedback on model correctness. +**How MetaEd Helps**: The Build command transforms `.metaed` source files into SQL scripts, XSD schemas, API metadata, interchange definitions, and documentation in a single operation. -#### JTBD 1.2. Compile model changes into deployable artifacts +#### JTBD 1.2. Validate a model **Personas**: Education Agency Developers, Ed-Fi Integration Partners, Ed-Fi Alliance Technical Team -**When** I am ready to test my model changes, **I want** to run a Build command that compiles all artifacts, **so that** I can verify correctness before deploying. +**When** I change `.metaed` files, **I need** the pipeline to report syntax, semantic, configuration, and plugin validation failures in a machine-detectable build result, **so that** errors can be identified and corrected before deployment. -**How MetaEd Helps**: The Build command transforms `.metaed` source files into SQL scripts, XSD schemas, API metadata, interchange definitions, and documentation in a single operation. +**How MetaEd Helps**: The pipeline collects and reports validation failures at every stage, stops on errors when configured to do so, and exits with a non-zero code for CI detection. #### JTBD 1.3. Deploy extension artifacts to the ODS/API @@ -64,7 +74,7 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO #### JTBD 1.4. Validate and produce artifacts in CI -**Personas**: Ed-Fi Alliance Technical Team, Ed-Fi Integration Partners +**Personas**: Ed-Fi Alliance Technical Team, Ed-Fi Integration Partners, Build and release engineers **When** I maintain a CI pipeline for data model changes, **I want** to run MetaEd headlessly via Node.js, **so that** I can validate and produce artifacts on every commit without a graphical environment. @@ -74,66 +84,152 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO **Personas**: Business Analysts, Education Agency Developers, Ed-Fi Integration Partners -**When** I need to understand the API shape of a data model, **I want** a generated API Catalog spreadsheet, **so that** I can catalog resources and their properties for comparison and documentation purposes. +**When** I need to understand the API shape of a data model, **I want** generated documentation including a handbook, API catalog, and data dictionaries, **so that** I can understand the data model without reading source files. -**How MetaEd Helps**: The API Catalog plugin generates an Excel workbook with Resources and Properties tabs that describe every endpoint and field derived from the model. +**How MetaEd Helps**: The pipeline generates an HTML Data Handbook, API Catalog Excel workbook, SQL Data Dictionary, and XML Data Dictionary from the same model. ## 2. Enterprise and System Context ### 2.1 External Systems and Integrations -- **Ed-Fi ODS/API** — the primary consumer of MetaEd-generated artifacts (SQL scripts, XSD, API metadata). MetaEd's Deploy command copies artifacts into the ODS/API source tree for its build process. -- **Ed-Fi API** - successor to the ODS/API, which also consumes MetaEd-generated artifacts, particularly the API metadata. -- **Ed-Fi Data Standard (Model packages)** — versioned npm packages (e.g., `@edfi/ed-fi-model-4.0`) containing the core `.metaed` files. Extensions reference these as dependencies. -- **Visual Studio Code** — host environment for the MetaEd IDE extension. -- **Node.js** — runtime environment for the MetaEd generator and all packages. +- **Local MetaEd project directories** provide `.metaed` files and optional `package.json` files with `metaEdProject` metadata. +- **Ed-Fi Data Standard npm packages** are package dependencies of `@edfi/metaed-console` for model packages `@edfi/ed-fi-model-4.0`, `@edfi/ed-fi-model-5.0`, `@edfi/ed-fi-model-5.1`, `@edfi/ed-fi-model-5.2`, `@edfi/ed-fi-model-6.0`, and `@edfi/ed-fi-model-6.1`. +- **Ed-Fi ODS/API source trees** consume deployed artifact files under `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` directories. +- **SQL Server and PostgreSQL** are downstream targets for generated database scripts; MetaEd-js generates scripts but does not execute them. +- **API Schema and API metadata consumers** consume `ApiSchema*.json` and `ApiModel*.json` outputs. +- **.NET SDK and NuGet/Azure Artifacts** are used by API Schema packaging automation under `eng/ApiSchema`. +- **GitHub Actions and Azure Artifacts** are used by repository workflows for linting, testing, publishing npm packages, and packaging API Schema artifacts. - **Ed-Fi npm Registry** — Azure DevOps-hosted npm feed for distributing MetaEd packages. -- **ANTLR 4.6** — parser generator framework used for the MetaEd grammar. +- **ANTLR 4.6 generated JavaScript grammar files** are part of the source tree used by the parser. ### 2.2 Deployment Boundaries -- MetaEd runs entirely on the developer's local machine (or CI agent). It does not communicate with external services at runtime. -- Generated artifacts are files on disk. MetaEd does not directly deploy to running servers or databases. -- The Deploy command writes to a local clone of the ODS/API source repositories. +- MetaEd-js build and deploy commands operate on local files. The build pipeline reads `.metaed` source files and plugin configuration files, then writes artifacts to disk. +- Deploy copies files into a local ODS/API source tree layout. It does not deploy to running services, execute SQL, start databases, or validate ODS/API runtime behavior. +- Publishing workflows communicate with package feeds, but normal build and deploy runtime behavior is entirely filesystem-based. +- MetaEd does not communicate with external services during normal operation and does not require credentials at runtime. ## 3. Functional Requirements -### 3.1 VS Code Extension (MetaEd IDE) - -- **FR-IDE-1**: The extension SHALL provide syntax highlighting for `.metaed` files. -- **FR-IDE-2**: The extension SHALL provide real-time semantic validation, reporting errors and warnings in the VS Code Problems panel. -- **FR-IDE-3**: Errors SHALL be displayed with red indicators on files/folders in the Explorer tree; warnings SHALL be displayed with gold indicators. -- **FR-IDE-4**: The extension SHALL provide a **Build** command (accessible from the editor toolbar) that compiles all `.metaed` files in the workspace into output artifacts. -- **FR-IDE-5**: The extension SHALL provide a **Deploy** command that copies build artifacts to the configured ODS/API source directory. -- **FR-IDE-6**: The extension SHALL expose workspace-level settings for: - - Accepted License (boolean, required) - - Target ODS/API Version (string) - - ODS/API Deployment Directory (path) - - Suppress Delete on Deploy (boolean) - - Alliance Mode (boolean) -- **FR-IDE-7**: The extension SHALL require acceptance of the Ed-Fi License Agreement before Build or Deploy can execute. -- **FR-IDE-8**: The extension SHALL support multi-folder workspaces, allowing users to open both the core data model and one or more extension projects simultaneously. -- **FR-IDE-9**: Build output SHALL be written to a `MetaEdOutput` directory in the last workspace folder. -- **FR-IDE-10**: The extension SHALL display a notification on build success or failure, with error details available in the output panel. +### 3.1 Project Input and Configuration + +- **FR-CFG-1**: The build CLI SHALL accept a JSON configuration file through `-c` / `--config` containing a top-level `metaEdConfiguration` object. +- **FR-CFG-2**: `metaEdConfiguration` SHALL support `artifactDirectory`, `deployDirectory`, `pluginTechVersion`, `projects`, `projectPaths`, `pluginConfigDirectories`, `defaultPluginTechVersion`, `allianceMode`, and `suppressPrereleaseVersion`; it MAY include `externalVariables` for Jsonnet configuration evaluation. +- **FR-CFG-3**: The pipeline SHALL require `projects` and `projectPaths` to have the same length and SHALL map each project metadata entry to the corresponding source path. +- **FR-CFG-4**: The file loader SHALL recursively load files with `.metaed`, `.metaEd`, `.MetaEd`, or `.METAED` extensions from each configured project path. +- **FR-CFG-5**: The build and deploy CLIs SHALL require exactly one Data Standard project, identified by namespace `EdFi`; no Data Standard project or multiple Data Standard projects SHALL cause failure. +- **FR-CFG-6**: The project scanner used by the deploy console source mode SHALL read `package.json` files containing `metaEdProject`, derive `namespaceName` by removing non-alphanumeric characters from `projectName` only when the result starts with an uppercase letter, include the package description when present, sort a project whose `projectName` is exactly `EdFi` ahead of other projects, otherwise sort discovered projects by project name, and de-duplicate discovered projects by project name. +- **FR-CFG-7**: The project scanner MAY accept `projectNames` overrides that update discovered project names and derived namespaces in discovery order. +- **FR-CFG-8**: Plugin configuration files SHALL be discovered from configured `pluginConfigDirectories`, or from input project directories when no plugin config directories are configured, as `{pluginShortName}.config.jsonnet` or `{pluginShortName}.config.json`, with Jsonnet preferred when both exist. +- **FR-CFG-9**: Plugin configuration loading SHALL support `externalVariables` for Jsonnet evaluation. +- **FR-CFG-10**: Plugin configuration rules SHALL support plugin-wide data or entity-matched data using `entity`, `namespace`, `core`, `extensions`, and `entityName` matching fields. +- **FR-CFG-11**: Configuration matching SHALL reject a single match rule that combines `namespace` with `core` or `extensions`. +- **FR-CFG-12**: The API Schema plugin SHALL validate plugin-specific configuration rules for `educationOrganizationSecurableElements` and `educationOrganizationIdentitySecurableElements`. +- **FR-CFG-13**: When project metadata does not provide `projectExtension`, the file loader SHALL use an empty project extension for namespace `EdFi` and `EXTENSION` for non-EdFi namespaces. +- **FR-CFG-14**: File loading SHALL fail the build when no `.metaed` files are found in any configured input directory. ### 3.2 Build Command -- **FR-BUILD-1**: The Build command SHALL compile all `.metaed` files from configured project paths into output artifacts. -- **FR-BUILD-2**: The Build command SHALL accept configuration via a JSON config file (`-c` / `--config` flag) or VS Code workspace settings. -- **FR-BUILD-3**: The Build command SHALL require the `-a` / `--accept-license` flag (or equivalent workspace setting) to acknowledge the Ed-Fi License Agreement. -- **FR-BUILD-4**: The Build command SHALL support the `--defaultPluginTechVersion` flag to specify the target ODS/API version in semver format. -- **FR-BUILD-5**: The Build command SHALL support the `--suppressPrereleaseVersion` flag (default: true) to suppress prerelease identifiers in version output. -- **FR-BUILD-6**: The Build command SHALL exit with code `1` on failure, enabling CI pipelines to detect build errors. -- **FR-BUILD-7**: The Build command SHALL produce output in a configured output directory; details about the output files depend on the plugins used at build time. - -### 3.3 Deploy Command - -- **FR-DEPLOY-1**: The Deploy command SHALL copy built artifacts from the artifact directory into the correct locations within the ODS/API source tree. -- **FR-DEPLOY-2**: The Deploy command SHALL accept configuration via a JSON config file or command-line switches (`-s` source, `-t` target). -- **FR-DEPLOY-3**: The Deploy command SHALL require the `-a` / `--accept-license` flag. -- **FR-DEPLOY-4**: The Deploy command SHALL support the `--core` flag to deploy core artifacts in addition to extensions. -- **FR-DEPLOY-5**: The Deploy command SHALL support the `--suppressDelete` flag to prevent deletion of existing extension artifact directories before deploying. -- **FR-DEPLOY-6**: When the `--core` flag is specified (ODS/API version ≥ 7.0), the Deploy command SHALL copy core artifacts into the configured `deployDirectory` under `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{dataStandardVersion}/Artifacts/`: +- **FR-BUILD-1**: `@edfi/metaed-console` SHALL run from Node.js and execute the MetaEd pipeline using the default plugin set. +- **FR-BUILD-2**: The build CLI SHALL require `-a` / `--accept-license` before executing. +- **FR-BUILD-3**: The build CLI SHALL accept `-c` / `--config` to specify a JSON configuration file. +- **FR-BUILD-4**: The build CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for all plugins. +- **FR-BUILD-5**: The default plugin technology version SHALL be `6.0.0` when not overridden by configuration or CLI argument. +- **FR-BUILD-6**: The build CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when no configuration object supplies a value. +- **FR-BUILD-7**: The build pipeline SHALL run initialization, plugin setup, file loading, syntax validation, file indexing, parse tree building, model building, namespace initialization, plugin configuration loading, validators, enhancers, generators, output writing, and validation file mapping. +- **FR-BUILD-8**: The build pipeline SHALL stop plugin execution on validation errors when `stopOnValidationFailure` is enabled. +- **FR-BUILD-9**: The build CLI SHALL exit with code `0` only when no validation error and no pipeline failure occurred; otherwise it SHALL exit with code `1`. +- **FR-BUILD-10**: Generated outputs SHALL be written under the configured artifact directory or a `MetaEdOutput` directory under the last input project path. Relative `artifactDirectory` values SHALL be resolved from the last input project path. +- **FR-BUILD-11**: Output writing SHALL group generated files by namespace, with the generator-provided folder path under that namespace. +- **FR-BUILD-12**: Output writing SHALL refuse to delete an existing output directory unless the path contains `MetaEdOutput`. +- **FR-BUILD-13**: Output writing SHALL refuse to write when `.metaed` files are found in the output location. + +### 3.3 Generated Artifact Families + +#### 3.3.1 ODS/API Metadata + +- **FR-ODSAPI-METADATA-1**: The build SHALL generate ODS/API metadata JSON files per namespace as `ApiMetadata/ApiModel.json` when `projectExtension` is empty or `ApiMetadata/ApiModel-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-odsapi`. + +#### 3.3.2 API Schema + +- **FR-API-SCHEMA-1**: The build SHALL generate API Schema JSON files per namespace as `ApiSchema/ApiSchema.json` when `projectExtension` is empty or `ApiSchema/ApiSchema-{projectExtension}.json` when it is not, through `metaed-plugin-edfi-api-schema`. +- **FR-API-SCHEMA-2**: API Schema output SHALL include `apiSchemaVersion` and a project schema containing project identity, endpoint name, `compatibleDsRange` for extension-project compatibility, description, resource schemas, resource-name mappings, case-insensitive endpoint-name mappings, abstract resources, education organization information, domains, extension-project flag, and core OpenAPI base documents where generated. Resource schemas SHALL include generated JSON schema-for-insert data, identity JSON paths, document path mappings, equality constraints, type-coercion JSON paths, decimal validation information, securable elements, authorization pathways, array uniqueness constraints, OpenAPI fragments, query-field mappings where applicable, school year enumeration resource schema when present, and optional relational naming metadata where generated. + +#### 3.3.3 SQL Server and PostgreSQL ODS SQL + +- **FR-ODS-SQL-1**: The build SHALL generate SQL Server ODS structure and data scripts under `Database/SQLServer/ODS/`. +- **FR-ODS-SQL-2**: The build SHALL generate PostgreSQL ODS structure and data scripts under `Database/PostgreSQL/ODS/`. +- **FR-ODS-SQL-3**: SQL ODS outputs SHALL include schema, table, foreign key, extended property, enumeration, and school year scripts when corresponding model data exists; core file names SHALL use `{prefix}-{suffix}.sql`, and extension file names SHALL include namespace context as `{prefix}-{projectExtension}-{namespaceName}-{suffix}.sql` or `{prefix}-{namespaceName}-{suffix}.sql` when `projectExtension` is empty. +- **FR-ODS-SQL-4**: SQL ODS outputs SHALL include ID column unique index scripts for tables that require generated ID indexes. +- **FR-ODS-SQL-5**: For technology versions `>=7.1.0`, SQL ODS outputs SHALL include education organization authorization index creation scripts when table data exists. +- **FR-ODS-SQL-6**: For technology versions `>=7.3.0`, SQL ODS outputs SHALL include aggregate ID column scripts and education organization authorization index update scripts when table data exists. +- **FR-ODS-SQL-7**: SQL-generating plugins SHALL include Apache-2.0 license headers in supported SQL templates only when `allianceMode` is enabled and the relevant target technology version satisfies `>=5.0.0`. + +#### 3.3.4 Change Query SQL + +- **FR-CHANGE-QUERY-SQL-1**: The change query plugins SHALL generate change schema, change version sequence, change version column/index, tracked delete schema, tracked delete table, delete tracking trigger, change version/key change trigger, and indirect update cascade trigger scripts under `Database/{SQLServer|PostgreSQL}/ODS/Structure/Changes/` when required model data exists. For target technology versions `>=7.3.0`, shared change schema and sequence files SHALL be generated for the core namespace and skipped for extension namespaces, and indirect update cascade triggers SHALL be generated when required model data exists. + +#### 3.3.5 Record Ownership SQL + +- **FR-RECORD-OWNERSHIP-SQL-1**: The record ownership plugins SHALL generate `0010-AddColumnOwnershipTokenForTable.sql` under `Database/{SQLServer|PostgreSQL}/ODS/Structure/RecordOwnership/` for aggregate root tables when the target technology version satisfies `>=3.3.0`. + +#### 3.3.6 XSD and Interchange Schemas + +- **FR-XSD-1**: The XSD plugin SHALL generate `XSD/Ed-Fi-Core.xsd` for the core namespace and `{projectExtension}-Ed-Fi-Extended-Core.xsd` for extension namespaces. +- **FR-XSD-2**: The XSD plugin SHALL generate `XSD/SchemaAnnotation.xsd` for the core namespace. +- **FR-XSD-3**: The XSD plugin SHALL generate interchange schemas under `Interchange/`, using `Interchange-{Name}.xsd` for core interchanges and `{projectExtension}-Interchange-{Name}-Extension.xsd` for extension interchanges. + +#### 3.3.7 Data Handbook + +- **FR-HANDBOOK-1**: The handbook plugin SHALL generate `Documentation/Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` and `Documentation/Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx`. + +#### 3.3.8 Data Dictionaries + +- **FR-DICTIONARY-1**: The SQL dictionary plugin SHALL generate `Documentation/DataDictionary/SqlDataDictionary.xlsx` with table and column worksheets derived from relational ODS model data. +- **FR-DICTIONARY-2**: The XML dictionary plugin SHALL generate `Documentation/DataDictionary/XmlDataDictionary.xlsx` with elements, complex types, and simple types worksheets derived from XSD model data. + +#### 3.3.9 API Catalog + +- **FR-API-CATALOG-1**: The API Catalog plugin SHALL generate `Documentation/Ed-Fi-API-Catalog/Ed-Fi-API-Catalog.xlsx`. +- **FR-API-CATALOG-2**: The API Catalog SHALL include `Resources` and `Properties` worksheets. +- **FR-API-CATALOG-3**: The API Catalog `Resources` worksheet SHALL include project, version, resource name, resource description, and domains. +- **FR-API-CATALOG-4**: The API Catalog `Properties` worksheet SHALL include project, version, resource name, property name, property description, data type, min length, max length, validation regular expression, identity key flag, nullable flag, and required flag. +- **FR-API-CATALOG-5**: The API Catalog SHALL omit the top-level `id` property from the properties worksheet. +- **FR-API-CATALOG-6**: The API Catalog SHALL recurse into common sub-schemas and reference schemas and represent nested property origins with dot-separated property paths. + +### 3.4 Model Validation + +- **FR-VAL-1**: The pipeline SHALL collect syntax validation failures per loaded file before building the aggregate parse tree and SHALL include them in the final validation result; when `stopOnValidationFailure` is enabled, existing error-category failures SHALL prevent plugin enhancers, plugin generators, and output writing. +- **FR-VAL-2**: The unified plugin SHALL validate core model relationships, including unresolved references, duplicate names, illegal extensions and subclasses, redeclared properties, invalid identity declarations and renames, invalid domain/subdomain/interchange membership, invalid namespace casing, and invalid shared/simple property bounds. +- **FR-VAL-3**: The unified advanced plugin SHALL validate merge directive paths and property compatibility, out-reference paths that need a merge directive or role name, common-property identity requirements, identity-name conflicts, and self-referencing property role names; it SHALL also report deprecation warnings for deprecated entities, extensions, subclasses, properties, domain item references, and interchange item references. +- **FR-VAL-4**: The ODS relational plugin SHALL warn for properties named `Discriminator` when the target technology version is `>=5.1.0`. +- **FR-VAL-5**: The change query plugin SHALL reject namespaces named `Changes`, case-insensitively, because that name is reserved by generated change query artifacts. +- **FR-VAL-6**: The ODS/API plugin SHALL warn when required choice properties appear in extensions and SHALL reject unsupported extension subclasses except EducationOrganization domain entity subclasses and GeneralStudentProgramAssociation association subclasses. +- **FR-VAL-7**: The XSD plugin SHALL suppress XSD generation for a namespace with duplicate entity names in dependency namespaces and SHALL report validation through the duplicate-name validator. +- **FR-VAL-8**: Deprecation validators SHALL report warnings for extension namespace elements, and SHALL report warnings for non-extension namespace elements only when `allianceMode` is enabled. + +### 3.5 Deploy Command + +- **FR-DEPLOY-1**: `@edfi/metaed-odsapi-deploy-console` SHALL run from Node.js and execute deploy tasks from `@edfi/metaed-odsapi-deploy`. +- **FR-DEPLOY-2**: The deploy CLI SHALL require `-a` / `--accept-license`. +- **FR-DEPLOY-3**: The deploy CLI SHALL accept configuration through `-c` / `--config`. +- **FR-DEPLOY-4**: When no configuration is supplied, the deploy CLI SHALL use `-s` / `--source`, `-t` / `--target`, and `-p` / `--projectNames` to scan project source directories, build artifacts into `MetaEdOutput` under the last resolved project path, and then deploy. +- **FR-DEPLOY-5**: The deploy CLI SHALL accept `-x` / `--defaultPluginTechVersion` to override the configured default technology version for build and deploy behavior. +- **FR-DEPLOY-6**: In source-scan mode, the deploy CLI SHALL support `--suppressPrereleaseVersion`, defaulting to `true` when the option is not supplied; in config-based mode, deploy SHALL use the `suppressPrereleaseVersion` value supplied by `metaEdConfiguration`. +- **FR-DEPLOY-7**: The deploy CLI SHALL accept `--core` to deploy core artifacts in addition to extension artifacts. +- **FR-DEPLOY-8**: The deploy CLI SHALL accept `--suppressDelete` to skip deletion of existing extension artifact directories. +- **FR-DEPLOY-9**: The deploy CLI SHALL accept `--additionalMssqlScriptsDirectory` and `--additionalPostgresScriptsDirectory`; when a core or extension artifact copy task runs for a supported `defaultPluginTechVersion`, deploy SHALL copy those scripts into the corresponding deployed ODS data folders. +- **FR-DEPLOY-10**: For `defaultPluginTechVersion` values `>=3.0.0`, deploy SHALL check that extension C# project directories exist under `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/` for each immediate artifact directory except `ApiSchema`, `Documentation`, and `EdFi`. +- **FR-DEPLOY-11**: Unless suppressed, deploy SHALL remove existing `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Artifacts` directories for `defaultPluginTechVersion` values `>=3.3.0`. +- **FR-DEPLOY-12**: Core artifact copy SHALL run only when `--core` is supplied and `defaultPluginTechVersion` satisfies `>=7.0.0`. +- **FR-DEPLOY-13**: Extension artifact copy SHALL run only when `defaultPluginTechVersion` satisfies `>=7.0.0`. +- **FR-DEPLOY-14**: Core deploy SHALL copy from artifact namespace `EdFi` into `Ed-Fi-ODS/Application/EdFi.Ods.Standard/Standard/{dataStandardVersion}/Artifacts/`. +- **FR-DEPLOY-15**: Extension deploy SHALL copy each non-EdFi project from an artifact source folder named by `projectName` into `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Versions/{projectVersion}/Standard/{dataStandardVersion}/Artifacts/`. +- **FR-DEPLOY-16**: For ODS/API technology versions satisfying `>=7.1.0`, deploy SHALL format Data Standard versions with prerelease suppression according to `suppressPrereleaseVersion`. +- **FR-DEPLOY-17**: Deploy SHALL refresh existing extension `.csproj` files by updating their filesystem modification timestamp after copy tasks run. +- **FR-DEPLOY-18**: For `defaultPluginTechVersion` values `>=3.3.0`, deploy SHALL warn when legacy `SupportingArtifacts` directories exist for the standard or extension projects. +- **FR-DEPLOY-19**: Deploy SHALL skip a generated source folder when that folder is absent rather than failing the entire deploy. +- **FR-DEPLOY-20**: Deploy SHALL map generated folders to ODS/API artifact folders as follows: - `ApiMetadata/` → `Metadata/` - `Database/SQLServer/ODS/Data/` → `MsSql/Data/Ods/` - `Database/SQLServer/ODS/Structure/` → `MsSql/Structure/Ods/` @@ -141,207 +237,179 @@ The Ed-Fi Alliance uses MetaEd internally to produce core components such as JSO - `Database/PostgreSQL/ODS/Structure/` → `PgSql/Structure/Ods/` - `Interchange/` → `Schemas/` - `XSD/` → `Schemas/` -- **FR-DEPLOY-7**: The Deploy command SHALL copy extension artifacts into the configured `deployDirectory` under `Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Versions/{projectVersion}/Standard/{dataStandardVersion}/Artifacts/` using the same mapping as FR-DEPLOY-6. -- **FR-DEPLOY-8**: The `deployDirectory` SHALL point to the parent directory containing `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folders. -- **FR-DEPLOY-9**: Unless `--suppressDelete` is specified, the Deploy command SHALL remove existing extension artifact directories (`Ed-Fi-ODS-Implementation/Application/EdFi.Ods.Extensions.{projectName}/Artifacts`) before deploying new artifacts. - -### 3.4 Configuration - -- **FR-CFG-1**: The system SHALL support a JSON configuration file with the following top-level structure: - - `metaEdConfiguration.artifactDirectory` — output directory for build artifacts - - `metaEdConfiguration.deployDirectory` — target for deploy - - `metaEdConfiguration.projects[]` — array of project definitions (projectName, namespaceName, projectExtension, projectVersion, description) - - `metaEdConfiguration.projectPaths[]` — array of filesystem paths to `.metaed` source directories - - `metaEdConfiguration.pluginConfigDirectories[]` — array of paths to plugin configuration files - - `metaEdConfiguration.defaultPluginTechVersion` — target ODS/API version - - `metaEdConfiguration.allianceMode` — enables editing of core model files - - `metaEdConfiguration.suppressPrereleaseVersion` — suppresses prerelease version identifiers -- **FR-CFG-2**: The system MAY support artifact-specific configuration files (`{pluginShortName}.config.json`) placed in plugin configuration directories. -- **FR-CFG-3**: Artifact-specific configuration files SHALL support matching rules based on entity type, namespace, core/extensions flag, and entity name. -- **FR-CFG-4**: Artifact-specific configuration data SHALL be validated using Joi schemas registered by each plugin. -- **FR-CFG-5**: Configuration matching options (`namespace`, `core`, `extensions`) SHALL be mutually exclusive within a single match rule. - -### 3.5 Build Outputs - -#### 3.5.1 SQL Scripts - -- **FR-SQL-1**: The Build SHALL generate SQL scripts compatible with both Microsoft SQL Server and PostgreSQL for creating an operational data store (ODS). -- **FR-SQL-2**: Generated SQL files SHALL include (but are not limited to): - - `xxxx-Schemas.sql` — core database schema definitions - - `xxxx-Tables.sql` — core database tables - - `xxxx-IdColumnUniqueIndexes.sql` — indexing for API support on core tables - - `xxxx-CreateTrackedChangeTables.sql` — core tracked delete tables for Change Queries - - `xxxx-CreateTriggersForDeleteTracking.sql` (MSSQL only) - triggers for tracked delete support -- **FR-SQL-3**: The Build SHALL generate separate SQL files for core and extension artifacts, with extension files prefixed by "EXTENSION" (e.g., `EXTENSION-xxxx-Tables.sql`). -- **FR-SQL-4**: Files prefixed with "EXTENSION" SHALL be additive to the standard ODS/API script set; files without "EXTENSION" SHALL replace the corresponding shipped files. - -#### 3.5.2 XSD Files - -- **FR-XSD-1**: The Build SHALL generate XSD files for defining the structure of bulk data exchange and bulk loading of Ed-Fi data. -- **FR-XSD-2**: Generated XSD files SHALL include: - - `Ed-Fi-Core.xsd` — the core Ed-Fi XSD - - `EXTENSION-Ed-Fi-Extended-Core.xsd` — the extension XSD - - `SchemaAnnotation.xsd` — annotation definitions - - `Interchange-*.xsd` — core interchange schemas - - `EXTENSION-Interchange-*.xsd` — extension interchange schemas - -#### 3.5.3 API Metadata - -- **FR-API-1**: The Build SHALL generate API metadata files used by the ODS/API build process, including: `ApiModel{prefix}.json` -- **FR-API-2**: The Build SHALL generate API metadata files used by the Ed-Fi API v8+ build and runtime processes, including: `ApiSchema{prefix}.json` - -#### 3.5.4 Data Handbook - -- **FR-HBK-1**: The Build SHALL generate a comprehensive data handbook combining core and extension definitions. -- **FR-HBK-2**: The handbook SHALL be produced in an HTML format: `Ed-Fi-Data-Handbook-Index.html` — interactive, searchable HTML file (self-contained, can be hosted on a web server or run locally) -- **FR-HBK-3**: The handbook MAY be produced in an Excel format: `Ed-Fi-Handbook.xlsx` - -#### 3.5.5 Data Dictionaries - -- **FR-DICT-1**: The Build MAY generate data dictionaries documenting generated structures: - - `SqlDataDictionary.xlsx` — documentation on core and extension database tables - - `XmlDataDictionary.xlsx` — documentation on core and extension XSD elements - -#### 3.5.6 API Catalog - -- **FR-CAT-1**: The Build SHALL generate an API Catalog spreadsheet (`Ed-Fi-API-Catalog.xlsx`) describing the data model as it will be represented in the Open API specification. -- **FR-CAT-2**: The API Catalog SHALL contain a **Resources** worksheet with columns: Project, Version, Resource Name, Resource Description, Domains. -- **FR-CAT-3**: The API Catalog SHALL contain a **Properties** worksheet with columns: Project, Version, Resource Name, Property Name, Property Description, Data Type, Min Length, Max Length, Validation RegEx, Is Identity Key, Is Nullable, Is Required. -- **FR-CAT-4**: Properties belonging to embedded Common types or References SHALL be prefixed with a dot-separated path indicating their origin (e.g., `address.streetNumberName`). -- **FR-CAT-5**: The `id` property SHALL be omitted from the Properties worksheet. -- **FR-CAT-6**: The API Catalog output SHALL appear in an `Ed-Fi-API-Catalog` folder within the artifact directory. - -### 3.6 Extension Project Structure - -- **FR-EXT-1**: An extension project SHALL contain a `package.json` with a `metaEdProject` field specifying `projectName` and `projectVersion`. -- **FR-EXT-2**: The `projectName` SHALL start with an uppercase alphabetic character and contain at least two alphabetic characters. -- **FR-EXT-3**: The `projectVersion` SHALL follow SemVer 2.0.0 format. -- **FR-EXT-4**: Extension projects SHALL use the standard MetaEd directory structure: - - `Association/`, `Choice/`, `Common/`, `Descriptor/`, `Domain/`, `DomainEntity/`, `Enumeration/`, `Interchange/`, `Shared/` - -### 3.7 Continuous Integration Support - -- **FR-CI-1**: MetaEd SHALL be operable directly through Node.js without requiring VS Code or a graphical environment. -- **FR-CI-2**: The `@edfi/metaed-console` package SHALL be installable from the Ed-Fi npm registry for use in CI pipelines. -- **FR-CI-3**: The `@edfi/metaed-odsapi-deploy` package SHALL be installable from the Ed-Fi npm registry for deploy in CI pipelines. -- **FR-CI-4**: Both packages SHALL support the same configuration file format used by the IDE. +- **FR-DEPLOY-21**: Deploy SHALL leave generated `ApiSchema/` and `Documentation/` outputs in the artifact directory because deploy tasks only map API metadata, database, interchange, and XSD artifact families into the ODS/API source tree. + +### 3.6 API Schema Packaging Automation + +- **FR-PKG-1**: The repository MAY create MetaEd build configuration files for API Schema packaging automation using `eng/ApiSchema/CreateMetaEdConfig.ps1`. +- **FR-PKG-2**: API Schema packaging configuration generation SHALL write `eng/ApiSchema/MetaEdConfig.json` with `artifactDirectory` set to the packaging `MetaEdOutput` path, a core namespace `EdFi` project named `Ed-Fi`, an optional extension project, `defaultPluginTechVersion`, `allianceMode` set to `true`, and `suppressPrereleaseVersion` set to `true`. Extension project versions SHALL come from `metaEdProject.projectVersion` in the extension `package.json` when present; otherwise the default SHALL be `1.1.0` for TPDM and `1.0.0` for other extensions. +- **FR-PKG-3**: The API Schema packaging automation MAY run MetaEd through `eng/ApiSchema/build.ps1 -Command RunMetaEd`. +- **FR-PKG-4**: The API Schema packaging automation MAY copy generated `ApiSchema*.json` files from `MetaEdOutput` into `eng/ApiSchema` package projects for Core, TPDM, Homograph, or Sample package variants; it MAY also copy core XSD and interchange files for Core and extension XSD/interchange files for TPDM and Sample, while the Homograph packaging path currently copies only API Schema JSON. +- **FR-PKG-5**: The API Schema packaging automation MAY build, pack, and publish .NET packages through `dotnet` and Azure Artifacts when the corresponding workflow commands and credentials are supplied. ## 4. Non-Functional Requirements ### Compatibility -- **NFR-COMPAT-1**: MetaEd SHALL run on Windows, macOS, and Linux (any platform supported by VS Code and Node.js). -- **NFR-COMPAT-2**: MetaEd SHALL support Node.js LTS versions current at the time of release. -- **NFR-COMPAT-3**: The MetaEd IDE extension SHALL be compatible with the current stable release of Visual Studio Code. -- **NFR-COMPAT-4**: Generated SQL scripts SHALL be compatible with both Microsoft SQL Server and PostgreSQL. -- **NFR-COMPAT-5**: MetaEd SHALL support multiple Ed-Fi Data Standard versions (configurable via `projectVersion` and `defaultPluginTechVersion`). +- **NFR-COMPAT-1**: MetaEd SHALL run on Windows, macOS, and Linux (any platform supported by Node.js). +- **NFR-COMPAT-2**: MetaEd SHALL support Node.js LTS versions current at the time of release; CI SHALL verify on Node 22. +- **NFR-COMPAT-3**: TypeScript SHALL compile to CommonJS modules targeting ES2017. +- **NFR-COMPAT-4**: Generated SQL artifacts SHALL distinguish SQL Server and PostgreSQL output directories and generator plugins, targeting both database platforms. +- **NFR-COMPAT-5**: MetaEd SHALL support multiple Ed-Fi Data Standard versions configurable via `projectVersion` and `defaultPluginTechVersion`. ### Licensing -- **NFR-LIC-1**: Usage of the Ed-Fi Unifying Data Model SHALL require explicit acceptance of the Ed-Fi License Agreement. -- **NFR-LIC-2**: The system SHALL not execute Build or Deploy commands until the license is accepted. -- **NFR-LIC-3**: The IDE itself SHALL be freely available on the Visual Studio Marketplace under the Apache-2.0 license. +- **NFR-LIC-1**: Usage of the Ed-Fi Unifying Data Model SHALL require explicit acceptance of the Ed-Fi License Agreement via the `-a` / `--accept-license` CLI flag. +- **NFR-LIC-2**: Packages SHALL be licensed as Apache-2.0 as declared in package metadata and repository license files. ### Reliability -- **NFR-REL-1**: MetaEd runs continuously in the IDE environment. The system SHALL NOT crash or throw unhandled exceptions during normal operation; failures SHALL be reported as validation messages. -- **NFR-REL-2**: Build failures SHALL be reported with clear, actionable error messages and a non-zero exit code. +- **NFR-REL-1**: The CLI SHALL return non-zero exit codes for validation errors, pipeline failures, uncaught pipeline exceptions, and unsuccessful deploy tasks. +- **NFR-REL-2**: The output writer SHALL protect source projects by refusing to delete output directories that do not contain `MetaEdOutput` in the path. +- **NFR-REL-3**: The output writer SHALL protect source projects by refusing to write into a directory containing `.metaed` files. +- **NFR-REL-4**: Deploy tasks SHALL stop at the first unsuccessful task and return the failure message. +- **NFR-REL-5**: Build failures SHALL be reported with clear, actionable error messages and a non-zero exit code. ### Performance -- **NFR-PERF-1**: Real-time validation in the IDE SHOULD complete within 2 seconds of a file save for typical extension projects. -- **NFR-PERF-2**: Full build of a standard-sized project SHOULD complete within 60 seconds on typical developer hardware. +- **NFR-PERF-1**: Full build of a standard-sized project SHOULD complete within 60 seconds on typical developer hardware. ### Security -- **NFR-SEC-1**: MetaEd SHALL NOT transmit data to external services during operation. -- **NFR-SEC-2**: MetaEd SHALL NOT store or require credentials for normal operation. +- **NFR-SEC-1**: Normal build and deploy runtime behavior SHALL operate on local files and SHALL NOT require credentials. +- **NFR-SEC-2**: Publishing workflows MAY use package-feed credentials, but those credentials are outside normal build and deploy runtime behavior. +- **NFR-SEC-3**: MetaEd SHALL NOT transmit data to external services during normal operation. ### Observability -- **NFR-OBS-1**: Build and deploy operations SHALL produce log output with sufficient detail to diagnose failures. -- **NFR-OBS-2**: The IDE SHALL display build progress notifications and final status. +- **NFR-OBS-1**: Build and deploy commands SHALL log major stages, plugin execution, artifact directory, deploy copy operations, warnings, and elapsed time. +- **NFR-OBS-2**: Validation failures SHALL carry category, message, and source/file mapping where available. ### SDLC -- **NFR-SDLC-1**: The repository SHALL maintain automated CI checks including TypeScript type checking, ESLint, and unit tests on every pull request. -- **NFR-SDLC-2**: Test coverage SHALL include both unit tests and database integration tests (SQL Server and PostgreSQL). +- **NFR-SDLC-1**: Pull request CI SHALL run TypeScript and ESLint checks through `npm run test:lint`. +- **NFR-SDLC-2**: Pull request CI SHALL run non-database unit tests, SQL Server database tests, and PostgreSQL database tests. +- **NFR-SDLC-3**: Repository workflows SHOULD run dependency review, CodeQL analysis, and OpenSSF Scorecard checks where configured. +- **NFR-SDLC-4**: Release workflows MAY publish npm packages to Azure Artifacts using Lerna. ## 5. System Architecture -### 5.1 Component Packages - -| Package | Description | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------- | -| `metaed-core` | Core engine providing parsing, model, pipeline, validation, enhancement, and generation APIs. | -| `metaed-console` | CLI entry point for running the MetaEd build process from the command line. | -| `metaed-default-plugins` | Returns the default ordered set of MetaEd plugins used for a standard build. | -| `metaed-odsapi-deploy` | Provides deployment tasks for copying generated artifacts into an Ed-Fi ODS/API target directory. | -| `metaed-odsapi-deploy-console` | CLI entry point for running the ODS/API deployment workflow. | -| `metaed-plugin-edfi-api-catalog` | Generates an Excel catalog of API resources and their properties from the MetaEd API schema. | -| `metaed-plugin-edfi-api-schema` | Builds the enhanced Ed-Fi API schema model used by downstream plugins and generators. | -| `metaed-plugin-edfi-handbook` | Generates the Ed-Fi Data Handbook outputs (HTML and Excel) from the MetaEd model. | -| `metaed-plugin-edfi-ods-changequery` | Provides the common change-query model logic for Ed-Fi ODS/API change tracking. | -| `metaed-plugin-edfi-ods-changequery-postgresql` | Generates PostgreSQL-specific change-query database artifacts. | -| `metaed-plugin-edfi-ods-changequery-sqlserver` | Generates SQL Server-specific change-query database artifacts. | -| `metaed-plugin-edfi-ods-postgresql` | Generates Ed-Fi ODS database artifacts for PostgreSQL. | -| `metaed-plugin-edfi-ods-recordownership` | Generates the record-ownership database artifacts used by the ODS/API. | -| `metaed-plugin-edfi-ods-recordownership-postgresql` | Generates record-ownership database artifacts for PostgreSQL. | -| `metaed-plugin-edfi-ods-recordownership-sqlserver` | Generates record-ownership database artifacts for SQL Server. | -| `metaed-plugin-edfi-ods-relational` | Provides the shared relational ODS generation logic used by database-specific plugins. | -| `metaed-plugin-edfi-ods-sqlserver` | Generates Ed-Fi ODS database artifacts for SQL Server. | -| `metaed-plugin-edfi-odsapi` | Generates the Ed-Fi ODS/API metadata artifacts and coordinates overall API build outputs. | -| `metaed-plugin-edfi-sql-dictionary` | Generates the Ed-Fi SQL data dictionary output. | -| `metaed-plugin-edfi-unified` | Provides the shared unified model foundation (validators, enhancers) used by multiple plugins. | -| `metaed-plugin-edfi-unified-advanced` | Provides the advanced unified model layer for richer cross-plugin generation. | -| `metaed-plugin-edfi-xml-dictionary` | Generates the Ed-Fi XML data dictionary output. | -| `metaed-plugin-edfi-xsd` | Generates Ed-Fi XSD schema artifacts. | - -### 5.2 Runtime and Deployment - -- **Runtime**: Node.js (with TypeScript compiled to ES2017/CommonJS) -- **Package Management**: npm workspaces managed by Lerna -- **Grammar**: ANTLR 4.6 (JavaScript target) -- **Distribution**: npm packages published to the Ed-Fi Azure DevOps npm registry; VS Code extension published to the Visual Studio Marketplace - -### 5.3 Processing Pipeline - -The system follows a sequential pipeline: **Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write**. The DSL and pipeline internals are documented separately and are out of scope for this PRD. +### 5.1 Processing Pipeline + +The build pipeline is sequential: + +1. Initialize MetaEd environment. +2. Set up plugins and target technology versions. +3. Load `.metaed` files. +4. Validate syntax. +5. Load file indexes. +6. Build the parse tree. +7. Walk builders to construct the semantic model. +8. Initialize namespaces. +9. Load plugin configuration. +10. For each plugin in dependency order, run validators, enhancers, and generators. +11. Write generated output. +12. Map validation failures to files. + +Plugins are ordered by `@edfi/metaed-default-plugins`, allowing upstream plugins to create semantic data used by downstream artifact generators. + +### 5.2 Component Packages + +| Package | Product Responsibility | +| --- | --- | +| `@edfi/metaed-core` | Core pipeline, configuration model, file loading, grammar parsing, model builders, validators/enhancers/generators contracts, plugin configuration loading, and output writing. | +| `@edfi/metaed-console` | Build CLI for running the MetaEd pipeline with default plugins. | +| `@edfi/metaed-default-plugins` | Default ordered plugin list used by the build and deploy-console build flow. | +| `@edfi/metaed-odsapi-deploy` | Deploy task library for copying generated artifacts into an ODS/API source tree. | +| `@edfi/metaed-odsapi-deploy-console` | Deploy CLI for source-scan build-and-deploy mode or config-based deploy mode. | +| `@edfi/metaed-plugin-edfi-unified` | Foundational Ed-Fi semantic validators and enhancers, including reference resolution, base class resolution, domain/subdomain links, merge directives, and shared type enrichment. | +| `@edfi/metaed-plugin-edfi-unified-advanced` | Additional validation for merge scenarios, deprecation warnings, common property identity rules, and self-referencing property role names. | +| `@edfi/metaed-plugin-edfi-ods-relational` | Shared relational ODS model enhancers and relational validators used by SQL Server, PostgreSQL, API Schema, dictionaries, change query, and record ownership plugins. | +| `@edfi/metaed-plugin-edfi-api-schema` | API Schema model enhancers, security-related configuration rules, OpenAPI fragment and relational metadata, and `ApiSchema*.json` generation with API Schema version metadata. | +| `@edfi/metaed-plugin-edfi-ods-sqlserver` | SQL Server ODS naming, schema/table/column enhancements, and SQL Server ODS SQL script generation. | +| `@edfi/metaed-plugin-edfi-ods-postgresql` | PostgreSQL ODS naming, schema/table/column enhancements, and PostgreSQL ODS SQL script generation. | +| `@edfi/metaed-plugin-edfi-xsd` | XSD model enhancers, duplicate-name validation for dependency namespaces, core/extension XSD generation, schema annotation generation, and interchange XSD generation. | +| `@edfi/metaed-plugin-edfi-ods-changequery` | Common change query validator, common model types, and shared generation helpers for change query SQL artifacts. | +| `@edfi/metaed-plugin-edfi-ods-changequery-sqlserver` | SQL Server-specific change query enhancers and SQL script generators. | +| `@edfi/metaed-plugin-edfi-ods-changequery-postgresql` | PostgreSQL-specific change query enhancers and SQL script generators. | +| `@edfi/metaed-plugin-edfi-ods-recordownership` | Common record ownership table enhancement that marks aggregate root tables for ownership token columns for target technology versions `>=3.3.0`. | +| `@edfi/metaed-plugin-edfi-ods-recordownership-sqlserver` | SQL Server record ownership SQL generation. | +| `@edfi/metaed-plugin-edfi-ods-recordownership-postgresql` | PostgreSQL record ownership SQL generation. | +| `@edfi/metaed-plugin-edfi-odsapi` | ODS/API model validators/enhancers and `ApiModel*.json` generation. | +| `@edfi/metaed-plugin-edfi-xml-dictionary` | XML data dictionary Excel generation from XSD model data. | +| `@edfi/metaed-plugin-edfi-sql-dictionary` | SQL data dictionary Excel generation from relational ODS model data. | +| `@edfi/metaed-plugin-edfi-handbook` | Handbook-specific model enhancers plus HTML and Excel Data Handbook generation. | +| `@edfi/metaed-plugin-edfi-api-catalog` | API Catalog Excel generation from API Schema and OpenAPI fragment data. | + +### 5.3 Default Plugin Order + +The default plugin order is: + +`edfiUnified`, `edfiUnifiedAdvanced`, `edfiOdsRelational`, `edfiApiSchema`, `edfiOdsSqlServer`, `edfiOdsPostgresql`, `edfiXsd`, `edfiOdsChangeQuery`, `edfiOdsChangeQuerySqlServer`, `edfiOdsChangeQueryPostgresql`, `edfiOdsRecordOwnership`, `edfiOdsRecordOwnershipSqlServer`, `edfiOdsRecordOwnershipPostgresql`, `edfiOdsApi`, `edfiXmlDictionary`, `edfiSqlDictionary`, `edfiHandbook`, `edfiApiCatalog`. + +### 5.4 Output Model + +Generators return `GeneratedOutput` objects containing a human-readable name, namespace, folder name, file name, and either string or binary stream output. The writer creates directories recursively and writes files beneath the artifact directory using `{namespace}/{folderName}/{fileName}`. ## 6. Out of Scope and Known Limitations -### Explicit Exclusions from the Document - -- **The MetaEd DSL syntax and semantics** — documented in the MetaEd Language Specification. -- **Internal compilation pipeline details** — the Initialize → Load → Parse → Build → Validate → Enhance → Generate → Write pipeline is documented in developer-facing architecture docs, not in this product requirements document. -- **Internal package implementation details** — individual package internals are documented in developer-facing architecture docs. -- **ODS/API runtime behavior** — MetaEd generates artifacts for the ODS/API; it does not define ODS/API runtime requirements. -- **Data Standard content** — the actual entities, descriptors, and associations in any Data Standard version are authored in `.metaed` files and are not part of this product's requirements. - -### Known Limitations of the Current Implementation - -- MetaEd does not validate that the ODS/API deployment directory is correctly structured before deploying; it assumes the target follows the expected repository layout. -- The IDE's workspace file includes user-specific paths (e.g., to `node_modules` inside the extension directory) that are not portable across machines and should not be committed to version control. -- Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent and costly errors. -- The Deploy command deletes existing extension artifact directories by default unless `--suppressDelete` is specified. +### 6.1 Explicit Exclusions + +- **MetaEd IDE (VS Code Extension)** — the Visual Studio Code extension for authoring `.metaed` files is maintained in a separate repository and is out of scope for this PRD. Its build and deploy capabilities are driven by the CLI packages documented here. +- **The MetaEd DSL language specification and grammar semantics** — documented separately. +- **ODS/API runtime behavior, API service hosting, database execution, migrations, and production deployment** — outside this PRD. +- **Data Standard content** — this repository processes model files but does not define data element requirements. + +### 6.2 Known Limitations + +- `MetaEdConfiguration` includes `pluginTechVersion`, and some fixtures contain it, but current plugin setup assigns every plugin the single `defaultPluginTechVersion`. +- The license acceptance CLI option is required by yargs, but the source does not validate the option value or persist acceptance. +- The build console's custom config loader resolves relative `-c` / `--config` paths relative to the console module directory; absolute config paths avoid that ambiguity. +- Config-based deploy mode runs deploy tasks against an existing `artifactDirectory`; source-scan mode is the path that builds before deploy. +- Config-based deploy mode copies the supplied `metaEdConfiguration` without merging defaults from `newMetaEdConfiguration`, and it does not apply the `--suppressPrereleaseVersion` CLI option to the supplied configuration. +- Source-scan deploy mode returns without deploying when required source-mode inputs are absent, including `source` or `projectNames`. +- After source-scan deploy mode attempts a build, the deploy console still invokes deploy tasks even when the build has set a non-zero exit code; if an existing artifact directory remains, deploy can attempt to copy those artifacts. +- Source-scan project discovery scans subdirectories only while no projects have been discovered yet, so mixed source inputs can miss nested projects after an earlier project has already been found. +- Source-scan project discovery de-duplicates projects by project name, so two discovered projects with the same project name cannot both be represented in one deploy build. +- Source-scan namespace derivation returns an empty namespace when `projectName` cannot be converted to an alphanumeric string that starts with an uppercase letter. +- Deploy copies extension artifacts from folders named by `projectName`, while generators write output folders using `namespaceName`; source-scan and config-based deployments where those values differ can cause deploy to skip generated extension folders. +- Source-scan project sorting only special-cases a project whose `projectName` is exactly `EdFi`; Data Standard packages with `projectName` values such as `Ed-Fi` are sorted alphabetically with other discovered projects after namespace derivation. +- Core and extension artifact copy tasks are no-ops for `defaultPluginTechVersion` values below `7.0.0`. +- A core-only deploy with no extension artifact folders can fail the extension project existence precheck when `allianceMode` is false, because that precheck runs before core copy and ignores root `EdFi`, `Documentation`, and `ApiSchema` folders. +- Deploy does not copy generated `ApiSchema/` or `Documentation/` artifact folders into the ODS/API source tree. +- Deploy assumes the local ODS/API source tree follows expected `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` folder conventions. +- The console packages declare Node.js `main` entry points but no package `bin` commands, so command execution is through Node.js entry points or repository scripts rather than package-installed binary names. +- API Schema packaging `RunMetaEd` copies `packages/metaed-plugin-edfi-api-schema/test/integration/edfiApiSchema.config.json` into the core project path before running MetaEd, so packaging output depends on that checked-in configuration fixture. +- The API Schema packaging workflow matrix uses `TPDM`, but the TPDM checkout step currently checks for lowercase `tpdm`, so that workflow step does not run for the TPDM matrix entries as written. +- The API Catalog generator contains hard-coded handling for EducationOrganization and SchoolYear reference schemas because those reference schemas are not always discoverable from generated OpenAPI fragments. +- The output writer's deletion guard is path-name based: it requires `MetaEdOutput` to appear in the output path before deleting an existing output directory. +- Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent errors. ## 7. Glossary -| Term | Definition | -| -------------------- | ----------------------------------------------------------------------------------------------------------------- | -| **MetaEd** | Umbrella term for the DSL, the compiler, the IDE, and the supporting packages. | -| **MetaEd DSL** | The domain-specific language used to define Ed-Fi data models. | -| **MetaEd IDE** | The Visual Studio Code extension providing authoring and build/deploy capabilities. | -| **MetaEd Generator** | The command-line application that compiles `.metaed` files into artifacts. | -| **ODS** | Operational Data Store — the database backing an Ed-Fi API. | -| **ODS/API** | The Ed-Fi ODS / API platform is the legacy reference implementation of REST API based on the Ed-Fi Data Standard. | -| **Ed-Fi API** | The successor to the ODS/API, which also consumes MetaEd-generated artifacts, particularly the API metadata. | -| **Data Standard** | The Ed-Fi Unifying Data Model, versioned and expressed in `.metaed` files. | -| **Extension** | A user-defined addition to the core Ed-Fi Data Standard. | -| **Alliance Mode** | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | -| **Plugin** | A MetaEd package that provides validators, enhancers, and/or generators for a specific output format. | -| **Artifact** | Any file generated by the MetaEd build process (SQL, XSD, JSON, HTML, Excel). | -| **Deploy** | The act of copying generated artifacts into the ODS/API source tree for its build process to consume. | -| **Namespace** | A scoping mechanism for MetaEd projects (e.g., `EdFi` for core, custom names for extensions). | -| **Tech Version** | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | -| **SemVer** | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | +| Term | Definition | +| --- | --- | +| MetaEd-js | The Node.js monorepo that parses MetaEd DSL projects and generates artifacts through plugins. | +| MetaEd | Umbrella term for the DSL, the compiler, and the supporting packages. | +| MetaEd DSL | The `.metaed` source language processed by this repository. | +| MetaEd Generator | The command-line application that compiles `.metaed` files into artifacts. | +| Data Standard project | The core Ed-Fi project identified by namespace `EdFi`; exactly one is required by the CLIs. | +| Extension project | A non-EdFi MetaEd project processed alongside the Data Standard project. | +| Namespace | The model and output grouping used by MetaEd to separate core and extension artifacts. | +| Project extension | A string used by generators in some extension artifact file names; non-EdFi projects default to `EXTENSION` when metadata omits it. | +| Default plugin technology version | The version string assigned to plugin environments to select version-specific generator behavior. | +| Plugin | A package that contributes validators, enhancers, generators, and optional configuration schemas. | +| Validator | A plugin function that reports syntax, semantic, relationship, or configuration failures. | +| Enhancer | A plugin function that adds derived semantic data to the model for downstream processing. | +| Generator | A plugin function that produces one or more generated outputs. | +| Artifact | A generated file written by the build pipeline, such as SQL, XSD, JSON, HTML, or Excel. | +| ODS | Operational Data Store — the database backing an Ed-Fi API. | +| ODS/API | The downstream Ed-Fi source tree layout targeted by deploy tasks; the legacy reference implementation of the Ed-Fi REST API. | +| Ed-Fi API | Successor to the ODS/API, which also consumes MetaEd-generated artifacts, particularly the API metadata. | +| API Model | ODS/API metadata JSON generated as `ApiModel*.json`. | +| API Schema | JSON schema output generated as `ApiSchema*.json` for API schema consumers. | +| Change Query | SQL artifacts for change version, tracked delete, and related change tracking support. | +| Record Ownership | SQL artifacts that add ownership token support to aggregate root tables when enabled by target version. | +| Alliance Mode | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | +| SemVer | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | +| Tech Version | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | From aa2756c82b111d0f2341f3c13a597bb324326911 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Mon, 22 Jun 2026 08:39:28 -0500 Subject: [PATCH 12/15] doc: fix a bad link, reformat file --- packages/metaed-core/docs/jsonnet-configuration.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/packages/metaed-core/docs/jsonnet-configuration.md b/packages/metaed-core/docs/jsonnet-configuration.md index ff2e11d83..b420c15bc 100644 --- a/packages/metaed-core/docs/jsonnet-configuration.md +++ b/packages/metaed-core/docs/jsonnet-configuration.md @@ -14,6 +14,7 @@ MetaEd now supports [Jsonnet](https://jsonnet.org/) configuration files in addit ## File Naming Configuration files can use either extension: + - `.config.json` - Standard JSON configuration - `.config.jsonnet` - Jsonnet configuration with advanced features @@ -26,6 +27,7 @@ Since JSON is a valid subset of Jsonnet, you can rename any `.config.json` file ## Basic Example ### Original JSON + ```json { "config": [ @@ -56,6 +58,7 @@ Since JSON is a valid subset of Jsonnet, you can rename any `.config.json` file ``` ### Improved Jsonnet + ```jsonnet // Define common values as variables local namespace = "EdFi"; @@ -109,6 +112,7 @@ local enableFeature = std.extVar("ENABLE_FEATURE") == "true"; ``` Pass external variables via MetaEd configuration: + ```javascript { externalVariables: { @@ -120,7 +124,8 @@ Pass external variables via MetaEd configuration: ### Complex Configuration Example -See [examples/edfiApiSchema.config.jsonnet](../examples/edfiApiSchema.config.jsonnet) for a comprehensive example showing: +See [examples/edfiApiSchema.config.jsonnet](../../metaed-plugin-edfi-api-schema/examples/edfiApiSchema.config.jsonnet) for a comprehensive example showing: + - Helper functions for creating rules - Version range constants - Conditional configuration @@ -140,4 +145,4 @@ All existing JSON configuration files continue to work without any changes. Json ### Validation -Configuration validation using Joi schemas works the same for both JSON and Jsonnet files. The Jsonnet is evaluated to JSON first, then validated against the plugin's configuration schema. \ No newline at end of file +Configuration validation using Joi schemas works the same for both JSON and Jsonnet files. The Jsonnet is evaluated to JSON first, then validated against the plugin's configuration schema. From 1c080d48d1befcada2532cbdedab4044e678cc7b Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Mon, 22 Jun 2026 08:56:05 -0500 Subject: [PATCH 13/15] doc: correct inaccuracies in package READMEs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix verifiably wrong claims found by comparing README content to source: - xml-dictionary: output folder is DataDictionary/ not Documentation/; sheet names are "Complex Types"/"Simple Types" (with spaces) - ods-changequery: initialize() registers no enhancers or generators; SQL generation is done by companion plugins using exported utilities - unified: is not a "pure enhancer plugin" — also registers ~70 validators - ods-recordownership: only flags aggregate root tables, not every table - sql-dictionary: depends on both ods-relational (model) and ods-sqlserver (table/column naming data) - metaed-core: GeneratedOutput description now includes the name field and correctly types resultStream as Buffer Co-Authored-By: Claude Sonnet 4.6 --- packages/metaed-core/README.md | 5 ++-- .../README.md | 30 +++++++------------ .../README.md | 5 ++-- .../README.md | 5 ++-- packages/metaed-plugin-edfi-unified/README.md | 11 +++---- .../README.md | 4 +-- 6 files changed, 28 insertions(+), 32 deletions(-) diff --git a/packages/metaed-core/README.md b/packages/metaed-core/README.md index 56f8f1af2..f33d181c9 100644 --- a/packages/metaed-core/README.md +++ b/packages/metaed-core/README.md @@ -21,8 +21,9 @@ abstractions, and pipeline orchestration used by all MetaEd packages. - Parsed MetaEd model (domain entities, associations, descriptors, etc.) - Pipeline execution results (validation failures, generated output) -- `GeneratedOutput` objects containing file name, folder, namespace, and content - (as string or stream) +- `GeneratedOutput` objects containing a human-readable name, namespace, file name, + folder name, and content as either a string (`resultString`) or a binary `Buffer` + (`resultStream`; takes precedence over `resultString` when set) ## Business Logic diff --git a/packages/metaed-plugin-edfi-ods-changequery/README.md b/packages/metaed-plugin-edfi-ods-changequery/README.md index 234506a14..6d609bd8f 100644 --- a/packages/metaed-plugin-edfi-ods-changequery/README.md +++ b/packages/metaed-plugin-edfi-ods-changequery/README.md @@ -1,7 +1,8 @@ # metaed-plugin-edfi-ods-changequery -MetaEd plugin providing shared change-query support for the ODS. Generates -database-agnostic SQL DDL for change tracking infrastructure. +MetaEd plugin providing shared change-query support for the ODS. Registers a namespace +validator and exports utilities used by the PostgreSQL and SQL Server companion plugins +to enrich models and generate change-tracking SQL. ## Input Configuration @@ -9,23 +10,14 @@ No plugin-specific configuration. Uses the MetaEd model and target technology ve ## Output -Generates SQL DDL scripts for change-query support (numbered for execution order): - -- `0010-CreateChangesSchema.sql` -- `0020-CreateChangeVersionSequence.sql` -- `0030-AddColumnChangeVersionForTables.sql` -- `0045-CreateTrackedDeleteSchema.sql` -- `0050-CreateTrackedDeleteTables.sql` -- `0060-CreateDeletedForTrackingTriggers.sql` -- `0070-AddIndexChangeVersionForTables.sql` -- `0130-CreateTriggerUpdateChangeVersion*.sql` -- `0230-CreateIndirectUpdateCascadeTriggers.sql` - -Script names vary by target technology version (style 6.0 uses alternate numbering). +No file artifacts. This plugin registers no enhancers or generators itself. SQL DDL for +change-query support (schemas, sequences, columns, triggers, indexes) is generated by +the platform-specific companion plugins (`metaed-plugin-edfi-ods-changequery-postgresql` +and `metaed-plugin-edfi-ods-changequery-sqlserver`), which call the shared utilities +exported from this package. ## Business Logic -Validates namespaces for change-query compatibility, enriches entities with -change-tracking metadata, and generates the common SQL objects (schemas, sequences, -columns, triggers, indexes) needed for ODS change tracking. Platform-specific SQL -is delegated to the PostgreSQL and SQL Server companion plugins. +Validates that no namespace is named "Changes" (which would conflict with the +change-tracking schema). Exports shared enhancer helpers and generator utilities consumed +by the companion plugins for platform-specific SQL generation. diff --git a/packages/metaed-plugin-edfi-ods-recordownership/README.md b/packages/metaed-plugin-edfi-ods-recordownership/README.md index 58ce42e17..54d9bc303 100644 --- a/packages/metaed-plugin-edfi-ods-recordownership/README.md +++ b/packages/metaed-plugin-edfi-ods-recordownership/README.md @@ -14,6 +14,7 @@ with `edfiOdsRecordOwnership` data on each table. ## Business Logic -Marks every ODS table with a record-ownership flag so that downstream platform-specific +Marks aggregate root ODS tables with a record-ownership flag so that downstream platform-specific generators (PostgreSQL, SQL Server) can add ownership-token columns when the feature is -enabled. Acts as the shared foundation for the record-ownership companion plugins. +enabled. Non-aggregate-root tables are not flagged. Acts as the shared foundation for the +record-ownership companion plugins. diff --git a/packages/metaed-plugin-edfi-sql-dictionary/README.md b/packages/metaed-plugin-edfi-sql-dictionary/README.md index 73e8b42a0..82bda31c6 100644 --- a/packages/metaed-plugin-edfi-sql-dictionary/README.md +++ b/packages/metaed-plugin-edfi-sql-dictionary/README.md @@ -4,8 +4,9 @@ MetaEd plugin that generates an Excel SQL data dictionary from ODS table metadat ## Input Configuration -No plugin-specific configuration. Depends on relational ODS model data produced by -`metaed-plugin-edfi-ods-relational`. +No plugin-specific configuration. Depends on ODS table and column model data from +`metaed-plugin-edfi-ods-relational` and SQL Server-specific naming data (table names, +column names, data types) from `metaed-plugin-edfi-ods-sqlserver`. ## Output diff --git a/packages/metaed-plugin-edfi-unified/README.md b/packages/metaed-plugin-edfi-unified/README.md index f52454426..ab94e249b 100644 --- a/packages/metaed-plugin-edfi-unified/README.md +++ b/packages/metaed-plugin-edfi-unified/README.md @@ -5,7 +5,7 @@ downstream plugins. ## Input Configuration -No plugin-specific configuration. This is a pure enhancer plugin. +No plugin-specific configuration. ## Output @@ -14,7 +14,8 @@ transformations used by other plugins (XSD, XML dictionary, ODS, etc.). ## Business Logic -Applies core "unified" model enhancements that shape entities, properties, and -relationships into forms required by downstream generators. Provides the shared model -foundation for interchange schemas, type hierarchies, and cross-entity relationships -that multiple output plugins depend on. +Validates a broad set of model rules (entity naming, property references, cardinality +constraints, extension correctness, etc.) and applies core model enhancements that shape +entities, properties, and relationships into forms required by downstream generators. +Provides the shared model foundation for interchange schemas, type hierarchies, and +cross-entity relationships that multiple output plugins depend on. diff --git a/packages/metaed-plugin-edfi-xml-dictionary/README.md b/packages/metaed-plugin-edfi-xml-dictionary/README.md index dc13e1e3c..230f9f330 100644 --- a/packages/metaed-plugin-edfi-xml-dictionary/README.md +++ b/packages/metaed-plugin-edfi-xml-dictionary/README.md @@ -11,9 +11,9 @@ No plugin-specific configuration. Depends on XSD model data produced by Generates a single Excel workbook: -- `Documentation/XmlDataDictionary.xlsx` +- `DataDictionary/XmlDataDictionary.xlsx` -The workbook contains three sheets: Elements, ComplexTypes, and SimpleTypes. +The workbook contains three sheets: Elements, Complex Types, and Simple Types. ## Business Logic From 9349e0c7251b164972f63b80e108f76a0e0ff7f5 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Mon, 22 Jun 2026 09:08:33 -0500 Subject: [PATCH 14/15] doc: address remaining inaccuracies from Brad's PR review Co-Authored-By: Claude Sonnet 4.6 --- packages/metaed-console/README.md | 6 +++--- packages/metaed-core/README.md | 7 ++++--- .../metaed-odsapi-deploy-console/README.md | 20 ++++++++++++------- .../metaed-plugin-edfi-handbook/README.md | 8 ++++---- .../README.md | 2 +- .../README.md | 2 +- .../README.md | 2 +- .../README.md | 2 +- packages/metaed-plugin-edfi-xsd/README.md | 5 ++++- 9 files changed, 32 insertions(+), 22 deletions(-) diff --git a/packages/metaed-console/README.md b/packages/metaed-console/README.md index 863033255..3576de334 100644 --- a/packages/metaed-console/README.md +++ b/packages/metaed-console/README.md @@ -6,7 +6,7 @@ Command-line interface for running the MetaEd build pipeline. CLI arguments via `yargs`: -- `--config / -c` — Path to a JSON configuration file (relative to working directory) +- `--config / -c` — Path to a JSON configuration file (relative paths are resolved relative to the console module directory; use absolute paths to avoid ambiguity) - `--defaultPluginTechVersion / -x` — Default plugin technology version - `--accept-license / -a` — Required flag to accept the license agreement - `--suppressPrereleaseVersion` — Suppress pre-release version in output paths (default: `true`) @@ -28,8 +28,8 @@ and generators in sequence. Logs timing information for each phase. ## Usage -1. Build the project from the parent directory with `npm run build`. -2. To confirm it is functional, try `node ./dist/index.js -h`. +1. Build the project from the repo root with `npm run build`. +2. To confirm it is functional, try `node packages/metaed-console/dist/index.js -h`. 3. The easiest way to run this is with a config file. See `metaed-edfi-5.2.json` for a fully-worked sample config file. Note that this shows Alliance Mode _on_, which is only appropriate in the Ed-Fi Alliance's build processes. External users should set this to `false`. To run with a config file: diff --git a/packages/metaed-core/README.md b/packages/metaed-core/README.md index f33d181c9..1ad50a126 100644 --- a/packages/metaed-core/README.md +++ b/packages/metaed-core/README.md @@ -9,7 +9,7 @@ abstractions, and pipeline orchestration used by all MetaEd packages. - `artifactDirectory` — Where generated artifacts are written - `deployDirectory` — Target for deployment operations -- `pluginTechVersion` — Technology version for plugin selection +- `pluginTechVersion` — Technology version field (present for compatibility; current plugin setup assigns every plugin `defaultPluginTechVersion`) - `defaultPluginTechVersion` — Fallback technology version - `projects` — Array of project definitions (name, version, namespace, paths) - `projectPaths` — File system paths for MetaEd source files @@ -27,7 +27,8 @@ abstractions, and pipeline orchestration used by all MetaEd packages. ## Business Logic -Orchestrates the sequential pipeline: initialize → load → parse → build → validate → -enhance → generate → write. Defines the plugin contract (validators, enhancers, +Orchestrates the sequential pipeline: initialize → load → parse → build → namespace +init → plugin config load → then for each plugin in dependency order: validate → +enhance → generate → write output. Defines the plugin contract (validators, enhancers, generators), the domain model types, and the shared infrastructure for file I/O, logging, and configuration resolution. diff --git a/packages/metaed-odsapi-deploy-console/README.md b/packages/metaed-odsapi-deploy-console/README.md index 4955bfb05..57e26569e 100644 --- a/packages/metaed-odsapi-deploy-console/README.md +++ b/packages/metaed-odsapi-deploy-console/README.md @@ -8,8 +8,8 @@ CLI arguments via `yargs`: - `--config / -c` — Path to JSON configuration file - `--source / -s` — Source project directories (array) -- `--target / -t` — Target ODS/API repository path -- `--projectNames / -p` — Project names to deploy (array) +- `--target / -t` — Parent directory containing the `Ed-Fi-ODS` and `Ed-Fi-ODS-Implementation` repositories +- `--projectNames / -p` — Project name overrides applied to discovered projects in discovery order (array) - `--defaultPluginTechVersion / -x` — Default plugin technology version - `--core` — Deploy core artifacts - `--suppressDelete` — Skip removal of existing extension artifacts @@ -20,12 +20,18 @@ CLI arguments via `yargs`: ## Output -Runs the MetaEd build pipeline for the specified projects, then deploys generated -artifacts into the target ODS/API repository. Exits with code 0 on success, 1 on +Deploys MetaEd artifacts into the target ODS/API repository structure. In source-scan +mode, also runs the MetaEd build pipeline first. Exits with code 0 on success, 1 on failure, and logs duration. ## Business Logic -Scans source directories for MetaEd projects, builds `MetaEdConfiguration`, runs the -generation pipeline, then delegates to `metaed-odsapi-deploy` to copy artifacts into -the destination repository structure. +Two operating modes: + +- **Source-scan mode** (`--source`/`--target`): scans source directories for MetaEd + projects, builds `MetaEdConfiguration`, runs the full generation pipeline, then + delegates to `metaed-odsapi-deploy` to copy artifacts into the destination repository + structure. +- **Config-based mode** (`--config`): uses the supplied `metaEdConfiguration` with a + pre-built `artifactDirectory` and runs only the deploy tasks — the build pipeline is + not executed. diff --git a/packages/metaed-plugin-edfi-handbook/README.md b/packages/metaed-plugin-edfi-handbook/README.md index bd1368b03..c22b6e99e 100644 --- a/packages/metaed-plugin-edfi-handbook/README.md +++ b/packages/metaed-plugin-edfi-handbook/README.md @@ -10,10 +10,10 @@ No plugin-specific configuration. Relies on model data enriched by upstream enha Generates two artifacts: -- `Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` — Interactive HTML single-page - application for browsing handbook entries -- `Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx` — Excel workbook with the same content in - spreadsheet form +- `Documentation/Ed-Fi-Handbook/Ed-Fi-Data-Handbook-Index.html` — Interactive HTML + single-page application for browsing handbook entries +- `Documentation/Ed-Fi-Handbook/Ed-Fi-Handbook.xlsx` — Excel workbook with the same + content in spreadsheet form ## Business Logic diff --git a/packages/metaed-plugin-edfi-ods-postgresql/README.md b/packages/metaed-plugin-edfi-ods-postgresql/README.md index d3e8d80ca..6d27657ea 100644 --- a/packages/metaed-plugin-edfi-ods-postgresql/README.md +++ b/packages/metaed-plugin-edfi-ods-postgresql/README.md @@ -18,7 +18,7 @@ Generates PostgreSQL ODS scripts under `Database/PostgreSQL/ODS/`: - `Data/0010-*-Enumerations.sql` — Descriptor/enumeration seed data - `Data/0020-*-SchoolYears.sql` — School year seed data -Additional scripts for indexes and authorization views are also generated. +Additional scripts for indexes and education organization authorization indexes are also generated. ## Business Logic diff --git a/packages/metaed-plugin-edfi-ods-sqlserver/README.md b/packages/metaed-plugin-edfi-ods-sqlserver/README.md index 50f4ff837..1ec33529e 100644 --- a/packages/metaed-plugin-edfi-ods-sqlserver/README.md +++ b/packages/metaed-plugin-edfi-ods-sqlserver/README.md @@ -18,7 +18,7 @@ Generates SQL Server ODS scripts under `Database/SQLServer/ODS/`: - `Data/0010-*-Enumerations.sql` — Descriptor/enumeration seed data - `Data/0020-*-SchoolYears.sql` — School year seed data -Additional scripts for indexes and authorization views are also generated. +Additional scripts for indexes and education organization authorization indexes are also generated. ## Business Logic diff --git a/packages/metaed-plugin-edfi-sql-dictionary/README.md b/packages/metaed-plugin-edfi-sql-dictionary/README.md index 82bda31c6..d1d332a5c 100644 --- a/packages/metaed-plugin-edfi-sql-dictionary/README.md +++ b/packages/metaed-plugin-edfi-sql-dictionary/README.md @@ -12,7 +12,7 @@ column names, data types) from `metaed-plugin-edfi-ods-sqlserver`. Generates a single Excel workbook: -- `DataDictionary/SqlDataDictionary.xlsx` +- `Documentation/DataDictionary/SqlDataDictionary.xlsx` The workbook contains two sheets: Tables (table names and descriptions) and Columns (column names, types, constraints per table). diff --git a/packages/metaed-plugin-edfi-xml-dictionary/README.md b/packages/metaed-plugin-edfi-xml-dictionary/README.md index 230f9f330..ef173df84 100644 --- a/packages/metaed-plugin-edfi-xml-dictionary/README.md +++ b/packages/metaed-plugin-edfi-xml-dictionary/README.md @@ -11,7 +11,7 @@ No plugin-specific configuration. Depends on XSD model data produced by Generates a single Excel workbook: -- `DataDictionary/XmlDataDictionary.xlsx` +- `Documentation/DataDictionary/XmlDataDictionary.xlsx` The workbook contains three sheets: Elements, Complex Types, and Simple Types. diff --git a/packages/metaed-plugin-edfi-xsd/README.md b/packages/metaed-plugin-edfi-xsd/README.md index 0794d40a2..94b60a5f3 100644 --- a/packages/metaed-plugin-edfi-xsd/README.md +++ b/packages/metaed-plugin-edfi-xsd/README.md @@ -9,11 +9,14 @@ transformations applied. ## Output -Generates XSD artifacts under `XSD/`: +Generates XSD artifacts in two directories: +`XSD/`: - `Ed-Fi-Core.xsd` (for core namespace) - `{Extension}-Ed-Fi-Extended-Core.xsd` (for extension namespaces) - `SchemaAnnotation.xsd` — Schema annotation definitions + +`Interchange/`: - `Interchange-{Name}.xsd` — Interchange schemas (for core) - `{Extension}-Interchange-{Name}-Extension.xsd` — Interchange extension schemas From e75d62312295fc75f73bfbb9549766635b1ef8d2 Mon Sep 17 00:00:00 2001 From: "Stephen A. Fuqua" Date: Mon, 22 Jun 2026 11:45:49 -0500 Subject: [PATCH 15/15] docs: correct PRD and package readme claims Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/PRD-4.x.md | 14 +++++++------- packages/metaed-console/README.md | 5 +++-- packages/metaed-plugin-edfi-api-schema/README.md | 2 +- packages/metaed-plugin-edfi-unified/README.md | 4 ++-- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/PRD-4.x.md b/docs/PRD-4.x.md index c2602c15f..0bd8720bb 100644 --- a/docs/PRD-4.x.md +++ b/docs/PRD-4.x.md @@ -9,7 +9,7 @@ > [!TIP] Document Scope > This PRD defines the product requirements for MetaEd version 4.x, covering the Node.js monorepo that implements the MetaEd generator and its command-line interfaces. It covers the core features, functional and non-functional requirements, system architecture, and known limitations. > -> It does _not cover_ the MetaEd DSL syntax and semantics, the internal compilation pipeline, the MetaEd IDE Visual Studio Code extension (maintained in a separate repository), the Ed-Fi ODS/API runtime behavior, database execution semantics, or the Data Standard content itself. +> It does _not cover_ the MetaEd DSL syntax and semantics, the MetaEd IDE Visual Studio Code extension (maintained in a separate repository), the Ed-Fi ODS/API runtime behavior, database execution semantics, or the Data Standard content itself. ## 1. Product Overview @@ -41,7 +41,7 @@ The product value is artifact consistency: SQL, XSD, API metadata, API schema, d ### 1.2 Target Users and Personas -- **Ed-Fi Alliance Technical Team / Data Standard maintainers** — maintain the core Ed-Fi Data Standard MetaEd project and need repeatable generated artifacts for release and packaging workflows. Require full control over core model files and use advanced features like plugin configuration, Alliance Mode, and CI builds. +- **Ed-Fi Alliance Technical Team / Data Standard maintainers** — maintain the core Ed-Fi Data Standard MetaEd project and need repeatable generated artifacts for release and packaging workflows. Use internal workflows that rely on plugin configuration, Alliance Mode-enabled builds, and CI automation. - **Education Agency Developers and Ed-Fi Integration Partners / Extension developers** — maintain additive MetaEd extension projects and need validation plus generated artifacts for ODS/API integration. Build and maintain extensions on behalf of education agencies using both CLI workflows and CI pipelines. - **Build and release engineers** — run MetaEd headlessly in CI, publish npm packages, package API Schema artifacts, and verify generated outputs against authoritative fixtures. - **Business Analysts / Downstream artifact consumers** — use generated spreadsheets, HTML documentation, JSON schema and metadata files, SQL scripts, and XSD files without reading `.metaed` source. Review generated documentation to understand the data model. @@ -251,8 +251,8 @@ The product value is artifact consistency: SQL, XSD, API metadata, API schema, d ### Compatibility -- **NFR-COMPAT-1**: MetaEd SHALL run on Windows, macOS, and Linux (any platform supported by Node.js). -- **NFR-COMPAT-2**: MetaEd SHALL support Node.js LTS versions current at the time of release; CI SHALL verify on Node 22. +- **NFR-COMPAT-1**: MetaEd SHALL run in Node.js-based local and CI environments compatible with its dependencies; this repository's automation currently verifies Ubuntu-based execution. +- **NFR-COMPAT-2**: Repository automation SHALL verify Node 22. - **NFR-COMPAT-3**: TypeScript SHALL compile to CommonJS modules targeting ES2017. - **NFR-COMPAT-4**: Generated SQL artifacts SHALL distinguish SQL Server and PostgreSQL output directories and generator plugins, targeting both database platforms. - **NFR-COMPAT-5**: MetaEd SHALL support multiple Ed-Fi Data Standard versions configurable via `projectVersion` and `defaultPluginTechVersion`. @@ -282,7 +282,7 @@ The product value is artifact consistency: SQL, XSD, API metadata, API schema, d ### Observability -- **NFR-OBS-1**: Build and deploy commands SHALL log major stages, plugin execution, artifact directory, deploy copy operations, warnings, and elapsed time. +- **NFR-OBS-1**: Default build and deploy runs SHALL log high-level progress, plugin execution, artifact directory, deploy copy operations, warnings, and elapsed time; more detailed stage tracing SHALL require debug logging. - **NFR-OBS-2**: Validation failures SHALL carry category, message, and source/file mapping where available. ### SDLC @@ -383,7 +383,7 @@ Generators return `GeneratedOutput` objects containing a human-readable name, na - The API Schema packaging workflow matrix uses `TPDM`, but the TPDM checkout step currently checks for lowercase `tpdm`, so that workflow step does not run for the TPDM matrix entries as written. - The API Catalog generator contains hard-coded handling for EducationOrganization and SchoolYear reference schemas because those reference schemas are not always discoverable from generated OpenAPI fragments. - The output writer's deletion guard is path-name based: it requires `MetaEdOutput` to appear in the output path before deleting an existing output directory. -- Alliance Mode enables editing of core files; non-Alliance users must leave this disabled to avoid inadvertent errors. +- Alliance Mode is an internal setting used by Ed-Fi Alliance workflows. In this repository it affects some validation, SQL header, and deploy behaviors; any IDE-side editability behavior is outside the scope of MetaEd-js. ## 7. Glossary @@ -410,6 +410,6 @@ Generators return `GeneratedOutput` objects containing a human-readable name, na | API Schema | JSON schema output generated as `ApiSchema*.json` for API schema consumers. | | Change Query | SQL artifacts for change version, tracked delete, and related change tracking support. | | Record Ownership | SQL artifacts that add ownership token support to aggregate root tables when enabled by target version. | -| Alliance Mode | A configuration setting that makes core model files editable; intended only for the Ed-Fi Alliance internal team. | +| Alliance Mode | An internal configuration setting used by Ed-Fi Alliance workflows. In MetaEd-js it affects selected validation, generation, and deploy behaviors; any IDE-side editability behavior is implemented outside this repository. | | SemVer | Semantic Versioning 2.0.0 — the versioning scheme used for project versions. | | Tech Version | The target ODS/API version (e.g., `6.0.0`) that determines which plugin behaviors and output formats to use. | diff --git a/packages/metaed-console/README.md b/packages/metaed-console/README.md index 3576de334..63803bb07 100644 --- a/packages/metaed-console/README.md +++ b/packages/metaed-console/README.md @@ -32,5 +32,6 @@ and generators in sequence. Logs timing information for each phase. 2. To confirm it is functional, try `node packages/metaed-console/dist/index.js -h`. 3. The easiest way to run this is with a config file. See `metaed-edfi-5.2.json` for a fully-worked sample config file. Note that this shows Alliance Mode _on_, which is only appropriate in the Ed-Fi Alliance's build - processes. External users should set this to `false`. To run with a config file: - `node ./dist/index.js -a -c ./metaed-edfi-5.2.json`. + processes. External users should set this to `false`. From the repo root, run with a config file using an absolute path + or a path that resolves from the console module directory, for example: + `node packages/metaed-console/dist/index.js -a -c packages/metaed-console/metaed-edfi-5.2.json`. diff --git a/packages/metaed-plugin-edfi-api-schema/README.md b/packages/metaed-plugin-edfi-api-schema/README.md index a9d81127b..fbde082dd 100644 --- a/packages/metaed-plugin-edfi-api-schema/README.md +++ b/packages/metaed-plugin-edfi-api-schema/README.md @@ -16,7 +16,7 @@ Registers two optional configuration schemas: Generates one JSON file per namespace: - `ApiSchema/ApiSchema.json` (for core) -- `ApiSchema/ApiSchema-{projectPrefix}.json` (for extensions) +- `ApiSchema/ApiSchema-{projectExtension}.json` (for extensions) ## Business Logic diff --git a/packages/metaed-plugin-edfi-unified/README.md b/packages/metaed-plugin-edfi-unified/README.md index ab94e249b..b3a8788aa 100644 --- a/packages/metaed-plugin-edfi-unified/README.md +++ b/packages/metaed-plugin-edfi-unified/README.md @@ -9,8 +9,8 @@ No plugin-specific configuration. ## Output -No file artifacts. This plugin only enriches the in-memory MetaEd model with unified -transformations used by other plugins (XSD, XML dictionary, ODS, etc.). +No file artifacts. This plugin validates the in-memory MetaEd model and enriches it with +unified transformations used by other plugins (XSD, XML dictionary, ODS, etc.). ## Business Logic