Ed-Fi Implementation Usage Profile — POC (Tracks A + B)

.gitignore

@@ -0,0 +1,8 @@
+node_modules/
+dist/
+*.tsbuildinfo
+coverage/
+.DS_Store
+output/
+packages/edfi-profile-studio/dist/
+*.log

README.md

@@ -0,0 +1,121 @@
+# Ed-Fi Implementation Usage Profile — POC
+
+Two complementary tools that let an Ed-Fi implementer describe **which resources
+and elements of the ODS/API are actually in use** in their deployment.
+
+The canonical Ed-Fi model and OpenAPI document are **never mutated**. Usage
+annotations live in a **sidecar profile** that references a published Data
+Standard version, and a generator turns that profile into artifacts consumers
+can actually use.
+
+## Why this matters
+
+The Ed-Fi ODS/API OpenAPI document describes the **model surface** — every
+resource and property the Data Standard defines. It does not describe what a
+particular agency or vendor **actually populates**. Consumers (downstream apps,
+analytics teams, RFP respondents, and increasingly AI/agent contexts) routinely
+ask *"which fields can we count on?"* — and today the answer is tribal knowledge
+or a spreadsheet.
+
+This POC fills that gap with a descriptive, tooling-friendly artifact:
+
+- An LLM-driven agent hitting an Ed-Fi API benefits enormously from knowing that
+  an absent `citizenshipStatusDescriptor` is **policy, not missing data**.
+- A consumer integrating against a district's ODS gets a **stripped OpenAPI doc**
+  showing only what's real, ready to render in Swagger UI.
+- A district documenting conformance for an RFP gets a **coverage report** in
+  one command.
+
+## Two tracks, one shared core
+
+The person who *knows* what's in use (data architect, integration lead, district
+CIO, vendor PM) is rarely the person who works in MetaEd (data modeler, platform
+engineer). Forcing the former to learn the latter's toolchain kills adoption — so
+there are two tracks that converge on a single profile format and a single
+validation core.
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  Track B: Profile Studio (browser, no toolchain)                 │
+│  Load Swagger ─▶ Mark up resources/properties ─▶ Export profile  │
+└─────────────────────────────────┬────────────────────────────────┘
+                                   │  profile.yaml (shared format)
+┌─────────────────────────────────▼────────────────────────────────┐
+│  Track A: MetaEd Plugin + CLI Generator                           │
+│  Read profile ─▶ Validate ─▶ Emit overlay, stripped spec, report  │
+└───────────────────────────────────────────────────────────────────┘
+```
+
+`edfi-profile-core` is the **only** place profile semantics are defined; the
+plugin and the studio both depend on it, so a profile authored in the browser is
+byte-for-byte the kind a developer would hand-write.
+
+## Packages
+
+| Package | What it is |
+|---------|------------|
+| [`edfi-profile-core`](packages/edfi-profile-core) | Shared model, YAML parser/serializer, OpenAPI spec index, validator. |
+| [`metaed-plugin-edfi-implementation-profile`](packages/metaed-plugin-edfi-implementation-profile) | The three generators (overlay, stripped spec, coverage report), shaped as a MetaEd plugin. |
+| [`edfi-profile-cli`](packages/edfi-profile-cli) | `edfi-profile generate` / `validate` CLI. |
+| [`edfi-profile-studio`](packages/edfi-profile-studio) | Browser SPA for authoring profiles (Track B). |
+
+## Quick start (Track A)
+
+```bash
+npm install
+npm run build
+npm test
+
+# Generate all three artifacts for the worked example:
+node packages/edfi-profile-cli/dist/cli.js generate \
+  --profile examples/sample-lea/sample-lea.profile.yaml \
+  --base    examples/ed-fi-base/openapi-5.2.yaml \
+  --out-dir ./output
+
+# Or run the full pipeline + summary:
+./demo.sh
+```
+
+Outputs land in `./output/`:
+
+- `overlay.yaml` — OpenAPI Overlay 1.0, purely additive `x-edfi-usage` blocks.
+- `openapi-stripped.yaml` — a complete, valid OpenAPI 3.0 doc with
+  not-implemented paths and not-populated properties removed.
+- `coverage.json` / `coverage.md` — machine- and human-readable coverage summary.
+
+## Quick start (Track B)
+
+```bash
+npm install
+npm run build                 # builds the shared core the studio depends on
+npm run build:studio          # static SPA in packages/edfi-profile-studio/dist
+npm run dev --workspace edfi-profile-studio   # or run the dev server
+```
+
+Then: **Load bundled Ed-Fi 5.2 sample** → mark up resources/properties → **Export
+profile.yaml** → feed it to the Track A CLI above. See
+[docs/studio-walkthrough.md](docs/studio-walkthrough.md).
+
+## The three usage statuses that change output
+
+| Status | Stripped spec behavior |
+|--------|------------------------|
+| `in-use` / `partial` / `planned` | Property/path kept; `x-edfi-usage` attached; partial/planned get a description note. |
+| `not-populated` | Property removed from the schema (and from `required`). |
+| `not-implemented` | The whole resource path is removed. |
+
+## Documentation
+
+- [docs/profile-format.md](docs/profile-format.md) — the sidecar `.profile.yaml` format.
+- [docs/extension-conventions.md](docs/extension-conventions.md) — the `x-edfi-usage` field reference.
+- [docs/studio-walkthrough.md](docs/studio-walkthrough.md) — Track B workflow.
+- [docs/design-rationale.md](docs/design-rationale.md) — why sidecar, why overlay, and the relationship to Ed-Fi Profiles.
+
+## POC scope
+
+This is a proof of concept. Three resources (Student, StudentSchoolAssociation,
+StudentEducationOrganizationAssociation) are enough to prove the pattern. The
+base spec in `examples/ed-fi-base/` is a self-contained curated subset of the
+Ed-Fi Data Standard 5.2 OpenAPI document. See the handoff doc for what is
+explicitly out of scope (runtime enforcement, profile composition, npm publish,
+full Data Standard coverage).

demo.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+#
+# Runs the full Track A pipeline against the Sample LEA worked example and
+# prints the coverage report. Pass --open to also try to open the stripped
+# OpenAPI doc in Swagger UI (best effort; requires `npx`).
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$ROOT"
+
+PROFILE="examples/sample-lea/sample-lea.profile.yaml"
+BASE="examples/ed-fi-base/openapi-5.2.yaml"
+OUT="output"
+
+echo "==> Building packages"
+npm run build --silent
+
+echo "==> Validating profile against base spec"
+node packages/edfi-profile-cli/dist/cli.js validate --profile "$PROFILE" --base "$BASE" || true
+
+echo
+echo "==> Generating artifacts"
+node packages/edfi-profile-cli/dist/cli.js generate \
+  --profile "$PROFILE" --base "$BASE" --out-dir "$OUT"
+
+echo
+echo "==> Coverage report ($OUT/coverage.md)"
+echo "------------------------------------------------------------"
+cat "$OUT/coverage.md"
+echo "------------------------------------------------------------"
+
+if [[ "${1:-}" == "--open" ]]; then
+  echo
+  echo "==> Rendering the stripped spec in Swagger UI (Ctrl-C to stop)"
+  npx --yes swagger-ui-watcher "$OUT/openapi-stripped.yaml" || \
+    echo "Could not launch Swagger UI; open $OUT/openapi-stripped.yaml in editor.swagger.io instead."
+fi

docs/design-rationale.md

@@ -0,0 +1,89 @@
+# Design rationale
+
+## Why a sidecar, not a model change
+
+The Ed-Fi Data Standard and its generated OpenAPI document describe the *model
+surface* — everything that *could* be populated. That document is canonical and
+shared across the whole ecosystem; mutating it per-deployment would fork it and
+break the shared contract. So usage lives in a **sidecar profile** that
+references a published Data Standard version (`extends.dataStandard` +
+`extends.version`) and is applied by tooling. The canonical artifacts are never
+touched.
+
+## Why an OpenAPI Overlay
+
+The [OpenAPI Overlay Specification 1.0](https://github.com/OAI/Overlay-Specification)
+exists precisely for "decorate a base document without editing it." Emitting an
+overlay means:
+
+- the base spec and the usage annotations version independently;
+- anyone with the base spec + overlay can reconstruct the annotated view;
+- the mechanism is standard, so other Overlay-aware tooling can consume it.
+
+The overlay is **purely additive**. The opinionated, lossy transformation
+(removing things) is a separate artifact — the stripped spec — so the two
+concerns never get conflated.
+
+## Why a separate stripped spec
+
+Consumers don't want "the full model plus a side file telling them what to
+ignore." They want *the actual surface*. The stripped spec is a complete, valid
+OpenAPI 3.0 document that:
+
+- removes `not-implemented` resource paths entirely,
+- removes `not-populated` properties (and fixes up `required`),
+- keeps `x-edfi-usage` on what remains, and
+- augments descriptions so Swagger UI is useful with **zero** custom rendering.
+
+Validating it with `@apidevtools/swagger-parser` is a hard acceptance criterion —
+if it doesn't validate as OpenAPI 3.0, the POC has failed its core promise.
+
+## Relationship to Ed-Fi Profiles
+
+This is the question that will come up first, so it's worth being precise.
+
+| | **Ed-Fi Profiles (XML)** | **Implementation Usage Profile (this POC)** |
+|---|---|---|
+| Purpose | **Enforcement** — what an API client *may* see/write. | **Description** — what the implementation *actually populates*. |
+| Enforced at | ODS/API runtime. | Nowhere; it's documentation/tooling metadata. |
+| Audience | Security/integration admins. | Data architects, consumers, RFP authors, AI agents. |
+| Shape | XML, predates modern OpenAPI tooling. | YAML sidecar + OpenAPI overlay/stripped spec. |
+
+They are **complementary, not competing**. Profiles say "client X is *allowed* to
+see fields A, B, C." A usage profile says "this deployment *populates* fields A
+and B; C is policy-empty." Ideally one source of truth could eventually generate
+both — that's a roadmap conversation, not a POC goal.
+
+## Why two tracks instead of "just use MetaEd"
+
+MetaEd is the right surface for the *generation pipeline* and the wrong surface
+for the *authoring audience*. The person who knows what's in use — a district
+data architect, a vendor PM, a consultant doing an assessment — will not clone a
+repo and run a Node toolchain. Track B (the browser studio) meets them where they
+are; Track A (the plugin/CLI) meets the platform engineers where they are. Both
+depend on the same `edfi-profile-core`, so the two audiences can never produce
+incompatible profiles.
+
+That role separation is itself the pitch: *the person who knows isn't the person
+who codes.* It defuses the predictable "MetaEd already does this" objection by
+making clear MetaEd is the wrong **authoring** surface even where it's the right
+**generation** surface.
+
+## The AI/agent angle
+
+An LLM-driven agent querying an Ed-Fi API has no way to tell an intentionally
+empty field from a data-quality hole. `citizenshipStatusDescriptor: not-populated`
+with `reason: "Not collected per board policy 5.4"` turns a silent null into
+explicit, machine-readable policy. As agent-mediated data access grows, "this
+null is expected" vs. "this null is missing" becomes a first-class need — and a
+likely wedge for getting the broader ecosystem interested.
+
+## Open questions deferred past the POC
+
+- **DSL vs. YAML** for the profile format (POC ships YAML).
+- **Profile composition / inheritance** (an agency extending a vendor base) —
+  the `extends:` block leaves room for it.
+- **Descriptor/enum-value granularity** ("we only use 3 values of
+  `GradeLevelDescriptor`") — POC stays at the property level.
+- **OpenAPI 3.1** — Ed-Fi emits 3.0 today; the overlay spec works for both.
+- **Studio CORS proxy** — POC documents the upload workaround instead.

docs/extension-conventions.md

@@ -0,0 +1,79 @@
+# `x-edfi-usage` extension conventions
+
+The generators attach usage metadata to the OpenAPI surface using a single
+vendor extension key: **`x-edfi-usage`**. It appears in two artifacts:
+
+- the **overlay** (`overlay.yaml`), as the `update` payload of each action, and
+- the **stripped spec** (`openapi-stripped.yaml`), inline on operations and
+  schema properties so the metadata travels with the document.
+
+It is valid OpenAPI: the `x-` prefix marks it as a specification extension, so
+Swagger UI, codegen, and `swagger-parser` all tolerate it.
+
+## Vocabulary note: `usage` → `status`
+
+The authoring vocabulary in the profile YAML uses `usage:`. On the wire, inside
+an OpenAPI document, the block uses `status:` so the annotation reads naturally
+next to other OpenAPI fields. The mapping is 1:1; this is the only renaming.
+
+## Resource-level block
+
+Attached to every operation on a resource's collection path:
+
+```yaml
+x-edfi-usage:
+  status: in-use
+  populatedBy: ["Infinite Campus SIS"]
+  refreshCadence: nightly
+  coverage: 1.0
+  notes: "All enrolled K-12 students"
+  reason: "..."        # present when status is not-implemented
+```
+
+## Property-level block
+
+Attached to a schema property node:
+
+```yaml
+x-edfi-usage:
+  status: partial
+  coverage: 0.62
+  source: "SIS.Student.MiddleName"
+  requiredByImplementation: true
+  notes: "..."
+  reason: "..."        # present when status is not-populated
+```
+
+## Field presence
+
+Only fields the author supplied are emitted — there are no `null` placeholders.
+This keeps the overlay diff-friendly and the stripped spec clean.
+
+## Description augmentation (stripped spec only)
+
+To make Swagger UI immediately useful **without** custom rendering, the stripped
+spec also appends a short human-readable note to a property's `description`:
+
+| Status | Appended note |
+|--------|---------------|
+| `partial` | `⚠️ Partially populated: ~62% of records. <notes>` |
+| `planned` | `🕒 Planned — not yet populated. <notes>` |
+| `in-use` + `requiredByImplementation` | `✅ Required by this implementation.` |
+
+The original description is preserved; the note is appended.
+
+## Overlay targeting
+
+Overlay actions target nodes by JSONPath, per OpenAPI Overlay Specification 1.0:
+
+```yaml
+- target: "$.paths['/ed-fi/students'].get"
+  update:
+    x-edfi-usage: { status: in-use, ... }
+- target: "$.components.schemas.edFi_student.properties.middleName"
+  update:
+    x-edfi-usage: { status: partial, coverage: 0.62 }
+```
+
+The overlay is **purely additive** — it never contains `remove` actions. Removal
+is the stripped spec's job.