MC-10-Draft PRD-Ing-Scoring#3
Conversation
MC-10-Draft PRD Phase 2: for Ingestion and Scoring of Business Requirements.
Removed the report requirements. Listed processes more in line with the NACHOS-POC3 explainer. Still in draft mode
There was a problem hiding this comment.
Pull request overview
Adds a new Phase 2 PRD describing the Metadata Catalog’s ingestion pipeline and NACHOS scoring engine, including functional requirements, non-functional requirements/quality gates, architecture overview, and configuration/environment variables.
Changes:
- Defines JTBD 1 ingestion requirements: source management, entity/domain matching, business-logic extraction, scope classification, and storage schema fields.
- Defines JTBD 3 scoring requirements: LLM evidence record shape, rule cascade + override/rescoring behavior, and persistence/history expectations.
- Documents NFRs, quality gates, pipeline diagrams, key tables, backlog questions, and env var references.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * the **Ingestion Pipeline** (JTBD 1), which converts raw SEA artifacts (Swagger/OAS files and supplemental business documentation) into structured, element-level records. | ||
| * and the **Scoring Engine** (JTBD 3), which assigns NACHOS complexity scores to those records. | ||
|
|
||
| Access the following PRD for an overview of the overall Metadata Project. **Link To be added** |
There was a problem hiding this comment.
| Access the following PRD for an overview of the overall Metadata Project. **Link To be added** | |
| _See the [project PRD](docs/PRD.md) for additional background information on the Metadata Collection project._ |
| |---|---| | ||
| | **Ed-Fi Alliance Staff** | Primary operators of both pipelines; consumers of score reports and standardization analytics | | ||
|
|
||
| Excluded from this PRD are: Vendors and SEA Staff. They will be interested in read only capacity to some of the data. Vendors and SEA Staff may be given read access to some of the data in the future. SEA Staff may be given access to write a commitment form, and submit corrections to the scoring and business requirements. |
|
|
||
| Unmatched entities SHALL be flagged for staff review rather than silently dropped. Staff MAY resolve an unmatched entity by manually mapping it or marking it as a state-specific extension. | ||
|
|
||
| **Question: do we produce a report to pull elements that do not match other state definitions** |
|
|
||
| The system SHALL accept a second input form — separate from the Swagger input — requesting the path(s) to the state's supplemental **business documentation** (e.g., data dictionaries, collection guides, data submission manuals, annotated spreadsheets). | ||
|
|
||
| The system SHALL run process that operates at the **element level**, associating documented business rules to the specific API field or descriptor they govern. The extraction SHALL capture: |
| | `data_type` | WHether it is a descriptor, reference, string.. | | ||
| | `cardinality` | WHether the element is an identity, required or optional | | ||
| | `source` | WHether it is core or extension | |
| │ Scope Classification (in-scope: state-populated, vendor cannot │ | ||
| │ PUT/POST/DELETE; out-of-scope: state-populated, GET only) │ |
| #### 2.2.3 Logic alignment, rescoring and manual score ovewrite | ||
|
|
||
| * **Story** as an Ed-Fi ALliance Staff, I must be able to add a new business rule for a particular element or an entity, and re-run the LLM scoring, showing the old and new NACHOS and adjusted NACHOS scores, along with the rules that it triggers, and decide if the new scores are acceptable. | ||
| * | ||
| * **Story** Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the new scores and rationale recorded against the row's evidence record. |
| | `scope_confidence` | High / Medium / Low | | ||
| | `created_at` | Timestamp of record creation | | ||
|
|
||
| As a user I want to have a clear understanding of the similarity of the element to the ed-Fi data standard, and if the deviation to the core happened at element definition or in the business requirements to send such element, so that I can match the subsequent cascade rules and logic to the actual data |
MC-10 Simplified Ingestion logic by making few changes. Document is still in draft mode.
There was a problem hiding this comment.
Reviewed the draft specifically for places where a requirement, as written, would work against reusing the POC's validated approach — where building to the spec would mean re-architecting, re-validating, or discarding something the POC already proved out.
The notes below are the spots where I think the requirement and the proven approach actually pull against each other. A couple (the scope definition and the eval metrics) also affect whether the POC's existing validation numbers still describe the Phase-2 system, so they're worth settling before they harden into acceptance criteria.
| | `element_type` | Element type from the Ed-Fi Data Standard | | ||
| | `element_cardinality` | Keys, optional, required, or optional conditional | | ||
|
|
||
| #### 2.1.3 Business Logic Extraction IDENTIFIED AS "JBTD 3: Standardization of Data Collection Metadata " in Initial PRD |
There was a problem hiding this comment.
This works against the POC's proven structure rather than extending it. In POC-3, ingestion is deterministic parsing and the LLM-based extraction (verbatim spans + confidence) happens at the scoring stage — that split is deliberate and is what the validation ran on. Specifying an extraction LLM inside ingestion (and the matching INGESTION_LLM_* vars in §7.1) moves extraction to a different stage than the one that was validated. If extracting earlier is genuinely intended, worth calling it out as a re-architecture; otherwise this belongs under the scoring JTBD.
|
|
||
| **Story:** As a user, I want each element classified as in-scope or out-of-scope based on who populates it and what write operations are permitted, so that scoring and analysis focus on the elements that carry real implementation burden for vendors. | ||
|
|
||
| The system SHALL run a scope classification for each element in the Swagger using the following definitions: |
There was a problem hiding this comment.
This is the one I'd most want to resolve before it's locked in. Defining in-scope by HTTP verb (vendor can PUT/POST/DELETE) changes which elements get scored — and the POC's validation numbers were produced by scoring every Ed-Fi-mappable element as in-scope (in_scope is currently hardcoded true) and comparing against human reviewers on that basis. Adopting verb-based scope filtering would mean the POC's existing agreement/recall figures no longer describe the system, so we'd be re-validating from scratch. It's also an unproven methodology assumption: worth confirming that vendor write-access is actually the right proxy for implementation burden before it becomes the gating definition for what gets scored.
| | `cross_entity_logic` | Boolean — does the row involve lookups or constraints across entity boundaries? | | ||
| | `aggregation` | Boolean — does the row require computed aggregates or derived values? | | ||
| | `concatenation` | Boolean — does the row require string construction from multiple fields? | | ||
| | `semantic_divergence` | Boolean — does the state narrow, broaden, or redefine the canonical Ed-Fi meaning? | |
There was a problem hiding this comment.
A subtle one: collapsing semantic fidelity to a single boolean works against how the POC computes the adjusted score. In POC-3 semantic fidelity is a multi-value class (aligned / divergent-explained / divergent-unclear), and those tiers drive different adjustment magnitudes (e.g. +0.5 vs +1.0). A boolean can't carry that distinction, so the adjusted-NACHOS logic the POC validated couldn't be reproduced from this field. Suggest keeping it as a small enum rather than a bool.
| | `extension_necessity` | Boolean — does the element require a non-standard extension or descriptor override? | | ||
| | `cited_spans` | List of verbatim excerpts from source documentation that support the above flags | | ||
| | `nachos_score` | Base NACHOS scalar (integer) | | ||
| | `adjusted_nachos` | NACHOS score after semantic-fidelity adjustment (integer) | |
There was a problem hiding this comment.
Type conflict worth catching early: the POC's adjusted score is a float on a 0–4.5 scale — the adjustments are half-steps (+0.5 / +1.0) stacked on the 0–3 base. Typing adjusted_nachos as an integer forces rounding that discards the validated granularity (and partly collapses the base-vs-adjusted distinction). Suggest decimal/float.
| > * **Documentation Gap** — flag for complex structures the documentation under-explains | ||
|
|
||
| > [!NOTE] | ||
| > **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, an adjustment SHALL be applied to the base NACHOS score. Exact adjustment magnitudes are subject to Alliance methodology review and SHALL be configurable without a code deployment. |
There was a problem hiding this comment.
This requirement pulls against the POC's auditability model — and against the PRD's own §1.4 "auditability over automation" principle. In POC-3, every change to scoring (including adjustment magnitudes) bumps a scoring-plan version, so any stored score can be tied back to the exact methodology that produced it. "Configurable without a code deployment" removes that anchor: live-editable magnitudes mean a score can't be reliably reproduced or traced to a methodology version. Suggest framing config changes as versioned (a new plan version on change) rather than free-form, so configurability and auditability don't conflict. (Related: the cascade itself lives in versioned code today, not a YAML file (§7.2) — worth treating that externalization as a Phase-2 design choice, and pointing the authoritative-cascade reference at the versioned methodology rather than "page 11" of the PDF, which is already a stale snapshot.)
|
|
||
| | ID | Metric | Target | Notes | | ||
| |---|---|---|---| | ||
| | QG-1 | Score-tier alignment | >= 80% | Measured against the evaluation set seeded by Ed-Fi's existing manual scores, treated as human-scored observations, not ground truth | |
There was a problem hiding this comment.
Suggest pairing this aggregate gate with a per-tier recall floor. As written, an ≥80% aggregate can pass while regressing on exactly the dimension the POC found weakest: recall on the most-complex tier (tier 3) is roughly 23%, and 0% in at least one state. Since that's a known limitation, an aggregate-only gate that can't see it works against using the POC's own evaluation honestly — a high-tier recall floor keeps the bar meaningful where the method is weakest.
| | ID | Metric | Target | Notes | | ||
| |---|---|---|---| | ||
| | QG-1 | Score-tier alignment | >= 80% | Measured against the evaluation set seeded by Ed-Fi's existing manual scores, treated as human-scored observations, not ground truth | | ||
| | QG-2 | F1 (overall) | >= 0.7 | Calibration signal against evaluation set | |
There was a problem hiding this comment.
The F1 (QG-2/QG-3) and Cohen's Kappa (QG-4) gates assume a framing the POC's evaluation doesn't use. The POC treats the human scores as a single reviewer's observations — signal, not ground truth (which §3.2 itself states) — and reports agreement %, a tier confusion matrix, and tier-delta distributions. F1 needs a defined positive class and "truth"; kappa needs two independent raters. Neither fits cleanly here, and the POC computes neither. Suggest gating on what the validated method actually produces — agreement / confusion-matrix-based thresholds — unless we're deliberately introducing a new ground-truth set and a second rater.
stephenfuqua
left a comment
stephenfuqua
left a comment
There was a problem hiding this comment.
PRD Review: Clarity and Consistency with docs/PRD.md
Critical: JTBD Numbering Inverted vs. Parent PRD
docs/PRD.md defines:
- JTBD 1 = Scoring Engine
- JTBD 3 = Standardization of Data Collection Metadata
- JTBD 4 = Storage of API Specifications
- JTBD 6 = Enrichment of API Specifications
PRD-Ingestion-Scoring.md inverts this:
- Section 2.1 is labeled "JTBD 1 — Ingestion Pipeline" — but JTBD 1 in the parent PRD is the Scoring Engine. The ingestion pipeline maps to JTBD 3, 4, and 6.
- Section 2.2 is labeled "JTBD 3 — Scoring Engine" — but JTBD 3 in the parent PRD is Standardization of Data Collection Metadata. The Scoring Engine is JTBD 1.
- Section 2.3 is labeled "JTBD 3 — Score Evaluation Analytics" — reuses JTBD 3 again, which is already incorrect in 2.2.
The header annotations IDENTIFIED AS "JTBD 1: Scoring" in Initial PRD (lines 53, 101) acknowledge this mismatch but leave it unresolved as inline drafting notes.
Recommendation: Correct Section 2.1 to reference JTBD 3/4/6 and Section 2.2 to reference JTBD 1. Section 2.3 should use a new or TBD JTBD number.
Critical: NFR ID Collision with Parent PRD
docs/PRD.md defines NFR-DATA-1 as the production database weekly full backup requirement.
PRD-Ingestion-Scoring.md Section 3.1 redefines NFR-DATA-1 as the transactionality requirement (all ingestion and scoring runs SHALL be transactional — no partial writes).
Same ID, different requirements. NFR-DATA-2 and NFR-DATA-3 in the ingestion PRD appear to correspond correctly to their counterparts in the parent PRD, so those align — but NFR-DATA-1 is a collision. The ingestion PRD's transactionality NFR needs a new ID (e.g., NFR-DATA-5 or NFR-ING-1).
Moderate: Draft Artifacts Left in the Document
Several unresolved drafting comments remain in the text:
- Line 95 (table cell):
- CAN WE GET THIS?appended to thestandard_entitydescription. Either resolve it or move to the Backlog section (Section 6). - Line 224:
> [!IMPORTANT - NOT REALLY]— not valid GitHub-flavored Markdown callout syntax, and "NOT REALLY" is confusing. If the POC Recommendation reference is informational, use> [!NOTE]or plain text. - Line 53 (section header):
- IDENTIFIED AS "JTBD 1: Scoring" in Initial PRDshould be resolved or removed once the JTBD numbering is corrected. - Line 101 (section header): Same pattern:
IDENTIFIED AS "JBTD 3: Standardization of Data Collection Metadata"— also contains a typo: "JBTD" should be "JTBD".
Moderate: Out-of-Scope Section References Wrong JTBD
Section 5, line 373:
"JTBD 3 (Score Evaluation Analytics) — requirements TBD in a subsequent document"
In the parent PRD, JTBD 3 is Standardization of Data Collection Metadata, not Score Evaluation Analytics. This out-of-scope item should either reference the correct JTBD number or drop the JTBD label if Score Evaluation Analytics does not yet have one assigned.
Moderate: Section 3.1 Title Misleading
Section 3.1 is titled "Implemented Requirements" but this is a PRD for work not yet built. "Implemented" implies these NFRs have already been delivered. Consider renaming to "System NFRs" or "Pipeline NFRs."
Minor: Stray Markdown and Typo in Section 2.2.3
Lines 235–236 contain an empty bullet (* on its own line) that renders as a blank list item. Also "ALliance" is a typo.
Minor: [!NOTE: TO BE VALIDATED] Is Non-Standard Syntax
Line 301 in the technology stack table:
> [!NOTE: TO BE VALIDATED]
GitHub callout syntax is [!NOTE], [!WARNING], etc. — no colon or appended text. This won't render as a styled callout. Use > [!NOTE] with the validation caveat in the body, or a regular blockquote.
Minor: Duplicate NACHOS Context Note in Section 2.2.1
The NACHOS Score Context dimensions are defined twice — once in the bulleted list before the evidence record table (lines 176–183) and again in the [!NOTE] block afterward (lines 207–212). The second block is a subset of the first. Remove one to avoid ambiguity about which is authoritative.
Minor: POC Recommendation Document Has No Link or Filename
Section 2.2.2 references "the POC Recommendation document (page 11)" but provides no link, filename, or location. Readers have no way to locate it. Add a link or filename, or add it to the Backlog table (Section 6) as an open dependency.
Minor: In-Scope / Out-of-Scope Definitions Use Inconsistent Axes
Section 2.1.4:
- In-scope: vendor-accessible AND vendor can PUT/POST/DELETE
- Out-of-scope: "populated by the state" AND GET-only
The in-scope definition doesn't mention state population, but the out-of-scope definition requires it. An element that the state populates but also allows vendor writes would be unclassified. Either add a third case for shared write access or align both definitions to a single classification axis (e.g., vendor write access only).
Summary
| Severity | Issue | Location |
|---|---|---|
| Critical | JTBD numbering inverted vs. parent PRD | §2.1, §2.2, §2.3 |
| Critical | NFR-DATA-1 ID collision with parent PRD | §3.1 |
| Moderate | Draft artifacts not cleaned up | §2.1, §2.1.3, §2.2.2 |
| Moderate | Out-of-scope item references wrong JTBD | §5 |
| Moderate | Section 3.1 titled "Implemented Requirements" | §3.1 |
| Minor | Empty bullet and typo | §2.2.3 |
| Minor | Invalid callout syntax [!NOTE: TO BE VALIDATED] |
§4.1 |
| Minor | Duplicate NACHOS context note | §2.2.1 |
| Minor | POC Recommendation doc has no link/filename | §2.2.2 |
| Minor | In/out-of-scope definitions use different axes | §2.1.4 |
| "browsable" | ||
| "browsable", | ||
| "inspectable", | ||
| "qualty" |
There was a problem hiding this comment.
Oops, there's a misspelling in there 😁
| "qualty" |
There was a problem hiding this comment.
Please move this file to the docs directory
…July status) June revisions that never made it into the PR copy: PRD reference updated to the merged docs/PRD.md; Integration Profile renamed to the PRD's Score Context term; the D-1 schema-gap note against the ingestion PRD storage table (PR Ed-Fi-Alliance-OSS#3 s2.1.5); semantic-fidelity modeled as enum + rationale. July status updates (dated [2026-07 update] notes): the POC now encodes the operator runbook in code (poc3 publish stage registry + freshness manifest + fail-loudly readers - the hosted job wraps the registry); the s8.4 human-override capture half exists (workbook round-trip into committed curation records the repository migration must ingest); D-3 gained the consolidated deliverable surface and the workbook-as-write- surface constraint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
…S access notes - Renumber sections 0-9 to 1-10 to match the parent PRD convention (review: Minor) - Schema-type callout in section 3: adjusted NACHOS is 0-4.5 in half-steps (base tier is the 0-3 integer); PR Ed-Fi-Alliance-OSS#3 section 2.2.1 currently types both as integer - the adjusted field needs a decimal type (review: Critical) - TWEDS access reconciliation in section 5: the TWEDSAPI endpoint serves a server-rendered .NET app, not an XML/REST feed - fresh pulls are version-pinned portal scrapes with a headless browser for entity detail pages; folded into D-2 (review: Moderate; PR Ed-Fi-Alliance-OSS#3 section 2.1.1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
MC-10 Streamlined Ingestion process. Still in draft mode. Need to update the 4.2 Pipeline Overview
MC-10- streamlined PRD, removed the technology stack portion, and simplified the overall document.
MC-10 Typo Fixes in settings.json , and fixed typo errors in the PRD for Ingestion and Scoring
MC-10-Draft PRD Phase 2: for Ingestion and Scoring of Business Requirements.