From 283a29a318164a9001720a520df237e6936ec272 Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Mon, 15 Jun 2026 14:42:26 -0500 Subject: [PATCH 1/6] MC-10-Draft PRD-Ing-Scoring MC-10-Draft PRD Phase 2: for Ingestion and Scoring of Business Requirements. --- PRD-Ingestion-Scoring.md | 477 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 477 insertions(+) create mode 100644 PRD-Ingestion-Scoring.md diff --git a/PRD-Ingestion-Scoring.md b/PRD-Ingestion-Scoring.md new file mode 100644 index 0000000..be2a41c --- /dev/null +++ b/PRD-Ingestion-Scoring.md @@ -0,0 +1,477 @@ +# Metadata Catalog — Ingestion & Scoring PRD + +> **Status:** Draft · **Last updated:** 2026-06-15 · **Owner:** Ed-Fi Alliance Staff + +--- + +## 1. Product Overview + +### 1.1 Background + +The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Two high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, and a **cluster analysis** that surfaces cross-state patterns. + +This PRD covers the phase 2 of automated ingestion and NACHOS scoring + +This document covers the two pipelines that make those workflows possible: + +* the **Ingestion Pipeline**, which converts raw SEA artifacts (Swagger/OAS files and supplemental business documentation) into structured, element-level records. +* and the **Scoring Engine**, which assigns NACHOS complexity scores to those records. + +Access the following PRD for an overview of the overall Metadata Project. **Link To be added** + +### 1.2 Problem Statement + +Vendor integration costs are a widely-cited barrier to expanding K-12 data access. Those costs stem from complex, aggregated, and inconsistent requirements across states. Without a systematic way to extract and score SEA specification metadata, Ed-Fi staff have no scalable method to: + +* Measure the complexity of a state's data collection requirements. +* Compare complexity across states. +* Identify which specific elements drive implementation burden. +* Prioritize standardization conversations with the highest-leverage states. + +Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. An LLM-assisted pipeline that produces auditable, human-overridable scores addresses all four gaps. + +### 1.3 Target Users for this PRD + +| User | Role | +|---|---| +| **Ed-Fi Alliance Staff** | Primary operators of both pipelines; consumers of score reports and standardization analytics | + +Excluded from this PRD are: Vendors and SEA Staff. They will be interested in read only capacity to some of the data. Vendors and SEA Staff may be given read access to some of the data in the future. SEA Staff may be given access to write a commitment form, and submit corrections to the scoring and business requirements. + +### 1.4 Design Principles + +**Auditability over automation.** Every score is traceable to an evidence record. A score without a traceable rationale is not acceptable. + +**Human override is a feature, not an exception.** Staff can override any machine-assigned score. Overrides and their rationale are persisted alongside the original evidence record. + +**Disagreement is signal.** Where the engine and a human reviewer assign different tiers, those rows are routed for review — they may surface a methodology question, not just an engine error. They are not silently tuned away to match one reviewer. + +**Distributions over rankings.** Cross-state comparisons SHALL report Score Context distributions (Implementation Shape, Documentation Style, Documentation Gap), not a single-scalar ranking that can misread documentation culture as quality. + +**No student data, no PII.** The system is scoped exclusively to specification metadata. No student data or personally identifiable information flows through any component at any time. + +--- + +## 2. Functional Requirements + +### 2.1 JTBD 1 — Ingestion Pipeline - IDENTIFIED AS "JTBD 1: Scoring" in Initial PRD + +**Story:** As Ed-Fi Alliance Staff, I want to load a state's Swagger/OAS specification and supplemental business documentation into a structured catalog, so that I have clean, element-level records ready for scoring and analysis. + +#### 2.1.1 Source Input Collection + +The system SHALL present an input form that accepts one or both of the following source types for the state's API specification: + +| Input Type | Description | +|---|---| +| **File upload** | One or more Swagger/OAS files (`.json` or `.yaml`) uploaded directly | +| **URL** | A publicly accessible URL pointing to a Swagger/OAS document | + +Possible formats: + +* URL to a webpage +* PDF files +* Excel files or worksheets, with multiple tabs +* URL to a webpage that is in Confluence format. +* Excel files with inside links to other webpages. + +**TBD: Does the ingestion UML need to know the format type for these sources. Have we seen word documents?** + +The system SHALL accept up to **10** total paths or files per ingestion run across both source types combined. + +The system SHALL validate that each provided file or URL resolves to a parseable Swagger/OAS document before proceeding. If a source fails validation the system SHALL report the failure and allow the user to correct or remove the invalid entry without re-submitting valid sources. + +> [!NOTE: DOES THIS LIMIT HELP CODING?] +> The 10-source limit applies to a single ingestion run. A state with more than 10 specification artifacts should be broken into multiple runs or the artifacts consolidated before upload. + +Columns resulting the end of this process: + +| Column | Description | +|---|---| +| `entity_name` | Entity name as it appears in the state's Swagger | +| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | +| `domain` | Ed-Fi domain (e.g., Student, Enrollment, Assessment) | +| `match_confidence` | High / Medium / Low / Unmatched | +| `match_notes` | Flags for extension entities, renamed entities, or descriptor overloads | +| `element_name` | element name from the state definition | +| `element_type` | entity name from the Ed-Fi Data Standard | +| `element_cardinality` | identify keys, optional, required, optional conditional | + +> [DO WE INCLUDE THESE?] +| `Ed-FI_element_name` | Best-match entity name from the state definition | +| `Ed-FI_element_type` | Best-match entity name from the Ed-Fi Data Standard | +| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | + +#### 2.1.2 Entity Matching to Standard Domain + +After parsing the Swagger/OAS document(s), the system SHALL match each API entity (resource, descriptor, association) to its corresponding **Ed-Fi Data Standard domain definition** using a standard domain registry. + +The system SHALL produce a match report with the following columns at minimum: + +| Column | Description | +|---|---| +| `entity_name` | Entity name as it appears in the state's Swagger | +| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | +| `domain` | Ed-Fi domain (e.g., Student, Enrollment, Assessment) | +| `match_confidence` | High / Medium / Low / Unmatched | +| `match_notes` | Flags for extension entities, renamed entities, or descriptor overloads | + +Unmatched entities SHALL be flagged for staff review rather than silently dropped. Staff MAY resolve an unmatched entity by manually mapping it or marking it as a state-specific extension. + +**Question: do we produce a report to pull elements that do not match other state definitions** + +#### 2.1.3 Business Logic Extraction IDENTIFIED AS "JBTD 3: Standardization of Data Collection Metadata " in Initial PRD + +**Story:** As Ed-Fi Alliance Staff, I want to extract data collection requirements from any SEA and store them in a standardized manner, so that I can analyze and help improve the SEA's requirements. + +**Acceptance Criteria:** + +* Standardized metadata storage with a schema that can represent normalized metadata plus provenance and curation status +* The standardized repository SHALL preserve provenance and curation status for each material metadata element, including whether it came from exact artifact import, inexact supplemental extraction, or manual editing. +* Phase 1 minimum acceptance: Ed-Fi Alliance Staff MUST be able to load, store, and manually curate standardized metadata in that repository even before any scoring or other downstream analytical consumer is delivered. +* Ed-Fi Alliance Staff MUST be able to manually curate standardized metadata regardless of whether it originated from exact import or inexact extraction. +* Phase 2 downstream-consumer readiness: the standardized repository SHALL expose standardized metadata programmatically to the scoring engine and other downstream use cases once those consumers are introduced. +* The schema SHALL be at the **(resource, property) grain** and SHALL map state documentation onto the **canonical Ed-Fi Data Standard surface**, so a source lens and an API-model (coverage) lens are possible. +* The repository SHALL retain the **source narrative text** (or stable, normalized references) sufficient to validate and display _cited spans_; a cited span cannot be shown against source text that was not retained. (The Phase-1 schema should anticipate this.) + +The system SHALL accept a second input form — separate from the Swagger input — requesting the path(s) to the state's supplemental **business documentation** (e.g., data dictionaries, collection guides, data submission manuals, annotated spreadsheets). + +This form SHALL accept up to **10** total file paths or URLs. Supported formats include PDF, Word (.docx), Excel (.xlsx), plain text, and HTML. + +The system SHALL run an LLM extraction process that operates at the **element level**, associating documented business rules to the specific API field or descriptor they govern. The extraction SHALL capture: + +* The element name (field path within the entity) +* The extracted business logic text (verbatim quoted span from the source document) +* The source document and page/section reference (cited span) +* A confidence score for the extraction + +> [!NOTE] +> The extraction LLM operates on documentation content only. It does not access student data or any data source beyond the files and URLs provided in this form. + +#### 2.1.4 Scope Classification + +The system SHALL run a scope classification process that determines, for each element in the Swagger, whether it is **in-scope** or **out-of-scope** for the state's data collection. + +An element is classified as in-scope if the state's Swagger or business documentation explicitly requires, uses, or references it. Elements present in the Ed-Fi Data Standard but absent from — or explicitly excluded by — the state's artifacts are classified as out-of-scope. + +The classification process SHALL emit a confidence level (High / Medium / Low) and a brief rationale for each out-of-scope classification. Staff MAY override any classification. + +#### 2.1.5 Ingestion Summary Report + +Upon completion of an ingestion run the system SHALL display an ingestion summary report. The report SHALL include, at minimum: + +| Metric | Description | +|---|---| +| Total elements retrieved | Count of all element-level records parsed from the Swagger source(s) | +| Elements with business logic | Count of elements for which at least one business rule was extracted | +| Elements in scope | Count of elements classified as in-scope | +| Elements out of scope | Count of elements classified as out-of-scope | +| Unmatched entities | Count of entities that could not be matched to the Ed-Fi Data Standard | +| Extraction confidence distribution | Breakdown of High / Medium / Low confidence extractions | + +The report SHALL be exportable as CSV and as a rendered HTML table. + +#### 2.1.6 Persistence + +After staff review and approval of an ingestion run, the system SHALL write the following data to the storage database for each element: + +| Field | Description | +|---|---| +| `run_id` | Unique identifier for the ingestion run | +| `state_id` | State identifier | +| `entity_name` | Entity name from the state's Swagger | +| `element_path` | Full field path within the entity | +| `standard_entity` | Matched Ed-Fi Data Standard entity | +| `domain` | Ed-Fi domain | +| `business_logic` | Extracted business rule text (null if none) | +| `cited_span` | Source document reference for the business logic | +| `in_scope` | Boolean scope flag | +| `scope_confidence` | High / Medium / Low | +| `created_at` | Timestamp of record creation | +| `run_approved_by` | Staff member who approved the run | + +The write operation SHALL be transactional. A partially written run SHALL be rolled back in full, not partially committed. + +--- + +### 2.2 JTBD 2 — Scoring Engine - ALSO IDENTIFIED AS "JTBD1: Scoring Engine" in Initial PRD + +**Story:** As Ed-Fi Alliance Staff — on behalf of Vendors — I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. + +#### 2.2.1 Scoring LLM Execution + +**Story:** As Ed-Fi Alliance Staff — on behalf of Vendors — I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. + +**Acceptance Criteria:** + +* Ed-Fi Alliance's existing manual scoring SHALL seed the evaluation set, treated as **human-scored observations** (used to measure agreement), not infallible ground truth. Before formal acceptance, Ed-Fi SHALL define the evaluation set and SHOULD grow it — ideally with more than one reviewer — so acceptance testing is credible. +* >= 80% score-tier alignment on rows scored by both the engine and a reviewer. +* >= 0.7 F1 overall and >= 0.9 F1 for "simple" (score-zero) rows, measured against the evaluation set as a calibration signal (not proof of correctness). +* >= 0.6 Cohen's Kappa (inter-rater agreement). +* Where the engine and a reviewer disagree on tier, treat those rows as the calibration target — they may point to a methodology question, not just an engine error, so route them to review rather than tuning them away to match one reviewer. +* Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the override and rationale recorded against the row's evidence record. +* **NACHOS Score Context (peer signal).** Every scored row SHALL carry, beside the scalar: _Implementation Shape_ (structural complexity, deterministic from the Swagger/API model), _Documentation Style_ (Prescriptive | Conceptual | Cross-reference | Regulatory | Unspecified), and _Documentation Gap_ (complex structure the documentation under-explains). +* **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, apply an adjustment (exact magnitudes subject to Alliance methodology review). +* **Evidence record (per row).** Every score SHALL emit a cached evidence record (extracted facts, confidence, cited spans, firing rule path, review routing) — extending the provenance model (NFR-DATA-2/3) to the scoring layer, so a score is inspectable without re-running the model. +* Cross-state outputs SHALL report Score Context **distributions**, not a single-scalar ranking that can misread documentation culture as quality. + +The system SHALL run a Scoring LLM against each in-scope element record produced by the Ingestion Pipeline. For each element the LLM SHALL extract and emit an **evidence record** containing, at minimum: + +| Field | Description | +|---|---| +| `element_path` | Full field path within the entity | +| `conditional_logic` | Boolean — does the row describe conditional or branching logic? | +| `cross_entity_logic` | Boolean — does the row involve lookups or constraints across entity boundaries? | +| `aggregation` | Boolean — does the row require computed aggregates or derived values? | +| `concatenation` | Boolean — does the row require string construction from multiple fields? | +| `semantic_divergence` | Boolean — does the state narrow, broaden, or redefine the canonical Ed-Fi meaning? | +| `documentation_style` | `Prescriptive` \| `Conceptual` \| `Cross-reference` \| `Regulatory` \| `Unspecified` | +| `extension_necessity` | Boolean — does the element require a non-standard extension or descriptor override? | +| `cited_spans` | List of verbatim excerpts from source documentation that support the above flags | +| `nachos_score` | Base NACHOS scalar (integer) | +| `adjusted_nachos` | NACHOS score after semantic-fidelity adjustment (integer) | +| `adjustment_rationale` | Plain-text explanation of any adjustment applied | +| `candidate_recommendations` | List of suggested simplification or standardization actions | +| `firing_rule_path` | The ordered list of rules from the rule cascade that produced this score | +| `confidence` | Overall confidence in the score assignment (High / Medium / Low) | +| `review_routing` | `Auto-accept` \| `Route-for-review` (see §2.2.2) | + +> [!NOTE] +> **NACHOS Score Context (peer signal).** Every scored row SHALL carry, alongside the scalar score: +> +> * **Implementation Shape** — structural complexity, deterministic from the Swagger/API model +> * **Documentation Style** — one of: Prescriptive, Conceptual, Cross-reference, Regulatory, Unspecified +> * **Documentation Gap** — flag for complex structures the documentation under-explains + +> [!NOTE] +> **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, an adjustment SHALL be applied to the base NACHOS score. Exact adjustment magnitudes are subject to Alliance methodology review and SHALL be configurable without a code deployment. + +#### 2.2.2 Rule Cascade + +The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` field in the evidence record. + +> [!IMPORTANT] +> Reference: _POC Recommendation document, page 11 — Rule Cascade table_. The rule cascade is the authoritative source for score assignment logic. Any discrepancy between this PRD and that document should be resolved in favor of the POC Recommendation document pending a formal methodology review. + +Staff SHALL be able to inspect the full `firing_rule_path` for any scored element to understand exactly which rules fired. + +#### 2.2.3 Score Reports + +Upon completion of a scoring run the system SHALL produce a tabular score report. The report SHALL include, at minimum, one row per scored element with the following columns: + +| Column | Description | +|---|---| +| `entity_name` | Entity name | +| `element_path` | Full field path | +| `domain` | Ed-Fi domain | +| `nachos_score` | Base NACHOS scalar | +| `adjusted_nachos` | Adjusted NACHOS scalar | +| `documentation_style` | Documentation style classification | +| `implementation_shape` | Structural complexity flag | +| `documentation_gap` | Boolean | +| `review_routing` | Auto-accept or Route-for-review | +| `has_override` | Boolean — has this row been manually overridden by staff? | + +The report SHALL be sortable and filterable by score, adjusted score, domain, documentation style, and review routing. It SHALL be exportable as CSV. + +Cross-state score reports SHALL report **Score Context distributions** (counts and percentages by Documentation Style, Implementation Shape tier, and Documentation Gap rate) rather than a single aggregate ranking. + +#### 2.2.4 Score Persistence + +After a scoring run is approved by staff, the system SHALL write scores to the storage database, extending each element record with the following fields: + +| Field | Description | +|---|---| +| `scoring_run_id` | Unique identifier for the scoring run | +| `nachos_score` | Base NACHOS scalar | +| `adjusted_nachos` | Adjusted NACHOS scalar | +| `implementation_shape` | Structural complexity classification | +| `documentation_style` | Documentation style classification | +| `documentation_gap` | Boolean | +| `evidence_record` | Full JSON evidence record (all LLM-extracted fields) | +| `firing_rule_path` | Ordered rule list that produced the score | +| `review_routing` | Auto-accept or Route-for-review | +| `override_score` | Staff-assigned score (null if not overridden) | +| `override_rationale` | Staff-provided rationale for override (null if not overridden) | +| `override_by` | Staff member who applied the override (null if not overridden) | +| `scored_at` | Timestamp of score assignment | + +The evidence record SHALL be stored in full so that any score is inspectable without re-running the model. + +#### 2.2.5 Scored Element Forms + +The system SHALL produce two filtered views (exportable as printable/shareable forms) from a completed scoring run: + +**Form A — Elements Requiring Review** (`adjusted_nachos > 0`) + +All elements where the Adjusted NACHOS score is greater than zero, sorted by adjusted score descending. This form surfaces elements that carry implementation complexity and are candidates for standardization conversations with the SEA. + +**Form B — Simple Elements** (`adjusted_nachos = 0`) + +All elements where the Adjusted NACHOS score equals zero. These elements are assessed as straightforward implementations with no deviation from the canonical Ed-Fi definition. A high F1 score on this form (target >= 0.9) is a key quality signal. + +Both forms SHALL display, at minimum: entity name, element path, domain, base NACHOS score, adjusted NACHOS score, documentation style, and any candidate recommendations. + +--- + +### 2.3 JTBD 3 — Score Evaluation Analytics _(TBD)_ + +Analysis of aggregate score data across states and across time is scoped for a future phase. Requirements will be defined separately. + +--- + +## 3. Non-Functional Requirements + +### 3.1 Implemented Requirements + +| ID | Category | Requirement | Implementation | +|---|---|---|---| +| NFR-PERF-1 | Performance | Ingestion of a single-state Swagger file (up to 500 endpoints) SHALL complete within 5 minutes | Async job queue; progress indicator displayed to user | +| NFR-PERF-2 | Performance | Scoring run for up to 2,000 elements SHALL complete within 15 minutes | Batched LLM calls; parallel processing where element order permits | +| NFR-DATA-1 | Data Integrity | All ingestion and scoring runs SHALL be transactional — no partial writes | Database transaction wrapping each run | +| NFR-DATA-2 | Provenance | Every ingestion record SHALL carry a `run_id`, `state_id`, and `created_at` timestamp | Applied at write time | +| NFR-DATA-3 | Provenance | Every score SHALL carry a full evidence record persisted alongside the scalar | `evidence_record` JSON column on score table | +| NFR-SEC-1 | Security | No student data or PII SHALL flow through any pipeline component | Input validation at ingestion form; static analysis gate in CI | +| NFR-ACC-1 | Accessibility | All reports and forms SHALL meet WCAG 2.1 AA | Verified with automated accessibility checker in CI | + +### 3.2 Quality Gates (Scoring Engine) + +| ID | Metric | Target | Notes | +|---|---|---|---| +| QG-1 | Score-tier alignment | >= 80% | Measured against the evaluation set seeded by Ed-Fi's existing manual scores, treated as human-scored observations, not ground truth | +| QG-2 | F1 (overall) | >= 0.7 | Calibration signal against evaluation set | +| QG-3 | F1 (simple / score-zero rows) | >= 0.9 | High precision on simple rows is critical — false positives here create unnecessary review burden | +| QG-4 | Cohen's Kappa | >= 0.6 | Inter-rater agreement between engine and human reviewer | +| QG-5 | Disagreement routing | 100% of tier disagreements routed for review | Disagreements are treated as calibration signal, not auto-resolved | + +> [!NOTE] +> Before formal acceptance testing, Ed-Fi Alliance SHALL define the composition of the evaluation set and SHOULD grow it — ideally with more than one reviewer — so that acceptance metrics are credible. The evaluation set is a living artifact, not a one-time snapshot. + +--- + +## 4. Architecture + +### 4.1 Technology Stack + +> [!NOTE: TO BE VALIDATED] +| Component | Technology | Notes | +|---|---|---| +| Ingestion LLM | Claude (Anthropic) | Configurable model; defaults to claude-sonnet-4-6 | +| Scoring LLM | Claude (Anthropic) | Same model pool; scoring prompt is distinct from ingestion prompt | +| Storage | PostgreSQL | Element catalog, evidence records, score tables | +| Job Queue | To be determined | Async execution of ingestion and scoring runs | +| API layer | To be determined | REST endpoints for form submissions, run status, report export | +| Frontend | To be determined | Input forms, progress views, report tables, scored element forms | + +### 4.2 Pipeline Overview + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ INGESTION PIPELINE │ +│ │ +│ [User: Swagger file/URL] │ +│ │ │ +│ ▼ │ +│ Parse & Validate OAS ──► Entity Match to Ed-Fi Domain Registry │ +│ │ │ +│ [User: Business doc paths/URLs] │ +│ │ │ +│ ▼ │ +│ Business Logic Extraction LLM (element-level, with cited spans) │ +│ │ │ +│ ▼ │ +│ Scope Classification ──► Ingestion Summary Report │ +│ │ │ +│ ▼ │ +│ [Staff review & approve] │ +│ │ │ +│ ▼ │ +│ Write to Storage DB (elements + business logic + scope flags) │ +└─────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────┐ +│ SCORING ENGINE │ +│ │ +│ Read in-scope elements from Storage DB │ +│ │ │ +│ ▼ │ +│ Scoring LLM ──► Evidence Record (flags, cited spans, confidence) │ +│ │ │ +│ ▼ │ +│ Rule Cascade ──► NACHOS Score + Adjusted NACHOS │ +│ │ │ +│ ▼ │ +│ Score Report (tabular) + Form A (adj > 0) + Form B (adj = 0) │ +│ │ │ +│ ▼ │ +│ [Staff review; override where needed] │ +│ │ │ +│ ▼ │ +│ Write scores + evidence records to Storage DB │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### 4.3 Data Model (Key Tables) + +| Table | Primary Key | Key Columns | +|---|---|---| +| `ingestion_runs` | `run_id` | `state_id`, `created_at`, `approved_by`, `status` | +| `elements` | `element_id` | `run_id`, `state_id`, `entity_name`, `element_path`, `standard_entity`, `domain`, `in_scope`, `business_logic`, `cited_span` | +| `scoring_runs` | `scoring_run_id` | `run_id` (FK → ingestion_runs), `created_at`, `approved_by`, `status` | +| `scores` | `score_id` | `element_id` (FK → elements), `scoring_run_id`, `nachos_score`, `adjusted_nachos`, `evidence_record` (JSON), `override_score`, `override_by` | + +--- + +## 5. Out of Scope + +The following are explicitly out of scope for this PRD: + +* JTBD 3 (Score Evaluation Analytics) — requirements TBD in a subsequent document +* Student data of any kind +* Real-time or webhook-based specification monitoring (batch ingestion only) +* Direct SEA portal access or automated fetching from SEA systems +* Vendor-facing score exposure (internal Ed-Fi staff use only in this phase) +* Natural language query interface (separate capability in the broader initiative) +* Cluster analysis across states (separate capability) + +--- + +## 6. Backlog + +Active issues are tracked in the project issue tracker. The following items are known open questions as of this draft: + +| ID | Area | Question | Status | +|---|---|---|---| +| TBD | Scoring | Exact NACHOS adjustment magnitudes for semantic divergence cases | Pending Alliance methodology review | +| TBD | Ingestion | Handling of multi-version Swagger files for the same state | Open | +| TBD | Scoring | Evaluation set composition and reviewer assignment | Ed-Fi SHALL define before acceptance testing | +| TBD | Architecture | Job queue technology selection | Open | +| TBD | Reporting | Format and delivery mechanism for cross-state distribution reports | Open | + +--- + +## 7. Environment Variable Reference + +### 7.1 Ingestion Pipeline + +| Variable | Description | Default | +|---|---|---| +| `INGESTION_LLM_MODEL` | Claude model ID for business logic extraction | `claude-sonnet-4-6` | +| `INGESTION_LLM_MAX_TOKENS` | Max tokens per extraction call | `4096` | +| `INGESTION_MAX_SOURCES` | Maximum files/URLs per ingestion run | `10` | +| `INGESTION_SUPPORTED_DOC_FORMATS` | Comma-separated list of accepted doc extensions | `pdf,docx,xlsx,txt,html` | +| `DB_CONNECTION_STRING` | PostgreSQL connection string | _(required)_ | + +### 7.2 Scoring Engine + +| Variable | Description | Default | +|---|---|---| +| `SCORING_LLM_MODEL` | Claude model ID for scoring | `claude-sonnet-4-6` | +| `SCORING_LLM_MAX_TOKENS` | Max tokens per scoring call | `4096` | +| `SCORING_BATCH_SIZE` | Elements per LLM batch call | `25` | +| `NACHOS_ADJUSTMENT_CONFIG` | Path to YAML file defining semantic-fidelity adjustment magnitudes | `config/nachos-adjustments.yaml` | +| `RULE_CASCADE_CONFIG` | Path to YAML file defining the rule cascade | `config/rule-cascade.yaml` | +| `QG_TIER_ALIGNMENT_THRESHOLD` | Score-tier alignment quality gate threshold | `0.80` | +| `QG_F1_OVERALL_THRESHOLD` | Overall F1 quality gate threshold | `0.70` | +| `QG_F1_SIMPLE_THRESHOLD` | F1 quality gate threshold for score-zero rows | `0.90` | +| `QG_KAPPA_THRESHOLD` | Cohen's Kappa quality gate threshold | `0.60` | From d1747e3cc02807384e4fd6f30284b93a995a147d Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Thu, 18 Jun 2026 18:05:58 -0500 Subject: [PATCH 2/6] MC-10 Updated PRD Removed the report requirements. Listed processes more in line with the NACHOS-POC3 explainer. Still in draft mode --- PRD-Ingestion-Scoring.md | 235 ++++++++++++++++----------------------- 1 file changed, 95 insertions(+), 140 deletions(-) diff --git a/PRD-Ingestion-Scoring.md b/PRD-Ingestion-Scoring.md index be2a41c..3e49288 100644 --- a/PRD-Ingestion-Scoring.md +++ b/PRD-Ingestion-Scoring.md @@ -1,6 +1,6 @@ # Metadata Catalog — Ingestion & Scoring PRD -> **Status:** Draft · **Last updated:** 2026-06-15 · **Owner:** Ed-Fi Alliance Staff +> **Status:** Draft · **Last updated:** 2026-06-18 · **Owner:** Ed-Fi Alliance Staff --- @@ -10,12 +10,12 @@ The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Two high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, and a **cluster analysis** that surfaces cross-state patterns. -This PRD covers the phase 2 of automated ingestion and NACHOS scoring +This PRD covers phase 2 of automated ingestion and NACHOS scoring. This document covers the two pipelines that make those workflows possible: -* the **Ingestion Pipeline**, which converts raw SEA artifacts (Swagger/OAS files and supplemental business documentation) into structured, element-level records. -* and the **Scoring Engine**, which assigns NACHOS complexity scores to those records. +* the **Ingestion Pipeline** (JTBD 1), which converts raw SEA artifacts (Swagger/OAS files and supplemental business documentation) into structured, element-level records. +* and the **Scoring Engine** (JTBD 3), which assigns NACHOS complexity scores to those records. Access the following PRD for an overview of the overall Metadata Project. **Link To be added** @@ -44,10 +44,6 @@ Excluded from this PRD are: Vendors and SEA Staff. They will be interested in r **Human override is a feature, not an exception.** Staff can override any machine-assigned score. Overrides and their rationale are persisted alongside the original evidence record. -**Disagreement is signal.** Where the engine and a human reviewer assign different tiers, those rows are routed for review — they may surface a methodology question, not just an engine error. They are not silently tuned away to match one reviewer. - -**Distributions over rankings.** Cross-state comparisons SHALL report Score Context distributions (Implementation Shape, Documentation Style, Documentation Gap), not a single-scalar ranking that can misread documentation culture as quality. - **No student data, no PII.** The system is scoped exclusively to specification metadata. No student data or personally identifiable information flows through any component at any time. --- @@ -56,35 +52,28 @@ Excluded from this PRD are: Vendors and SEA Staff. They will be interested in r ### 2.1 JTBD 1 — Ingestion Pipeline - IDENTIFIED AS "JTBD 1: Scoring" in Initial PRD -**Story:** As Ed-Fi Alliance Staff, I want to load a state's Swagger/OAS specification and supplemental business documentation into a structured catalog, so that I have clean, element-level records ready for scoring and analysis. +**Story:** As Ed-Fi Alliance Staff, I want to load a state's Swagger/OAS specification and supplemental business documentation into a structured catalog with one row per attribute and the state's specifications, so that I have clean, element-level records with business requirements and standard domain mapping ready for scoring and analysis. #### 2.1.1 Source Input Collection -The system SHALL present an input form that accepts one or both of the following source types for the state's API specification: - -| Input Type | Description | -|---|---| -| **File upload** | One or more Swagger/OAS files (`.json` or `.yaml`) uploaded directly | -| **URL** | A publicly accessible URL pointing to a Swagger/OAS document | - -Possible formats: - -* URL to a webpage -* PDF files -* Excel files or worksheets, with multiple tabs -* URL to a webpage that is in Confluence format. -* Excel files with inside links to other webpages. +**Story:** As a user, I want to view the list of sources used to read business logic and be able to edit those sources — change a path, add more files, or remove an entry — so that I can manage the inputs driving the ingestion without restarting the process from scratch. -**TBD: Does the ingestion UML need to know the format type for these sources. Have we seen word documents?** +The system SHALL display the current list of configured source paths and URLs before and after an ingestion run, and SHALL allow a user to add, edit, or remove any entry from that list. -The system SHALL accept up to **10** total paths or files per ingestion run across both source types combined. +The system SHALL support the following source types for supplemental business documentation: -The system SHALL validate that each provided file or URL resolves to a parseable Swagger/OAS document before proceeding. If a source fails validation the system SHALL report the failure and allow the user to correct or remove the invalid entry without re-submitting valid sources. +| Source Type | Description | +|---|---| +| **Confluence page** | A URL pointing to a Confluence page | +| **GitHub + Excel matrices** | A GitHub repository path containing Excel-based data matrices | +| **TWEDS** | Versioned XML artifacts or REST API access to the Texas Web-Enabled Data Standards | +| **PDF** | PDF documents such as data dictionaries or submission manuals | +| **Excel** | `.xlsx` files or worksheets, including those with multiple tabs or internal links to other resources | +| **HTML / URL** | A publicly accessible webpage | -> [!NOTE: DOES THIS LIMIT HELP CODING?] -> The 10-source limit applies to a single ingestion run. A state with more than 10 specification artifacts should be broken into multiple runs or the artifacts consolidated before upload. +The system SHALL validate that each provided file or URL is resolvable before proceeding. If a source fails validation, the system SHALL report the failure and allow the user to correct or remove the invalid entry without re-submitting valid sources. -Columns resulting the end of this process: +The output of the ingestion process SHALL contain one row per attribute with the following columns: | Column | Description | |---|---| @@ -93,20 +82,20 @@ Columns resulting the end of this process: | `domain` | Ed-Fi domain (e.g., Student, Enrollment, Assessment) | | `match_confidence` | High / Medium / Low / Unmatched | | `match_notes` | Flags for extension entities, renamed entities, or descriptor overloads | -| `element_name` | element name from the state definition | -| `element_type` | entity name from the Ed-Fi Data Standard | -| `element_cardinality` | identify keys, optional, required, optional conditional | - -> [DO WE INCLUDE THESE?] -| `Ed-FI_element_name` | Best-match entity name from the state definition | -| `Ed-FI_element_type` | Best-match entity name from the Ed-Fi Data Standard | -| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | +| `element_name` | Element name from the state definition | +| `element_type` | Element type from the Ed-Fi Data Standard | +| `element_cardinality` | Keys, optional, required, or optional conditional | +| `business_logic` | Extracted business rule text (verbatim quoted span from source documentation) | +| `cited_span` | Source document and page/section reference for the business logic | +| `in_scope` | Boolean scope classification | #### 2.1.2 Entity Matching to Standard Domain +**Story:** As a user, I want each element from the state Swagger matched to its Ed-Fi standard domain so that the ingestion output identifies the standard domain for every entity and surfaces any elements that could not be matched. + After parsing the Swagger/OAS document(s), the system SHALL match each API entity (resource, descriptor, association) to its corresponding **Ed-Fi Data Standard domain definition** using a standard domain registry. -The system SHALL produce a match report with the following columns at minimum: +The system SHALL produce a match result with the following columns at minimum: | Column | Description | |---|---| @@ -136,84 +125,72 @@ Unmatched entities SHALL be flagged for staff review rather than silently droppe The system SHALL accept a second input form — separate from the Swagger input — requesting the path(s) to the state's supplemental **business documentation** (e.g., data dictionaries, collection guides, data submission manuals, annotated spreadsheets). -This form SHALL accept up to **10** total file paths or URLs. Supported formats include PDF, Word (.docx), Excel (.xlsx), plain text, and HTML. - -The system SHALL run an LLM extraction process that operates at the **element level**, associating documented business rules to the specific API field or descriptor they govern. The extraction SHALL capture: +The system SHALL run process that operates at the **element level**, associating documented business rules to the specific API field or descriptor they govern. The extraction SHALL capture: * The element name (field path within the entity) * The extracted business logic text (verbatim quoted span from the source document) * The source document and page/section reference (cited span) * A confidence score for the extraction -> [!NOTE] -> The extraction LLM operates on documentation content only. It does not access student data or any data source beyond the files and URLs provided in this form. - #### 2.1.4 Scope Classification -The system SHALL run a scope classification process that determines, for each element in the Swagger, whether it is **in-scope** or **out-of-scope** for the state's data collection. - -An element is classified as in-scope if the state's Swagger or business documentation explicitly requires, uses, or references it. Elements present in the Ed-Fi Data Standard but absent from — or explicitly excluded by — the state's artifacts are classified as out-of-scope. +**Story:** As a user, I want each element classified as in-scope or out-of-scope based on who populates it and what write operations are permitted, so that scoring and analysis focus on the elements that carry real implementation burden for vendors. -The classification process SHALL emit a confidence level (High / Medium / Low) and a brief rationale for each out-of-scope classification. Staff MAY override any classification. +The system SHALL run a scope classification for each element in the Swagger using the following definitions: -#### 2.1.5 Ingestion Summary Report +* **In-scope:** An element is in-scope if it is accessible by the vendor (SIS, assessment, or other vendor), and the vendor **can** perform a PUT, POST, or DELETE on it. These are elements where the state sets the requirements and vendors must conform. -Upon completion of an ingestion run the system SHALL display an ingestion summary report. The report SHALL include, at minimum: +* **Out-of-scope:** An element is out-of-scope if it is populated by the state and can only be retrieved by vendors via a GET. These are read-only state-owned elements that vendors consume but do not write. -| Metric | Description | -|---|---| -| Total elements retrieved | Count of all element-level records parsed from the Swagger source(s) | -| Elements with business logic | Count of elements for which at least one business rule was extracted | -| Elements in scope | Count of elements classified as in-scope | -| Elements out of scope | Count of elements classified as out-of-scope | -| Unmatched entities | Count of entities that could not be matched to the Ed-Fi Data Standard | -| Extraction confidence distribution | Breakdown of High / Medium / Low confidence extractions | +The classification process SHALL emit a confidence level (High / Medium / Low) and a brief rationale for each out-of-scope classification. Staff MAY override any classification. -The report SHALL be exportable as CSV and as a rendered HTML table. +#### 2.1.5 Data Storage -#### 2.1.6 Persistence +**Story:** As a user, I want all ingested element data, business logic, and scope classifications automatically stored in the database so that the records are available for scoring and analysis. -After staff review and approval of an ingestion run, the system SHALL write the following data to the storage database for each element: +The system SHALL write the following data to the storage database for each element upon completion of an ingestion run: | Field | Description | |---|---| | `run_id` | Unique identifier for the ingestion run | | `state_id` | State identifier | | `entity_name` | Entity name from the state's Swagger | +| `element_name` | Name of the element as defined by the state | +| `data_type` | WHether it is a descriptor, reference, string.. | +| `cardinality` | WHether the element is an identity, required or optional | +| `source` | WHether it is core or extension | | `element_path` | Full field path within the entity | | `standard_entity` | Matched Ed-Fi Data Standard entity | | `domain` | Ed-Fi domain | -| `business_logic` | Extracted business rule text (null if none) | +| `business_requirements` | Extracted business rule text (null if none) | | `cited_span` | Source document reference for the business logic | | `in_scope` | Boolean scope flag | | `scope_confidence` | High / Medium / Low | | `created_at` | Timestamp of record creation | -| `run_approved_by` | Staff member who approved the run | - -The write operation SHALL be transactional. A partially written run SHALL be rolled back in full, not partially committed. +As a user I want to have a clear understanding of the similarity of the element to the ed-Fi data standard, and if the deviation to the core happened at element definition or in the business requirements to send such element, so that I can match the subsequent cascade rules and logic to the actual data --- -### 2.2 JTBD 2 — Scoring Engine - ALSO IDENTIFIED AS "JTBD1: Scoring Engine" in Initial PRD +### 2.2 JTBD 3 — Scoring Engine -**Story:** As Ed-Fi Alliance Staff — on behalf of Vendors — I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. +**Story:** As Ed-Fi Alliance Staff, I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. #### 2.2.1 Scoring LLM Execution -**Story:** As Ed-Fi Alliance Staff — on behalf of Vendors — I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. +**Story:** As a user, I want the scoring engine to extract facts from the business logic using an LLM, evaluate deterministic facts computed from the record structure, and assign scores using a rules cascade, so that each element has a well-supported, auditable complexity score. -**Acceptance Criteria:** +**Acceptance Criteria:** DRAFT +The resulting NACHOS scores and Adjusted NACHOS score for each record have all the rationale criteria that trigger the scoring, so that a user can link the actual scores to the rationale, and then link the rationale to the swagger and elements that trigger that rationale. + +**NACHOS Score Context (peer signal).** Every scored row SHALL carry key dimension results from the scoring process, beside the scalar and the previous elements listed. These elements are most likely produced by the LLM or deterministic questions. These are: -* Ed-Fi Alliance's existing manual scoring SHALL seed the evaluation set, treated as **human-scored observations** (used to measure agreement), not infallible ground truth. Before formal acceptance, Ed-Fi SHALL define the evaluation set and SHOULD grow it — ideally with more than one reviewer — so acceptance testing is credible. -* >= 80% score-tier alignment on rows scored by both the engine and a reviewer. -* >= 0.7 F1 overall and >= 0.9 F1 for "simple" (score-zero) rows, measured against the evaluation set as a calibration signal (not proof of correctness). -* >= 0.6 Cohen's Kappa (inter-rater agreement). -* Where the engine and a reviewer disagree on tier, treat those rows as the calibration target — they may point to a methodology question, not just an engine error, so route them to review rather than tuning them away to match one reviewer. -* Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the override and rationale recorded against the row's evidence record. -* **NACHOS Score Context (peer signal).** Every scored row SHALL carry, beside the scalar: _Implementation Shape_ (structural complexity, deterministic from the Swagger/API model), _Documentation Style_ (Prescriptive | Conceptual | Cross-reference | Regulatory | Unspecified), and _Documentation Gap_ (complex structure the documentation under-explains). -* **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, apply an adjustment (exact magnitudes subject to Alliance methodology review). -* **Evidence record (per row).** Every score SHALL emit a cached evidence record (extracted facts, confidence, cited spans, firing rule path, review routing) — extending the provenance model (NFR-DATA-2/3) to the scoring layer, so a score is inspectable without re-running the model. -* Cross-state outputs SHALL report Score Context **distributions**, not a single-scalar ranking that can misread documentation culture as quality. +* Documentation_completeness (definition/rules/types/descriptors present or not, deterministic from the Swagger/API model, user override), +* business_logic_complexity (Conditional | Conceptual | Cross-reference | Regulatory | Unspecified), +* All the Drivers for adjustment (necessary extension | Unnecessary extension | cross-entity logic) +* Extension_justification (why is necessary, or standalone) +* Semantic_fidelity (TO BE CONFIRMED: narrowed/broadened/divergent?) +* Definition qualty (present, >= 20 chars, implementable) +* Element name alignment (matches Ed-Fi's name) The system SHALL run a Scoring LLM against each in-scope element record produced by the Ingestion Pipeline. For each element the LLM SHALL extract and emit an **evidence record** containing, at minimum: @@ -233,8 +210,9 @@ The system SHALL run a Scoring LLM against each in-scope element record produced | `adjustment_rationale` | Plain-text explanation of any adjustment applied | | `candidate_recommendations` | List of suggested simplification or standardization actions | | `firing_rule_path` | The ordered list of rules from the rule cascade that produced this score | +| `tier_name` | Name of the NACHOS tier matched by the rule cascade | | `confidence` | Overall confidence in the score assignment (High / Medium / Low) | -| `review_routing` | `Auto-accept` \| `Route-for-review` (see §2.2.2) | +| `review_routing` | `Auto-accept` \| `Route-for-review` | > [!NOTE] > **NACHOS Score Context (peer signal).** Every scored row SHALL carry, alongside the scalar score: @@ -246,39 +224,34 @@ The system SHALL run a Scoring LLM against each in-scope element record produced > [!NOTE] > **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, an adjustment SHALL be applied to the base NACHOS score. Exact adjustment magnitudes are subject to Alliance methodology review and SHALL be configurable without a code deployment. -#### 2.2.2 Rule Cascade +#### 2.2.2 Rule Cascade and Score Overrides + +**Story:** As a user, I want to see all the rules and cascade elements that are triggered for each row, along with the associated NACHOS score, Adjusted NACHOS score, and the name of the tier matched, so that I can understand exactly how each score was derived. + +**Story:** As a user, I want to adjust scoring rules for specific fields with a rationale, or add new logic specific to a state, and trigger rescoring — with the full history of scoring changes preserved — so that I can refine scores based on domain expertise without losing the prior scoring record. -The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` field in the evidence record. +The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` and `tier_name` fields in the evidence record. > [!IMPORTANT] > Reference: _POC Recommendation document, page 11 — Rule Cascade table_. The rule cascade is the authoritative source for score assignment logic. Any discrepancy between this PRD and that document should be resolved in favor of the POC Recommendation document pending a formal methodology review. -Staff SHALL be able to inspect the full `firing_rule_path` for any scored element to understand exactly which rules fired. +Staff SHALL be able to inspect the full `firing_rule_path` and `tier_name` for any scored element to understand exactly which rules fired and which tier was matched. -#### 2.2.3 Score Reports +Staff SHALL be able to override a rule for a specific field by providing a written rationale. Staff SHALL also be able to add state-specific scoring logic for any field. Both overrides and additions SHALL trigger a rescore of the affected element(s). -Upon completion of a scoring run the system SHALL produce a tabular score report. The report SHALL include, at minimum, one row per scored element with the following columns: +The system SHALL preserve a complete history of scoring changes per element, including the original machine-assigned score, each override or addition applied, the rationale provided, the staff member who made the change, and the resulting rescored values. -| Column | Description | -|---|---| -| `entity_name` | Entity name | -| `element_path` | Full field path | -| `domain` | Ed-Fi domain | -| `nachos_score` | Base NACHOS scalar | -| `adjusted_nachos` | Adjusted NACHOS scalar | -| `documentation_style` | Documentation style classification | -| `implementation_shape` | Structural complexity flag | -| `documentation_gap` | Boolean | -| `review_routing` | Auto-accept or Route-for-review | -| `has_override` | Boolean — has this row been manually overridden by staff? | +#### 2.2.3 Logic alignment, rescoring and manual score ovewrite -The report SHALL be sortable and filterable by score, adjusted score, domain, documentation style, and review routing. It SHALL be exportable as CSV. +* **Story** as an Ed-Fi ALliance Staff, I must be able to add a new business rule for a particular element or an entity, and re-run the LLM scoring, showing the old and new NACHOS and adjusted NACHOS scores, along with the rules that it triggers, and decide if the new scores are acceptable. +* +* **Story** Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the new scores and rationale recorded against the row's evidence record. -Cross-state score reports SHALL report **Score Context distributions** (counts and percentages by Documentation Style, Implementation Shape tier, and Documentation Gap rate) rather than a single aggregate ranking. +#### 2.2.4 Score and Override Storage -#### 2.2.4 Score Persistence +**Story:** As a user, I want all scores, evidence records, rule overrides, and the full scoring history stored in the database so that any score is inspectable without re-running the model and all changes are traceable over time. -After a scoring run is approved by staff, the system SHALL write scores to the storage database, extending each element record with the following fields: +The system SHALL write scores and evidence records to the storage database upon completion of a scoring run, extending each element record with the following fields: | Field | Description | |---|---| @@ -298,19 +271,7 @@ After a scoring run is approved by staff, the system SHALL write scores to the s The evidence record SHALL be stored in full so that any score is inspectable without re-running the model. -#### 2.2.5 Scored Element Forms - -The system SHALL produce two filtered views (exportable as printable/shareable forms) from a completed scoring run: - -**Form A — Elements Requiring Review** (`adjusted_nachos > 0`) - -All elements where the Adjusted NACHOS score is greater than zero, sorted by adjusted score descending. This form surfaces elements that carry implementation complexity and are candidates for standardization conversations with the SEA. - -**Form B — Simple Elements** (`adjusted_nachos = 0`) - -All elements where the Adjusted NACHOS score equals zero. These elements are assessed as straightforward implementations with no deviation from the canonical Ed-Fi definition. A high F1 score on this form (target >= 0.9) is a key quality signal. - -Both forms SHALL display, at minimum: entity name, element path, domain, base NACHOS score, adjusted NACHOS score, documentation style, and any candidate recommendations. +The system SHALL maintain a scoring history table that records each override or state-specific rule addition, preserving the prior score, the change applied, the rationale, and the rescored result. --- @@ -326,13 +287,11 @@ Analysis of aggregate score data across states and across time is scoped for a f | ID | Category | Requirement | Implementation | |---|---|---|---| -| NFR-PERF-1 | Performance | Ingestion of a single-state Swagger file (up to 500 endpoints) SHALL complete within 5 minutes | Async job queue; progress indicator displayed to user | -| NFR-PERF-2 | Performance | Scoring run for up to 2,000 elements SHALL complete within 15 minutes | Batched LLM calls; parallel processing where element order permits | | NFR-DATA-1 | Data Integrity | All ingestion and scoring runs SHALL be transactional — no partial writes | Database transaction wrapping each run | | NFR-DATA-2 | Provenance | Every ingestion record SHALL carry a `run_id`, `state_id`, and `created_at` timestamp | Applied at write time | | NFR-DATA-3 | Provenance | Every score SHALL carry a full evidence record persisted alongside the scalar | `evidence_record` JSON column on score table | | NFR-SEC-1 | Security | No student data or PII SHALL flow through any pipeline component | Input validation at ingestion form; static analysis gate in CI | -| NFR-ACC-1 | Accessibility | All reports and forms SHALL meet WCAG 2.1 AA | Verified with automated accessibility checker in CI | +| NFR-ACC-1 | Accessibility | All views and forms SHALL meet WCAG 2.1 AA | Verified with automated accessibility checker in CI | ### 3.2 Quality Gates (Scoring Engine) @@ -344,9 +303,6 @@ Analysis of aggregate score data across states and across time is scoped for a f | QG-4 | Cohen's Kappa | >= 0.6 | Inter-rater agreement between engine and human reviewer | | QG-5 | Disagreement routing | 100% of tier disagreements routed for review | Disagreements are treated as calibration signal, not auto-resolved | -> [!NOTE] -> Before formal acceptance testing, Ed-Fi Alliance SHALL define the composition of the evaluation set and SHOULD grow it — ideally with more than one reviewer — so that acceptance metrics are credible. The evaluation set is a living artifact, not a one-time snapshot. - --- ## 4. Architecture @@ -367,47 +323,45 @@ Analysis of aggregate score data across states and across time is scoped for a f ``` ┌─────────────────────────────────────────────────────────────────────┐ -│ INGESTION PIPELINE │ +│ INGESTION PIPELINE (JTBD 1) │ │ │ -│ [User: Swagger file/URL] │ +│ [User: view / edit source list] │ │ │ │ │ ▼ │ -│ Parse & Validate OAS ──► Entity Match to Ed-Fi Domain Registry │ +│ Parse & Validate Swagger ──► Entity Match to Ed-Fi Domain Registry │ │ │ │ -│ [User: Business doc paths/URLs] │ +│ [User: Business doc paths/URLs — Confluence, GitHub/Excel, │ +│ TWEDS, PDF, Excel, HTML] │ │ │ │ │ ▼ │ -│ Business Logic Extraction LLM (element-level, with cited spans) │ +│ Business Logic Extraction LLM (element-level, verbatim + │ +│ provenance pointer) │ │ │ │ │ ▼ │ -│ Scope Classification ──► Ingestion Summary Report │ -│ │ │ -│ ▼ │ -│ [Staff review & approve] │ +│ Scope Classification (in-scope: state-populated, vendor cannot │ +│ PUT/POST/DELETE; out-of-scope: state-populated, GET only) │ │ │ │ │ ▼ │ │ Write to Storage DB (elements + business logic + scope flags) │ └─────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────┐ -│ SCORING ENGINE │ +│ SCORING ENGINE (JTBD 3) │ │ │ │ Read in-scope elements from Storage DB │ │ │ │ │ ▼ │ -│ Scoring LLM ──► Evidence Record (flags, cited spans, confidence) │ -│ │ │ -│ ▼ │ -│ Rule Cascade ──► NACHOS Score + Adjusted NACHOS │ -│ │ │ +│ Scoring LLM ──► LLM-extracted facts + deterministic facts │ +│ │ computed from record structure │ │ ▼ │ -│ Score Report (tabular) + Form A (adj > 0) + Form B (adj = 0) │ +│ Rule Cascade ──► NACHOS Score + Adjusted NACHOS + Tier Name │ │ │ │ │ ▼ │ -│ [Staff review; override where needed] │ +│ [User: inspect firing_rule_path + tier_name per row; │ +│ adjust rules or add state-specific logic; trigger rescore] │ │ │ │ │ ▼ │ -│ Write scores + evidence records to Storage DB │ +│ Write scores + evidence records + override history to Storage DB │ └─────────────────────────────────────────────────────────────────────┘ ``` @@ -417,8 +371,9 @@ Analysis of aggregate score data across states and across time is scoped for a f |---|---|---| | `ingestion_runs` | `run_id` | `state_id`, `created_at`, `approved_by`, `status` | | `elements` | `element_id` | `run_id`, `state_id`, `entity_name`, `element_path`, `standard_entity`, `domain`, `in_scope`, `business_logic`, `cited_span` | -| `scoring_runs` | `scoring_run_id` | `run_id` (FK → ingestion_runs), `created_at`, `approved_by`, `status` | -| `scores` | `score_id` | `element_id` (FK → elements), `scoring_run_id`, `nachos_score`, `adjusted_nachos`, `evidence_record` (JSON), `override_score`, `override_by` | +| `scoring_runs` | `scoring_run_id` | `run_id` (FK → ingestion_runs), `created_at`, `status` | +| `scores` | `score_id` | `element_id` (FK → elements), `scoring_run_id`, `nachos_score`, `adjusted_nachos`, `tier_name`, `evidence_record` (JSON), `override_score`, `override_by` | +| `scoring_history` | `history_id` | `score_id` (FK → scores), `prior_score`, `change_type`, `rationale`, `changed_by`, `rescored_at` | --- From c35443daf0fd0d643694c7b7203ef9f491eabb51 Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Mon, 22 Jun 2026 10:44:26 -0500 Subject: [PATCH 3/6] MC-10 Simplified Ingestion logic MC-10 Simplified Ingestion logic by making few changes. Document is still in draft mode. --- .vscode/settings.json | 4 +++- PRD-Ingestion-Scoring.md | 43 +++++++++++++++------------------------- 2 files changed, 19 insertions(+), 28 deletions(-) diff --git a/.vscode/settings.json b/.vscode/settings.json index a9fce93..ce6a10f 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -91,6 +91,8 @@ "source.fixAll.markdownlint": "explicit" }, "cSpell.words": [ - "browsable" + "browsable", + "inspectable", + "qualty" ] } diff --git a/PRD-Ingestion-Scoring.md b/PRD-Ingestion-Scoring.md index 3e49288..ee214b6 100644 --- a/PRD-Ingestion-Scoring.md +++ b/PRD-Ingestion-Scoring.md @@ -21,14 +21,14 @@ Access the following PRD for an overview of the overall Metadata Project. **Lin ### 1.2 Problem Statement -Vendor integration costs are a widely-cited barrier to expanding K-12 data access. Those costs stem from complex, aggregated, and inconsistent requirements across states. Without a systematic way to extract and score SEA specification metadata, Ed-Fi staff have no scalable method to: +Ed-Fi staff have no scalable method to: * Measure the complexity of a state's data collection requirements. * Compare complexity across states. * Identify which specific elements drive implementation burden. * Prioritize standardization conversations with the highest-leverage states. -Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. An LLM-assisted pipeline that produces auditable, human-overridable scores addresses all four gaps. +Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. The ingestion process, and an LLM-assisted pipeline that produces auditable, human-overridable scores produce the output for dashboards and other analytics, and the list of elements that require evaluation to reduce complexity. ### 1.3 Target Users for this PRD @@ -73,41 +73,30 @@ The system SHALL support the following source types for supplemental business do The system SHALL validate that each provided file or URL is resolvable before proceeding. If a source fails validation, the system SHALL report the failure and allow the user to correct or remove the invalid entry without re-submitting valid sources. -The output of the ingestion process SHALL contain one row per attribute with the following columns: - -| Column | Description | -|---|---| -| `entity_name` | Entity name as it appears in the state's Swagger | -| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | -| `domain` | Ed-Fi domain (e.g., Student, Enrollment, Assessment) | -| `match_confidence` | High / Medium / Low / Unmatched | -| `match_notes` | Flags for extension entities, renamed entities, or descriptor overloads | -| `element_name` | Element name from the state definition | -| `element_type` | Element type from the Ed-Fi Data Standard | -| `element_cardinality` | Keys, optional, required, or optional conditional | -| `business_logic` | Extracted business rule text (verbatim quoted span from source documentation) | -| `cited_span` | Source document and page/section reference for the business logic | -| `in_scope` | Boolean scope classification | - #### 2.1.2 Entity Matching to Standard Domain **Story:** As a user, I want each element from the state Swagger matched to its Ed-Fi standard domain so that the ingestion output identifies the standard domain for every entity and surfaces any elements that could not be matched. After parsing the Swagger/OAS document(s), the system SHALL match each API entity (resource, descriptor, association) to its corresponding **Ed-Fi Data Standard domain definition** using a standard domain registry. -The system SHALL produce a match result with the following columns at minimum: +**Job Story** Each entity needs to be matched to a key domain based on the list of entities per domain provided, in this way, analysis across domains and states can be standardized. For extended entities without a direct domain map, the process needs to map to the closest domain, based on the common names in the entity title, for example: coursetranscript_ext is very similar to coursetranscript, and therefore allocated to the same domain. If the entity's name is completely different than an ed-fi entity, then the domain listed will be: state_specific. + +The process should try to match the extended entities to domain as much as possible. + +**Job Story** Unmatched entities SHALL be flagged for staff review rather than silently dropped. Staff MAY resolve an unmatched entity by manually mapping it or marking it as a state-specific extension. + +The output of the ingestion process SHALL contain one row per attribute with the following columns: | Column | Description | |---|---| | `entity_name` | Entity name as it appears in the state's Swagger | -| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard | +| `standard_entity` | Best-match entity name from the Ed-Fi Data Standard - CAN WE GET THIS?| | `domain` | Ed-Fi domain (e.g., Student, Enrollment, Assessment) | | `match_confidence` | High / Medium / Low / Unmatched | | `match_notes` | Flags for extension entities, renamed entities, or descriptor overloads | - -Unmatched entities SHALL be flagged for staff review rather than silently dropped. Staff MAY resolve an unmatched entity by manually mapping it or marking it as a state-specific extension. - -**Question: do we produce a report to pull elements that do not match other state definitions** +| `element_name` | Element name from the state definition | +| `element_type` | Element type from the Ed-Fi Data Standard | +| `element_cardinality` | Keys, optional, required, or optional conditional | #### 2.1.3 Business Logic Extraction IDENTIFIED AS "JBTD 3: Standardization of Data Collection Metadata " in Initial PRD @@ -189,7 +178,7 @@ The resulting NACHOS scores and Adjusted NACHOS score for each record have all t * All the Drivers for adjustment (necessary extension | Unnecessary extension | cross-entity logic) * Extension_justification (why is necessary, or standalone) * Semantic_fidelity (TO BE CONFIRMED: narrowed/broadened/divergent?) -* Definition qualty (present, >= 20 chars, implementable) +* Definition quality (present, >= 20 chars, implementable) * Element name alignment (matches Ed-Fi's name) The system SHALL run a Scoring LLM against each in-scope element record produced by the Ingestion Pipeline. For each element the LLM SHALL extract and emit an **evidence record** containing, at minimum: @@ -232,7 +221,7 @@ The system SHALL run a Scoring LLM against each in-scope element record produced The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` and `tier_name` fields in the evidence record. -> [!IMPORTANT] +> [!IMPORTANT - NOT REALLY] > Reference: _POC Recommendation document, page 11 — Rule Cascade table_. The rule cascade is the authoritative source for score assignment logic. Any discrepancy between this PRD and that document should be resolved in favor of the POC Recommendation document pending a formal methodology review. Staff SHALL be able to inspect the full `firing_rule_path` and `tier_name` for any scored element to understand exactly which rules fired and which tier was matched. @@ -271,7 +260,7 @@ The system SHALL write scores and evidence records to the storage database upon The evidence record SHALL be stored in full so that any score is inspectable without re-running the model. -The system SHALL maintain a scoring history table that records each override or state-specific rule addition, preserving the prior score, the change applied, the rationale, and the rescored result. +The system SHALL maintain a scoring history table that records each override or state-specific rule addition, preserving the prior score, the change applied, the rationale, and the re-scored result. --- From beb4a913c12d53c17af34f7e1353c39b73145f9e Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Mon, 6 Jul 2026 17:51:26 -0500 Subject: [PATCH 4/6] MC-10 Streamlined Ingestion process MC-10 Streamlined Ingestion process. Still in draft mode. Need to update the 4.2 Pipeline Overview --- .../PRD-Ingestion-Scoring.md | 86 ++++++++++--------- 1 file changed, 44 insertions(+), 42 deletions(-) rename PRD-Ingestion-Scoring.md => docs/PRD-Ingestion-Scoring.md (84%) diff --git a/PRD-Ingestion-Scoring.md b/docs/PRD-Ingestion-Scoring.md similarity index 84% rename from PRD-Ingestion-Scoring.md rename to docs/PRD-Ingestion-Scoring.md index ee214b6..46f7dac 100644 --- a/PRD-Ingestion-Scoring.md +++ b/docs/PRD-Ingestion-Scoring.md @@ -1,6 +1,6 @@ # Metadata Catalog — Ingestion & Scoring PRD -> **Status:** Draft · **Last updated:** 2026-06-18 · **Owner:** Ed-Fi Alliance Staff +> **Status:** Draft · **Last updated:** 2026-7-6 · **Owner:** Ed-Fi Alliance Staff --- @@ -30,15 +30,38 @@ Ed-Fi staff have no scalable method to: Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. The ingestion process, and an LLM-assisted pipeline that produces auditable, human-overridable scores produce the output for dashboards and other analytics, and the list of elements that require evaluation to reduce complexity. -### 1.3 Target Users for this PRD +### 1.3 Solutions to the Problem + +To solve the problems associated to the manual scoring, there will be automated processes to automatically: (1) pull a state Data Standard model configuration (handled in the ingestion pipeline); (2) to define the Business logic imposed to vendors for each element, and apply the two NACHOS scores based on the logic (handled in the scoring pipeline); (3) to store these outputs in a storage database; (4) to produce a list of elements with some complexity that should be reviewed with the states, and (5) to support a manual override of some scores by adding new business requirements as these are learned from state conversations and updating the database with the new scores; and (6) to support further analytical work using the newly updated database. + +**Story: At the end of the ingestion and scoring processes, the storage database will be updated with a state-specific Ed-Fi data model, with all the business requirements and logic for all elements that are passed to the vendors, their NACHOS and Adjusted NACHOS scores and rationale for the scoring and recommendations as a minimum, so that Ed-Fi members can run reports on the data, analyze common added complexity elements. At the end of the scoring processes there will be a list artefact that lists the elements with some complexity scores, so that the Ed-Fi architecture team can discuss solutions with state agencies to reduce their complexity.** + +The database will be updated with all the scoring information, including + +* Updated swagger for any state, indicating the elements that are sent to SIS, Assessment or any other vendor for their processes, including extended entities, and/or extended elements within core entities. +* Updated business logic, which indicates what processes have to be run for a vendor to send that element. A data element can trigger multiple business logic. +* NACHOS and Adjusted NACHOS Scores for each element, rationale for the scores, along with recommendation if applicable. +* Updated business logic and scores from manual review. + +### 1.4 Term Definitions + +| Term | Definition | +|---|---| +| **State-specific swagger** | in this document, it refers to all the elements that are required by that state for state reporting, including extensions, and if these extensions are necessary or not along with rationale to why they are necessary| +| **Extension** | Any data element, including entities, not listed in the Ed-Fi Data Standard in a specific version. Thus if the state uses Ed-Fi DS 4.0, then extensions refer to any element or entity not listed in the Ed-FI DS v4.0| +| **Source File** |Resulting file from the ingestion pipeline, which contains any data element of the data model listed by a state| +| **Spine File** |Resulting file from the ingestion pipeline, which contains the business logic identified for any element by a particular state | +| **In Scope file** |File that contains the data elements, including the entities, for a state-specific data model AND that are updated by vendors. The in-scope file excludes values updated by the state agency, which are only pull by vendors| + +### 1.5 Target Users for this PRD | User | Role | |---|---| | **Ed-Fi Alliance Staff** | Primary operators of both pipelines; consumers of score reports and standardization analytics | -Excluded from this PRD are: Vendors and SEA Staff. They will be interested in read only capacity to some of the data. Vendors and SEA Staff may be given read access to some of the data in the future. SEA Staff may be given access to write a commitment form, and submit corrections to the scoring and business requirements. +Excluded from this PRD are: SIS, Assessment or other Vendors, as SEA Staff. They will be interested in read only capacity to some of the data. Vendors and SEA Staff may be given read access to some of the data in the future. SEA Staff may be given access to write a commitment form, and submit corrections to the scoring and business requirements. -### 1.4 Design Principles +### 1.6 Design Principles **Auditability over automation.** Every score is traceable to an evidence record. A score without a traceable rationale is not acceptable. @@ -98,19 +121,11 @@ The output of the ingestion process SHALL contain one row per attribute with the | `element_type` | Element type from the Ed-Fi Data Standard | | `element_cardinality` | Keys, optional, required, or optional conditional | -#### 2.1.3 Business Logic Extraction IDENTIFIED AS "JBTD 3: Standardization of Data Collection Metadata " in Initial PRD - -**Story:** As Ed-Fi Alliance Staff, I want to extract data collection requirements from any SEA and store them in a standardized manner, so that I can analyze and help improve the SEA's requirements. +### 2.2 JTBD 3 — Scoring Engine -**Acceptance Criteria:** +**Story:** As Ed-Fi Alliance Staff, I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. -* Standardized metadata storage with a schema that can represent normalized metadata plus provenance and curation status -* The standardized repository SHALL preserve provenance and curation status for each material metadata element, including whether it came from exact artifact import, inexact supplemental extraction, or manual editing. -* Phase 1 minimum acceptance: Ed-Fi Alliance Staff MUST be able to load, store, and manually curate standardized metadata in that repository even before any scoring or other downstream analytical consumer is delivered. -* Ed-Fi Alliance Staff MUST be able to manually curate standardized metadata regardless of whether it originated from exact import or inexact extraction. -* Phase 2 downstream-consumer readiness: the standardized repository SHALL expose standardized metadata programmatically to the scoring engine and other downstream use cases once those consumers are introduced. -* The schema SHALL be at the **(resource, property) grain** and SHALL map state documentation onto the **canonical Ed-Fi Data Standard surface**, so a source lens and an API-model (coverage) lens are possible. -* The repository SHALL retain the **source narrative text** (or stable, normalized references) sufficient to validate and display _cited spans_; a cited span cannot be shown against source text that was not retained. (The Phase-1 schema should anticipate this.) +#### 2.2.1 Business Logic Extraction The system SHALL accept a second input form — separate from the Swagger input — requesting the path(s) to the state's supplemental **business documentation** (e.g., data dictionaries, collection guides, data submission manuals, annotated spreadsheets). @@ -121,7 +136,7 @@ The system SHALL run process that operates at the **element level**, associating * The source document and page/section reference (cited span) * A confidence score for the extraction -#### 2.1.4 Scope Classification +#### 2.2.2 Scope Classification **Story:** As a user, I want each element classified as in-scope or out-of-scope based on who populates it and what write operations are permitted, so that scoring and analysis focus on the elements that carry real implementation burden for vendors. @@ -133,7 +148,7 @@ The system SHALL run a scope classification for each element in the Swagger usin The classification process SHALL emit a confidence level (High / Medium / Low) and a brief rationale for each out-of-scope classification. Staff MAY override any classification. -#### 2.1.5 Data Storage +#### 2.2.3 Data Storage **Story:** As a user, I want all ingested element data, business logic, and scope classifications automatically stored in the database so that the records are available for scoring and analysis. @@ -157,14 +172,10 @@ The system SHALL write the following data to the storage database for each eleme | `scope_confidence` | High / Medium / Low | | `created_at` | Timestamp of record creation | -As a user I want to have a clear understanding of the similarity of the element to the ed-Fi data standard, and if the deviation to the core happened at element definition or in the business requirements to send such element, so that I can match the subsequent cascade rules and logic to the actual data +TBD in discussion with Chris, Suganya: As a user I want to have a clear understanding of the similarity of an extended element to the ed-Fi data standard, so that I can further inquiry with the state agency the reasoning for having an extension instead of the Ed-Fi core element --- -### 2.2 JTBD 3 — Scoring Engine - -**Story:** As Ed-Fi Alliance Staff, I want to run a scoring engine that assigns a complexity score to each SEA data collection requirement, so that I can help the SEA align their data collection with the Ed-Fi Data Standard. - -#### 2.2.1 Scoring LLM Execution +#### 2.2.4 Scoring LLM Execution **Story:** As a user, I want the scoring engine to extract facts from the business logic using an LLM, evaluate deterministic facts computed from the record structure, and assign scores using a rules cascade, so that each element has a well-supported, auditable complexity score. @@ -213,7 +224,7 @@ The system SHALL run a Scoring LLM against each in-scope element record produced > [!NOTE] > **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, an adjustment SHALL be applied to the base NACHOS score. Exact adjustment magnitudes are subject to Alliance methodology review and SHALL be configurable without a code deployment. -#### 2.2.2 Rule Cascade and Score Overrides +#### 3. Rule Cascade and Score Overrides **Story:** As a user, I want to see all the rules and cascade elements that are triggered for each row, along with the associated NACHOS score, Adjusted NACHOS score, and the name of the tier matched, so that I can understand exactly how each score was derived. @@ -221,24 +232,19 @@ The system SHALL run a Scoring LLM against each in-scope element record produced The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` and `tier_name` fields in the evidence record. -> [!IMPORTANT - NOT REALLY] -> Reference: _POC Recommendation document, page 11 — Rule Cascade table_. The rule cascade is the authoritative source for score assignment logic. Any discrepancy between this PRD and that document should be resolved in favor of the POC Recommendation document pending a formal methodology review. - Staff SHALL be able to inspect the full `firing_rule_path` and `tier_name` for any scored element to understand exactly which rules fired and which tier was matched. -Staff SHALL be able to override a rule for a specific field by providing a written rationale. Staff SHALL also be able to add state-specific scoring logic for any field. Both overrides and additions SHALL trigger a rescore of the affected element(s). - -The system SHALL preserve a complete history of scoring changes per element, including the original machine-assigned score, each override or addition applied, the rationale provided, the staff member who made the change, and the resulting rescored values. - -#### 2.2.3 Logic alignment, rescoring and manual score ovewrite +#### 3.1 Logic alignment, rescoring and manual score overwrite * **Story** as an Ed-Fi ALliance Staff, I must be able to add a new business rule for a particular element or an entity, and re-run the LLM scoring, showing the old and new NACHOS and adjusted NACHOS scores, along with the rules that it triggers, and decide if the new scores are acceptable. * * **Story** Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the new scores and rationale recorded against the row's evidence record. -#### 2.2.4 Score and Override Storage +**Story:** As a user, I want to pull all scores, evidence records, rule overrides, and the full scoring history stored in the database so that any score is inspectable without re-running the model and all changes are traceable over time. + +The system SHALL preserve a complete history of scoring changes per element, including the original machine-assigned score, each override or addition applied, the rationale provided, the staff member who made the change, and the resulting rescored values. -**Story:** As a user, I want all scores, evidence records, rule overrides, and the full scoring history stored in the database so that any score is inspectable without re-running the model and all changes are traceable over time. +#### 3.2 Score and Override Storage The system SHALL write scores and evidence records to the storage database upon completion of a scoring run, extending each element record with the following fields: @@ -247,24 +253,20 @@ The system SHALL write scores and evidence records to the storage database upon | `scoring_run_id` | Unique identifier for the scoring run | | `nachos_score` | Base NACHOS scalar | | `adjusted_nachos` | Adjusted NACHOS scalar | -| `implementation_shape` | Structural complexity classification | -| `documentation_style` | Documentation style classification | -| `documentation_gap` | Boolean | -| `evidence_record` | Full JSON evidence record (all LLM-extracted fields) | -| `firing_rule_path` | Ordered rule list that produced the score | -| `review_routing` | Auto-accept or Route-for-review | -| `override_score` | Staff-assigned score (null if not overridden) | +| `override_nachos_score` | Staff-assigned score (null if not overridden) | | `override_rationale` | Staff-provided rationale for override (null if not overridden) | | `override_by` | Staff member who applied the override (null if not overridden) | | `scored_at` | Timestamp of score assignment | +| `override_adjusted_nachos_score` | Calculated over override_nachos_score (null if not overridden) | +TBD with Chris, Suganya: Should we label the nachos_score as OLD, and then update new nachos-score with the overwritten value. The evidence record SHALL be stored in full so that any score is inspectable without re-running the model. The system SHALL maintain a scoring history table that records each override or state-specific rule addition, preserving the prior score, the change applied, the rationale, and the re-scored result. --- -### 2.3 JTBD 3 — Score Evaluation Analytics _(TBD)_ +### 4. JTBD 3 — Score Evaluation Analytics _(TBD)_ Analysis of aggregate score data across states and across time is scoped for a future phase. Requirements will be defined separately. From 6f4fc8f9373f60b3c8d5d69c122a55f308268f3f Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Tue, 7 Jul 2026 16:58:43 -0500 Subject: [PATCH 5/6] MC-10- streamlined PRD MC-10- streamlined PRD, removed the technology stack portion, and simplified the overall document. --- docs/PRD-Ingestion-Scoring.md | 223 ++++++++-------------------------- 1 file changed, 54 insertions(+), 169 deletions(-) diff --git a/docs/PRD-Ingestion-Scoring.md b/docs/PRD-Ingestion-Scoring.md index 46f7dac..6e09bbc 100644 --- a/docs/PRD-Ingestion-Scoring.md +++ b/docs/PRD-Ingestion-Scoring.md @@ -8,7 +8,7 @@ ### 1.1 Background -The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Two high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, and a **cluster analysis** that surfaces cross-state patterns. +The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Three high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, a process that involves conversations with the SEAs to align their speicifications to the Ed-Fi Data Standard (reduction in commplexity), and a **cluster analysis** that surfaces cross-state patterns. This PRD covers phase 2 of automated ingestion and NACHOS scoring. @@ -17,7 +17,7 @@ This document covers the two pipelines that make those workflows possible: * the **Ingestion Pipeline** (JTBD 1), which converts raw SEA artifacts (Swagger/OAS files and supplemental business documentation) into structured, element-level records. * and the **Scoring Engine** (JTBD 3), which assigns NACHOS complexity scores to those records. -Access the following PRD for an overview of the overall Metadata Project. **Link To be added** +The PRD for the overall Metadata Project is referenced in this link. **Link To be added** ### 1.2 Problem Statement @@ -28,38 +28,51 @@ Ed-Fi staff have no scalable method to: * Identify which specific elements drive implementation burden. * Prioritize standardization conversations with the highest-leverage states. -Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. The ingestion process, and an LLM-assisted pipeline that produces auditable, human-overridable scores produce the output for dashboards and other analytics, and the list of elements that require evaluation to reduce complexity. +Manual scoring is slow, inconsistent across reviewers, and does not scale to the number of states and elements in scope. ### 1.3 Solutions to the Problem -To solve the problems associated to the manual scoring, there will be automated processes to automatically: (1) pull a state Data Standard model configuration (handled in the ingestion pipeline); (2) to define the Business logic imposed to vendors for each element, and apply the two NACHOS scores based on the logic (handled in the scoring pipeline); (3) to store these outputs in a storage database; (4) to produce a list of elements with some complexity that should be reviewed with the states, and (5) to support a manual override of some scores by adding new business requirements as these are learned from state conversations and updating the database with the new scores; and (6) to support further analytical work using the newly updated database. +The ingestion process, and an LLM-assisted pipeline that produces auditable, human-overridable scores produce the output for dashboards and other analytics, and the list of elements that require evaluation to reduce complexity. +There will be automated processes to: -**Story: At the end of the ingestion and scoring processes, the storage database will be updated with a state-specific Ed-Fi data model, with all the business requirements and logic for all elements that are passed to the vendors, their NACHOS and Adjusted NACHOS scores and rationale for the scoring and recommendations as a minimum, so that Ed-Fi members can run reports on the data, analyze common added complexity elements. At the end of the scoring processes there will be a list artefact that lists the elements with some complexity scores, so that the Ed-Fi architecture team can discuss solutions with state agencies to reduce their complexity.** +* (1) pull the most recent Data Standard model configuration for a specific state (handled in the ingestion pipeline); + +* (2) to define the Business logic imposed to vendors for each element, and apply the two NACHOS scores based on the logic (handled in the scoring pipeline); + +* (3) to store these outputs in a storage database; + +* (4) TBD: to produce a list of elements that should be reviewed with the states, in order to reduce their complexity; and -The database will be updated with all the scoring information, including +* (5) to support a manual override of some scores based on state conversations. + +**Story: At the end of the ingestion and scoring processes, the storage database will be updated with the state-specific Ed-Fi data model and derived results from the ingestion and scoring pipelines, so that an analytical engine can read the data to run reports, produce dashboards and run cluster analyses. At the end of the scoring processes, there will be an additional artefact, which lists the elements with some complexity scores, so that the Ed-Fi architecture team can discuss solutions with state agencies to reduce such complexity.** + +The database will be updated with all the ingestion and scoring information, including * Updated swagger for any state, indicating the elements that are sent to SIS, Assessment or any other vendor for their processes, including extended entities, and/or extended elements within core entities. * Updated business logic, which indicates what processes have to be run for a vendor to send that element. A data element can trigger multiple business logic. -* NACHOS and Adjusted NACHOS Scores for each element, rationale for the scores, along with recommendation if applicable. -* Updated business logic and scores from manual review. +* NACHOS and Adjusted NACHOS Scores for each element, rationale for the scores, along with the applicable recommendations. +* Updated business logic and scores from manual review and rationale for the updates. ### 1.4 Term Definitions | Term | Definition | |---|---| | **State-specific swagger** | in this document, it refers to all the elements that are required by that state for state reporting, including extensions, and if these extensions are necessary or not along with rationale to why they are necessary| -| **Extension** | Any data element, including entities, not listed in the Ed-Fi Data Standard in a specific version. Thus if the state uses Ed-Fi DS 4.0, then extensions refer to any element or entity not listed in the Ed-FI DS v4.0| -| **Source File** |Resulting file from the ingestion pipeline, which contains any data element of the data model listed by a state| -| **Spine File** |Resulting file from the ingestion pipeline, which contains the business logic identified for any element by a particular state | -| **In Scope file** |File that contains the data elements, including the entities, for a state-specific data model AND that are updated by vendors. The in-scope file excludes values updated by the state agency, which are only pull by vendors| +| **Extension** | Any data element, including a whole entity, not listed in the Ed-Fi Data Standard in a specific version. For example, if the state uses Ed-Fi DS 4.0, then an extension refers to any element or entity not listed in the Ed-FI DS v4.0| +| **Source File** |Resulting artifact from the ingestion pipeline, which contains any data element of the data model listed by a state| +| **Spine File** |Resulting artifact from the ingestion pipeline, which contains the business logic identified for any element by a particular state | +| **Gap Log File** |Resulting artifact from the ingestion pipeline, which matches the elements that are both in the source and spine files. This artifact represents the elements from the state that have business requirements | +| **In Scope attribute** |Attribute that indicates if the data element needs to be populated by vendors via Pull, PUSH, GET in which case it is in-scope = true. If the element is populated by the state and only read by the vendor, then in-scope = false. Descriptors are in-scope| ### 1.5 Target Users for this PRD | User | Role | |---|---| | **Ed-Fi Alliance Staff** | Primary operators of both pipelines; consumers of score reports and standardization analytics | +| **Development and Analytical team** | Creators of the pipelines and of the system | -Excluded from this PRD are: SIS, Assessment or other Vendors, as SEA Staff. They will be interested in read only capacity to some of the data. Vendors and SEA Staff may be given read access to some of the data in the future. SEA Staff may be given access to write a commitment form, and submit corrections to the scoring and business requirements. +Excluded from this PRD are: SIS, Assessment or other Vendors, and SEA Staff. ### 1.6 Design Principles @@ -138,7 +151,7 @@ The system SHALL run process that operates at the **element level**, associating #### 2.2.2 Scope Classification -**Story:** As a user, I want each element classified as in-scope or out-of-scope based on who populates it and what write operations are permitted, so that scoring and analysis focus on the elements that carry real implementation burden for vendors. +**Story:** As a user, I want each element classified as in-scope or out-of-scope based on the scope definition below, so that scoring and analysis focus on the elements that carry real implementation burden for vendors. The system SHALL run a scope classification for each element in the Swagger using the following definitions: @@ -152,7 +165,7 @@ The classification process SHALL emit a confidence level (High / Medium / Low) a **Story:** As a user, I want all ingested element data, business logic, and scope classifications automatically stored in the database so that the records are available for scoring and analysis. -The system SHALL write the following data to the storage database for each element upon completion of an ingestion run: +The system SHALL write the following data to the storage database for each element: | Field | Description | |---|---| @@ -172,7 +185,6 @@ The system SHALL write the following data to the storage database for each eleme | `scope_confidence` | High / Medium / Low | | `created_at` | Timestamp of record creation | -TBD in discussion with Chris, Suganya: As a user I want to have a clear understanding of the similarity of an extended element to the ed-Fi data standard, so that I can further inquiry with the state agency the reasoning for having an extension instead of the Ed-Fi core element --- #### 2.2.4 Scoring LLM Execution @@ -197,16 +209,16 @@ The system SHALL run a Scoring LLM against each in-scope element record produced | Field | Description | |---|---| | `element_path` | Full field path within the entity | -| `conditional_logic` | Boolean — does the row describe conditional or branching logic? | -| `cross_entity_logic` | Boolean — does the row involve lookups or constraints across entity boundaries? | -| `aggregation` | Boolean — does the row require computed aggregates or derived values? | -| `concatenation` | Boolean — does the row require string construction from multiple fields? | -| `semantic_divergence` | Boolean — does the state narrow, broaden, or redefine the canonical Ed-Fi meaning? | +| `conditional_logic` | String — does the row describe conditional or branching logic? | +| `cross_entity_logic` | String — does the row involve lookups or constraints across entity boundaries? | +| `aggregation` | String — does the row require computed aggregates or derived values? | +| `concatenation` | String — does the row require string construction from multiple fields? | +| `semantic_divergence` | Number — does the state narrow, broaden, or redefine the canonical Ed-Fi meaning? | | `documentation_style` | `Prescriptive` \| `Conceptual` \| `Cross-reference` \| `Regulatory` \| `Unspecified` | -| `extension_necessity` | Boolean — does the element require a non-standard extension or descriptor override? | +| `extension_necessity` | String — does the element require a non-standard extension or descriptor override? | | `cited_spans` | List of verbatim excerpts from source documentation that support the above flags | -| `nachos_score` | Base NACHOS scalar (integer) | -| `adjusted_nachos` | NACHOS score after semantic-fidelity adjustment (integer) | +| `nachos_score` | Base NACHOS scalar (decimal/float) | +| `adjusted_nachos` | NACHOS score after semantic-fidelity adjustment (decimal / float) | | `adjustment_rationale` | Plain-text explanation of any adjustment applied | | `candidate_recommendations` | List of suggested simplification or standardization actions | | `firing_rule_path` | The ordered list of rules from the rule cascade that produced this score | @@ -214,64 +226,44 @@ The system SHALL run a Scoring LLM against each in-scope element record produced | `confidence` | Overall confidence in the score assignment (High / Medium / Low) | | `review_routing` | `Auto-accept` \| `Route-for-review` | -> [!NOTE] -> **NACHOS Score Context (peer signal).** Every scored row SHALL carry, alongside the scalar score: -> -> * **Implementation Shape** — structural complexity, deterministic from the Swagger/API model -> * **Documentation Style** — one of: Prescriptive, Conceptual, Cross-reference, Regulatory, Unspecified -> * **Documentation Gap** — flag for complex structures the documentation under-explains - -> [!NOTE] -> **Semantic-fidelity adjustment.** When a state narrows, broadens, or changes the canonical Ed-Fi meaning of an element, an adjustment SHALL be applied to the base NACHOS score. Exact adjustment magnitudes are subject to Alliance methodology review and SHALL be configurable without a code deployment. - #### 3. Rule Cascade and Score Overrides **Story:** As a user, I want to see all the rules and cascade elements that are triggered for each row, along with the associated NACHOS score, Adjusted NACHOS score, and the name of the tier matched, so that I can understand exactly how each score was derived. -**Story:** As a user, I want to adjust scoring rules for specific fields with a rationale, or add new logic specific to a state, and trigger rescoring — with the full history of scoring changes preserved — so that I can refine scores based on domain expertise without losing the prior scoring record. +Staff SHALL be able to inspect the full `firing_rule_path` and `tier_name` for any scored element to understand exactly which rules fired and which tier was matched. -The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` and `tier_name` fields in the evidence record. +--- -Staff SHALL be able to inspect the full `firing_rule_path` and `tier_name` for any scored element to understand exactly which rules fired and which tier was matched. +### 4. List of elements that present some complexity _(Not clear if this requirement is part of this PRD)_ + +Analysis of aggregate score data across states and across time is scoped for a future phase. Requirements will be defined separately. -#### 3.1 Logic alignment, rescoring and manual score overwrite +--- -* **Story** as an Ed-Fi ALliance Staff, I must be able to add a new business rule for a particular element or an entity, and re-run the LLM scoring, showing the old and new NACHOS and adjusted NACHOS scores, along with the rules that it triggers, and decide if the new scores are acceptable. -* -* **Story** Ed-Fi Alliance Staff MUST be able to override the assigned score based on human judgment, with the new scores and rationale recorded against the row's evidence record. +### Score Override -**Story:** As a user, I want to pull all scores, evidence records, rule overrides, and the full scoring history stored in the database so that any score is inspectable without re-running the model and all changes are traceable over time. +**Story:** As a user, I want to adjust scoring rules for specific fields with a rationale, or add new logic specific to a state, and trigger rescoring — with the full history of scoring changes preserved — so that I can refine scores based on domain expertise without losing the prior scoring record. -The system SHALL preserve a complete history of scoring changes per element, including the original machine-assigned score, each override or addition applied, the rationale provided, the staff member who made the change, and the resulting rescored values. +The scoring logic SHALL implement the rule cascade defined in the **POC Recommendation document (page 11)**. The rule cascade determines which combination of extracted flags maps to which NACHOS tier and drives the `firing_rule_path` and `tier_name` fields in the evidence record. -#### 3.2 Score and Override Storage +The system SHALL preserve a complete history of scoring changes per element, including the original machine-assigned score, each override or addition applied, the rationale provided, the staff member who made the change, and the resulting new values. -The system SHALL write scores and evidence records to the storage database upon completion of a scoring run, extending each element record with the following fields: +The system SHALL write scores and evidence records to the storage database upon completion of a rescoring, extending each element record with the following fields: | Field | Description | |---|---| | `scoring_run_id` | Unique identifier for the scoring run | -| `nachos_score` | Base NACHOS scalar | -| `adjusted_nachos` | Adjusted NACHOS scalar | -| `override_nachos_score` | Staff-assigned score (null if not overridden) | +| `nachos_score` | Staff-assigned Base NACHOS score (previous value if not overridden) | +| `adjusted_nachos` | Staff-assigned Adjusted NACHOS score (previous value if not overridden) | +| `former_nachos_score` | Former Base NACHOS score | | `override_rationale` | Staff-provided rationale for override (null if not overridden) | | `override_by` | Staff member who applied the override (null if not overridden) | | `scored_at` | Timestamp of score assignment | -| `override_adjusted_nachos_score` | Calculated over override_nachos_score (null if not overridden) | +| `former_adjusted_nachos_score` | Former Adjusted_nachos_score (null if not overridden) | TBD with Chris, Suganya: Should we label the nachos_score as OLD, and then update new nachos-score with the overwritten value. The evidence record SHALL be stored in full so that any score is inspectable without re-running the model. -The system SHALL maintain a scoring history table that records each override or state-specific rule addition, preserving the prior score, the change applied, the rationale, and the re-scored result. - ---- - -### 4. JTBD 3 — Score Evaluation Analytics _(TBD)_ - -Analysis of aggregate score data across states and across time is scoped for a future phase. Requirements will be defined separately. - ---- - ## 3. Non-Functional Requirements ### 3.1 Implemented Requirements @@ -282,7 +274,6 @@ Analysis of aggregate score data across states and across time is scoped for a f | NFR-DATA-2 | Provenance | Every ingestion record SHALL carry a `run_id`, `state_id`, and `created_at` timestamp | Applied at write time | | NFR-DATA-3 | Provenance | Every score SHALL carry a full evidence record persisted alongside the scalar | `evidence_record` JSON column on score table | | NFR-SEC-1 | Security | No student data or PII SHALL flow through any pipeline component | Input validation at ingestion form; static analysis gate in CI | -| NFR-ACC-1 | Accessibility | All views and forms SHALL meet WCAG 2.1 AA | Verified with automated accessibility checker in CI | ### 3.2 Quality Gates (Scoring Engine) @@ -298,73 +289,7 @@ Analysis of aggregate score data across states and across time is scoped for a f ## 4. Architecture -### 4.1 Technology Stack - -> [!NOTE: TO BE VALIDATED] -| Component | Technology | Notes | -|---|---|---| -| Ingestion LLM | Claude (Anthropic) | Configurable model; defaults to claude-sonnet-4-6 | -| Scoring LLM | Claude (Anthropic) | Same model pool; scoring prompt is distinct from ingestion prompt | -| Storage | PostgreSQL | Element catalog, evidence records, score tables | -| Job Queue | To be determined | Async execution of ingestion and scoring runs | -| API layer | To be determined | REST endpoints for form submissions, run status, report export | -| Frontend | To be determined | Input forms, progress views, report tables, scored element forms | - -### 4.2 Pipeline Overview - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ INGESTION PIPELINE (JTBD 1) │ -│ │ -│ [User: view / edit source list] │ -│ │ │ -│ ▼ │ -│ Parse & Validate Swagger ──► Entity Match to Ed-Fi Domain Registry │ -│ │ │ -│ [User: Business doc paths/URLs — Confluence, GitHub/Excel, │ -│ TWEDS, PDF, Excel, HTML] │ -│ │ │ -│ ▼ │ -│ Business Logic Extraction LLM (element-level, verbatim + │ -│ provenance pointer) │ -│ │ │ -│ ▼ │ -│ Scope Classification (in-scope: state-populated, vendor cannot │ -│ PUT/POST/DELETE; out-of-scope: state-populated, GET only) │ -│ │ │ -│ ▼ │ -│ Write to Storage DB (elements + business logic + scope flags) │ -└─────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────┐ -│ SCORING ENGINE (JTBD 3) │ -│ │ -│ Read in-scope elements from Storage DB │ -│ │ │ -│ ▼ │ -│ Scoring LLM ──► LLM-extracted facts + deterministic facts │ -│ │ computed from record structure │ -│ ▼ │ -│ Rule Cascade ──► NACHOS Score + Adjusted NACHOS + Tier Name │ -│ │ │ -│ ▼ │ -│ [User: inspect firing_rule_path + tier_name per row; │ -│ adjust rules or add state-specific logic; trigger rescore] │ -│ │ │ -│ ▼ │ -│ Write scores + evidence records + override history to Storage DB │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -### 4.3 Data Model (Key Tables) - -| Table | Primary Key | Key Columns | -|---|---|---| -| `ingestion_runs` | `run_id` | `state_id`, `created_at`, `approved_by`, `status` | -| `elements` | `element_id` | `run_id`, `state_id`, `entity_name`, `element_path`, `standard_entity`, `domain`, `in_scope`, `business_logic`, `cited_span` | -| `scoring_runs` | `scoring_run_id` | `run_id` (FK → ingestion_runs), `created_at`, `status` | -| `scores` | `score_id` | `element_id` (FK → elements), `scoring_run_id`, `nachos_score`, `adjusted_nachos`, `tier_name`, `evidence_record` (JSON), `override_score`, `override_by` | -| `scoring_history` | `history_id` | `score_id` (FK → scores), `prior_score`, `change_type`, `rationale`, `changed_by`, `rescored_at` | +For technology Stack, Pipeline Overview, Data Model and Environment Variables, please refer to the Data Map document -Add link here. --- @@ -379,45 +304,5 @@ The following are explicitly out of scope for this PRD: * Vendor-facing score exposure (internal Ed-Fi staff use only in this phase) * Natural language query interface (separate capability in the broader initiative) * Cluster analysis across states (separate capability) - ---- - -## 6. Backlog - -Active issues are tracked in the project issue tracker. The following items are known open questions as of this draft: - -| ID | Area | Question | Status | -|---|---|---|---| -| TBD | Scoring | Exact NACHOS adjustment magnitudes for semantic divergence cases | Pending Alliance methodology review | -| TBD | Ingestion | Handling of multi-version Swagger files for the same state | Open | -| TBD | Scoring | Evaluation set composition and reviewer assignment | Ed-Fi SHALL define before acceptance testing | -| TBD | Architecture | Job queue technology selection | Open | -| TBD | Reporting | Format and delivery mechanism for cross-state distribution reports | Open | - ---- - -## 7. Environment Variable Reference - -### 7.1 Ingestion Pipeline - -| Variable | Description | Default | -|---|---|---| -| `INGESTION_LLM_MODEL` | Claude model ID for business logic extraction | `claude-sonnet-4-6` | -| `INGESTION_LLM_MAX_TOKENS` | Max tokens per extraction call | `4096` | -| `INGESTION_MAX_SOURCES` | Maximum files/URLs per ingestion run | `10` | -| `INGESTION_SUPPORTED_DOC_FORMATS` | Comma-separated list of accepted doc extensions | `pdf,docx,xlsx,txt,html` | -| `DB_CONNECTION_STRING` | PostgreSQL connection string | _(required)_ | - -### 7.2 Scoring Engine - -| Variable | Description | Default | -|---|---|---| -| `SCORING_LLM_MODEL` | Claude model ID for scoring | `claude-sonnet-4-6` | -| `SCORING_LLM_MAX_TOKENS` | Max tokens per scoring call | `4096` | -| `SCORING_BATCH_SIZE` | Elements per LLM batch call | `25` | -| `NACHOS_ADJUSTMENT_CONFIG` | Path to YAML file defining semantic-fidelity adjustment magnitudes | `config/nachos-adjustments.yaml` | -| `RULE_CASCADE_CONFIG` | Path to YAML file defining the rule cascade | `config/rule-cascade.yaml` | -| `QG_TIER_ALIGNMENT_THRESHOLD` | Score-tier alignment quality gate threshold | `0.80` | -| `QG_F1_OVERALL_THRESHOLD` | Overall F1 quality gate threshold | `0.70` | -| `QG_F1_SIMPLE_THRESHOLD` | F1 quality gate threshold for score-zero rows | `0.90` | -| `QG_KAPPA_THRESHOLD` | Cohen's Kappa quality gate threshold | `0.60` | +* It is not clear if creating the list of elements that carry some complexity is part of this PRD. It seems to belong to dashboard-related processes + \ No newline at end of file From a496a6e90d1b4f51e19cecee2bef3d8796564c4d Mon Sep 17 00:00:00 2001 From: Maria Ragone Date: Tue, 7 Jul 2026 17:14:35 -0500 Subject: [PATCH 6/6] MC-10 Fixes to settings and other typos MC-10 Typo Fixes in settings.json , and fixed typo errors in the PRD for Ingestion and Scoring --- .vscode/settings.json | 2 +- docs/PRD-Ingestion-Scoring.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.vscode/settings.json b/.vscode/settings.json index ce6a10f..60758d4 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -93,6 +93,6 @@ "cSpell.words": [ "browsable", "inspectable", - "qualty" + "quality" ] } diff --git a/docs/PRD-Ingestion-Scoring.md b/docs/PRD-Ingestion-Scoring.md index 6e09bbc..4f53c88 100644 --- a/docs/PRD-Ingestion-Scoring.md +++ b/docs/PRD-Ingestion-Scoring.md @@ -8,7 +8,7 @@ ### 1.1 Background -The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Three high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, a process that involves conversations with the SEAs to align their speicifications to the Ed-Fi Data Standard (reduction in commplexity), and a **cluster analysis** that surfaces cross-state patterns. +The Data Standard Metadata Collection and Usage initiative collects, stores, and analyzes metadata about Ed-Fi Data Standard implementations — specifically the API specifications and supplemental data collection rules that state education agencies (SEAs) publish for vendors. Three high-value workflows sit at the center of the initiative: a **scoring engine** that measures the complexity of each SEA's data collection requirements, a process that involves conversations with the SEAs to align their specifications to the Ed-Fi Data Standard (reduction in complexity), and a **cluster analysis** that surfaces cross-state patterns. This PRD covers phase 2 of automated ingestion and NACHOS scoring. @@ -58,12 +58,12 @@ The database will be updated with all the ingestion and scoring information, inc | Term | Definition | |---|---| -| **State-specific swagger** | in this document, it refers to all the elements that are required by that state for state reporting, including extensions, and if these extensions are necessary or not along with rationale to why they are necessary| -| **Extension** | Any data element, including a whole entity, not listed in the Ed-Fi Data Standard in a specific version. For example, if the state uses Ed-Fi DS 4.0, then an extension refers to any element or entity not listed in the Ed-FI DS v4.0| -| **Source File** |Resulting artifact from the ingestion pipeline, which contains any data element of the data model listed by a state| +| **State-specific swagger** | in this document, it refers to all the elements that are required by that state for state reporting, including extensions, and if these extensions are necessary or not along with rationale to why they are necessary | +| **Extension** | Any data element, including a whole entity, not listed in the Ed-Fi Data Standard in a specific version. For example, if the state uses Ed-Fi DS 4.0, then an extension refers to any element or entity not listed in the Ed-FI DS v4.0 | +| **Source File** |Resulting artifact from the ingestion pipeline, which contains any data element of the data model listed by a state | | **Spine File** |Resulting artifact from the ingestion pipeline, which contains the business logic identified for any element by a particular state | | **Gap Log File** |Resulting artifact from the ingestion pipeline, which matches the elements that are both in the source and spine files. This artifact represents the elements from the state that have business requirements | -| **In Scope attribute** |Attribute that indicates if the data element needs to be populated by vendors via Pull, PUSH, GET in which case it is in-scope = true. If the element is populated by the state and only read by the vendor, then in-scope = false. Descriptors are in-scope| +| **In Scope attribute** |Attribute that indicates if the data element needs to be populated by vendors (in-scope = true). If the element is populated by the state and only read by the vendor, then in-scope = false. Descriptors are in-scope | ### 1.5 Target Users for this PRD @@ -266,7 +266,7 @@ The evidence record SHALL be stored in full so that any score is inspectable wit ## 3. Non-Functional Requirements -### 3.1 Implemented Requirements +### 3.1 Implementation Requirements | ID | Category | Requirement | Implementation | |---|---|---|---|