Candidate design: hosted scoring pipeline (JTBD 1)#2
Conversation
High-level design for operationalizing the NACHOS POC-3 scoring engine as an automated, hosted pipeline. Covers two integration approaches (repository-fed target state and a phased bridge), a compute platform comparison left open for team decision, a trigger and cost model against NFR-INFRA-2, and open questions proposed for the PRD decision log. Companion to the PRD under review in PR Ed-Fi-Alliance-OSS#1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
||
| * Status: Draft — candidate design for discussion, not settled architecture | ||
| * Author: Chris Moffatt (Touchdown Consulting) | ||
| * Relates to: [*Data Standard Metadata Collection and Usage* PRD (PR #1)](https://github.com/Ed-Fi-Alliance-OSS/Metadata-Catalog/pull/1) — JTBD 1 (Scoring Engine), JTBD 3 (Standardization of Data Collection Metadata), Phase 2 (Scoring and automated metadata extraction) |
There was a problem hiding this comment.
The full PRD has now been merged.
There was a problem hiding this comment.
Fixed in 7eaa417 — the metadata block now points at the merged docs/PRD.md.
Design Doc Review:
|
| Severity | Issue | Section |
|---|---|---|
| Critical | Review routing: 4-category POC vs. binary PRD field | §1.1 / PRD §2.2.1 |
| Critical | NACHOS scalar: non-integer range (0–4.5) vs. integer type in PRD | §1.1 / PRD §2.2.1 |
| Significant | "Integration Profile" vs. "NACHOS Score Context" / "Implementation Shape" naming | §1.1, §2 |
| Significant | 22-field input contract vs. ~15-field storage schema in ingestion PRD | §2 / PRD §2.1.5 |
| Moderate | Six open questions (D-1–D-6) not added to either PRD's backlog | §9 |
| Moderate | Semantic-fidelity: "justification string" vs. separate adjusted_nachos + adjustment_rationale fields |
§2 |
| Moderate | TWEDS: headless-browser scrape vs. REST API in ingestion PRD | §4 / PRD §2.1.1 |
| Moderate | PRD link points to the PR, not the merged file | Metadata block |
| Minor | NFR-TEST-1 80% coverage threshold not confirmed for existing test suite | §8.5 |
| Minor | Quality gate acceptance metrics not cross-referenced | §8.5–8.6 |
| Minor | Section numbering starts at §0 | Throughout |
Architecture Review — PR #2: Candidate design: hosted scoring pipeline (JTBD 1)Note Review by Claude Code (Sonnet 4.6) with light editing by Stephen.
Architecture Model Used
Layers (from PRD §2 and §5):
Key rules referenced in this review:
🔴 CriticalNone found. 🟠 HighNone found. 🟡 MediumTerminology drift: "Integration Profile" vs. "Score Context" Note Are "Integration Profile" and "Score Context" truly referring to the same thing? I need to read more carefully myself to validate. Either way, there might be some term confusion to address. Review-queue routing labels not reconciled with JTBD 14 🔵 LowOQ-1 (OpenMetadata) not acknowledged as a predecessor decision Note SF: OpenMetadata has now been rejected as an option. GitHub Actions as production scheduler — NFR-CICD-1 conformance needs explicit sign-off Note SF: Interesting observation. GitHub Actions is likely to become a scheduling engine in some regard, but we haven't yet worked through the security implications and guardrails. New adapter code coverage against NFR-TEST-1 not addressed ⚪ Nit
Note SF: I'm always fighting back against the AI tendency to add horizontal lines before headings. For me, the heading is sufficient. But I'm not so concerned that I've put that into §8.6 framing slightly ambiguous against the PRD's distributions mandate — The PRD's cross-state distribution requirement (JTBD 1 SHALL) is not negotiable; §8.6 could be read as leaving it open. Clarify that distributions are required regardless of how the primary display element is rendered. D-4 (LLM provider ownership) should cross-reference NFR-CICD-3/4 — Both NFRs already assign system maintenance and monitoring to Ed-Fi Alliance Staff; citing them in D-4 ties the open question to existing obligations. VerdictArchitectural fitness: ALIGNED
Overall: This is a thorough, PRD-consistent candidate design that maps the POC's production-ready properties directly to JTBD 1 acceptance criteria, making the evidence-record and override gaps explicit and addressing them. The two Medium findings (terminology reconciliation, review-queue routing vs. JTBD 14 status model) are not blockers but should be resolved before the design is used to drive implementation work, so engineering doesn't build two conflicting naming systems. 🤖 Generated with Claude Code |
…July status) June revisions that never made it into the PR copy: PRD reference updated to the merged docs/PRD.md; Integration Profile renamed to the PRD's Score Context term; the D-1 schema-gap note against the ingestion PRD storage table (PR Ed-Fi-Alliance-OSS#3 s2.1.5); semantic-fidelity modeled as enum + rationale. July status updates (dated [2026-07 update] notes): the POC now encodes the operator runbook in code (poc3 publish stage registry + freshness manifest + fail-loudly readers - the hosted job wraps the registry); the s8.4 human-override capture half exists (workbook round-trip into committed curation records the repository migration must ingest); D-3 gained the consolidated deliverable surface and the workbook-as-write- surface constraint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
|
Pushed a revision (2026-07-02) with two batches of updates: June alignment edits that hadn't made it into this copy: the PRD reference now points at the merged Dated
No changes to the platform comparison (§6) or the open-questions table beyond the D-3 annotation. |
…S access notes - Renumber sections 0-9 to 1-10 to match the parent PRD convention (review: Minor) - Schema-type callout in section 3: adjusted NACHOS is 0-4.5 in half-steps (base tier is the 0-3 integer); PR Ed-Fi-Alliance-OSS#3 section 2.2.1 currently types both as integer - the adjusted field needs a decimal type (review: Critical) - TWEDS access reconciliation in section 5: the TWEDSAPI endpoint serves a server-rendered .NET app, not an XML/REST feed - fresh pulls are version-pinned portal scrapes with a headless browser for entity detail pages; folded into D-2 (review: Moderate; PR Ed-Fi-Alliance-OSS#3 section 2.1.1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
|
Thanks Stephen. Both reviews became the checklist for the last two pushes ( Section numbers: per your last Minor finding the doc now numbers §1–§10 (was §0–§9), so every section shifted +1. I cite new numbers with the old in parentheses; the PR description is updated to match. Design-doc review
Architecture review
PRD-side fixes for PR #3PR #3 is still open, so it can absorb these directly: (a) |
What this adds
docs/design/scoring-pipeline.md— a candidate high-level design for operationalizing the NACHOS scoring engine (PRD JTBD 1, Phase 2) as an automated, hosted pipeline. It is offered in the same spirit as the PRD's System Architecture section: to guide later design work, not to assert settled architecture.Companion to the PRD under review in #1 — it uses the PRD's JTBD/NFR vocabulary throughout, so reading the PRD first helps.
Summary
Notes for reviewers
🤖 Generated with Claude Code