Skip to content

Candidate design: hosted scoring pipeline (JTBD 1)#2

Open
touchdownllc wants to merge 4 commits into
Ed-Fi-Alliance-OSS:mainfrom
touchdownllc:design/hosted-scoring-pipeline
Open

Candidate design: hosted scoring pipeline (JTBD 1)#2
touchdownllc wants to merge 4 commits into
Ed-Fi-Alliance-OSS:mainfrom
touchdownllc:design/hosted-scoring-pipeline

Conversation

@touchdownllc

@touchdownllc touchdownllc commented Jun 11, 2026

Copy link
Copy Markdown

What this adds

docs/design/scoring-pipeline.md — a candidate high-level design for operationalizing the NACHOS scoring engine (PRD JTBD 1, Phase 2) as an automated, hosted pipeline. It is offered in the same spirit as the PRD's System Architecture section: to guide later design work, not to assert settled architecture.

Companion to the PRD under review in #1 — it uses the PRD's JTBD/NFR vocabulary throughout, so reading the PRD first helps.

Summary

  • What the POC already is (§2): an extract-then-rule pipeline — the LLM extracts narrow facts with verbatim evidence spans; deterministic rules compose scores. Several production-grade properties already exist (idempotent resumable runs, prompt-cache-as-checkpoint, batch transport, cost caps, versioning discipline, ~1,700-test suite), alongside the properties that must change (local filesystem, human-sequenced stages, fused ingestion).
  • The central design move (§3): a schema contract on both sides of the scorer. The POC's element record maps to the JTBD 3 (resource, property) grain, and its per-record evidence sidecar aligns with the JTBD 1 evidence-record acceptance criteria nearly clause for clause — the one gap (staff override) is designed in §9.4.
  • Two integration approaches: Approach A (§4) — a repository-fed scoring service, the target state; Approach B (§5) — a phased bridge that containerizes the POC chain first and swaps the input side later, built against the same adapter contracts so the swap is not a rewrite.
  • Compute platform comparison (§7): Azure Container Apps Jobs vs GitHub Actions-as-runner vs Durable Functions, left open for the team's decision.
  • Cost model (§8) against NFR-INFRA-2, from POC actuals.
  • Open questions (§10) proposed as additions to the PRD decision log.

Notes for reviewers

  • The source POC repo (touchdownllc/nachos-ai-poc-3) is private; project team members have access, so in-doc links to its methodology docs and ADRs will resolve for you but not for other readers. Transfer to the Ed-Fi Alliance org under Apache-2.0 is assumed in the doc as a prerequisite of this work.
  • Happy to restructure, split, or fold parts of this into the PRD itself if that serves the Initial PRD #1 review better.

🤖 Generated with Claude Code

High-level design for operationalizing the NACHOS POC-3 scoring engine
as an automated, hosted pipeline. Covers two integration approaches
(repository-fed target state and a phased bridge), a compute platform
comparison left open for team decision, a trigger and cost model
against NFR-INFRA-2, and open questions proposed for the PRD decision
log. Companion to the PRD under review in PR Ed-Fi-Alliance-OSS#1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Jun 11, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@stephenfuqua stephenfuqua requested a review from CSR2017 June 23, 2026 15:24
Comment thread docs/design/scoring-pipeline.md Outdated

* Status: Draft — candidate design for discussion, not settled architecture
* Author: Chris Moffatt (Touchdown Consulting)
* Relates to: [*Data Standard Metadata Collection and Usage* PRD (PR #1)](https://github.com/Ed-Fi-Alliance-OSS/Metadata-Catalog/pull/1) — JTBD 1 (Scoring Engine), JTBD 3 (Standardization of Data Collection Metadata), Phase 2 (Scoring and automated metadata extraction)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full PRD has now been merged.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7eaa417 — the metadata block now points at the merged docs/PRD.md.

@stephenfuqua

Copy link
Copy Markdown
Contributor

Design Doc Review: docs/design/scoring-pipeline.md

This review checks the candidate design document for clarity and consistency with docs/PRD.md (the initiative PRD) and PRD-Ingestion-Scoring.md (PR #3, the Phase 2 detail PRD). JTBD numbering in this document is correct — unlike PR #3, it consistently identifies JTBD 1 as the Scoring Engine and JTBD 3 as Standardization.


Critical: Review Routing Definitions Conflict with PRD-Ingestion-Scoring.md

The design doc (§1.1 review queue) routes flagged rows to one of four buckets: POLICY / DATA_MODEL / SCORING / ANALYST.

PR #3 (PRD-Ingestion-Scoring.md §2.2.1) defines review_routing as a binary field: Auto-accept | Route-for-review.

These are inconsistent. The four-category scheme from the POC appears richer and more actionable than the binary flag in the PRD. The PRD should be updated to reflect the actual routing design, or the design doc should explain how the four POC categories collapse to the two-value PRD field.


Critical: NACHOS Scalar Described as Non-Integer in the Design Doc, Integer in the PRD

The design doc (§1.1) defines the NACHOS scalar as 0–4.5: complexity tier 0–3 plus extension adjustments, implying fractional values.

PRD-Ingestion-Scoring.md §2.2.1 defines both nachos_score and adjusted_nachos as integer type.

These cannot both be correct. If extension adjustments produce values like 3.5 or 4.5, the PRD's schema is wrong. If scores are always integers, the design doc's range description is misleading. One of them needs to be corrected and the other updated to match.


Significant: "Integration Profile" vs. "NACHOS Score Context" Naming Mismatch

The design doc uses the term Integration Profile to describe the two-axis output (structural depth × documentation style, plus documentation-gap flag). Both docs/PRD.md and PRD-Ingestion-Scoring.md use NACHOS Score Context with components called Implementation Shape and Documentation Style.

These appear to be the same concept under different names. The design doc should use the PRD's terminology, or the PRD should be updated — but the current state leaves a reader unsure whether these are the same thing or two different artifacts.


Significant: Scoring Input Contract May Exceed the Planned Storage Schema

Section 2 states the scoring engine needs the POC's "22-field element record" as its input contract and lists structural fields including "canonical Ed-Fi mapping, extension attribution, descriptor enumeration, reference shape, FK chain depth, reference fan-out, sub-collections."

PRD-Ingestion-Scoring.md §2.1.5 defines the storage schema with approximately 15 fields. Fields for FK chain depth, reference fan-out, sub-collection shape, and extension attribution are not listed.

If those fields are not in the storage schema when the scoring engine lands, Approach A cannot execute as described. Open question D-1 in §9 partially covers this ("Does the JTBD 3 schema carry the full scoring input contract?") but the gap between the 22-field requirement and the 15-field PRD schema deserves an explicit callout — and ideally the missing fields should be added to the PRD's storage table or flagged as a design open question there.


Moderate: Open Questions in §9 Not Reflected in Either PRD's Backlog

The document proposes six open questions (D-1 through D-6) for addition to the PRD decision log, but neither docs/PRD.md nor PRD-Ingestion-Scoring.md currently contains them. If these questions are genuinely blocking (D-1, D-2, D-5 especially), they should be added to the Backlog/Open Questions section of the relevant PRD, not just listed here in the design doc.


Moderate: Semantic-Fidelity Adjustment Described Differently from PRD Schema

In §2 (output contract table), semantic-fidelity adjustment is described as "Scope-delta / fidelity-divergence adjustments carried in the score justification string."

PRD-Ingestion-Scoring.md §2.2.1 models this as two distinct fields: adjusted_nachos (an integer) and adjustment_rationale (plain text). The design doc's description of a single "justification string" doesn't map cleanly to a separate adjusted scalar plus a rationale field.


Moderate: TWEDS Access Method Inconsistency

The design doc (§4, Phase B1 notes) says fresh TWEDS scrapes "need a headless browser." PRD-Ingestion-Scoring.md §2.1.1 lists TWEDS as supporting "Versioned XML artifacts or REST API access." These are different access mechanisms for the same source. The headless-browser approach (scraping the documentation portal) and the REST API approach (the TWEDS API) should be reconciled, or the PRD should explicitly acknowledge both methods and document which is preferred.


Moderate: PRD Link Points to the PR, Not the Merged File

The metadata block links to "PRD (PR #1)" at the pull request URL. PR #1 is now merged and the PRD lives at docs/PRD.md. Update the link to point to the file path in the main branch, or use the GitHub blob URL, so the reference remains stable after the PR is closed.


Minor: NFR-TEST-1 Coverage Threshold Not Confirmed for the Existing Test Suite

§8.5 says the ~1,700-test suite "becomes the CI gate per NFR-TEST-1." NFR-TEST-1 requires at least 80% unit test coverage. The design doc does not state whether the current test suite meets that threshold. Worth confirming — or noting it as something to verify during the transfer to the Ed-Fi org.


Minor: Quality Gate Metrics Not Cross-Referenced

docs/PRD.md JTBD 1 specifies acceptance metrics: ≥80% score-tier alignment, ≥0.7 F1 overall, ≥0.9 F1 for score-zero rows, ≥0.6 Cohen's Kappa. The design doc mentions the POC achieved 80.2% exact-match (§1.1) but doesn't map the POC's calibration state result to these four quality gates or indicate whether the current test suite covers them. A brief cross-reference in §8.5 or §8.6 would help the reader understand how production acceptance testing relates to what the POC already measured.


Minor: Section Numbering Starts at §0

The document opens with ## 0. Purpose and the purpose section says it covers §1–§9. Starting at §0 is unusual and mildly confusing since the table in §0 says it covers things in §1–§9, while §0 itself is the orientation section. Renumbering to start at §1 (with §1 = Purpose) would follow the parent PRD's convention.


Summary

Severity Issue Section
Critical Review routing: 4-category POC vs. binary PRD field §1.1 / PRD §2.2.1
Critical NACHOS scalar: non-integer range (0–4.5) vs. integer type in PRD §1.1 / PRD §2.2.1
Significant "Integration Profile" vs. "NACHOS Score Context" / "Implementation Shape" naming §1.1, §2
Significant 22-field input contract vs. ~15-field storage schema in ingestion PRD §2 / PRD §2.1.5
Moderate Six open questions (D-1–D-6) not added to either PRD's backlog §9
Moderate Semantic-fidelity: "justification string" vs. separate adjusted_nachos + adjustment_rationale fields §2
Moderate TWEDS: headless-browser scrape vs. REST API in ingestion PRD §4 / PRD §2.1.1
Moderate PRD link points to the PR, not the merged file Metadata block
Minor NFR-TEST-1 80% coverage threshold not confirmed for existing test suite §8.5
Minor Quality gate acceptance metrics not cross-referenced §8.5–8.6
Minor Section numbering starts at §0 Throughout

@stephenfuqua

stephenfuqua commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Architecture Review — PR #2: Candidate design: hosted scoring pipeline (JTBD 1)

Note

Review by Claude Code (Sonnet 4.6) with light editing by Stephen.

No issue linked.
Branch: design/hosted-scoring-pipelinemain
Note: Worktree checkout failed (branch not present on remote). Review is based on the PR diff and local main branch documentation. Full codebase context unavailable.

Architecture Model Used

Sources: docs/PRD.md (primary), docs/superpowers/specs/2026-06-02-prd-phasing-design.md, docs/superpowers/plans/2026-06-02-prd-clarity-phasing.md

Layers (from PRD §2 and §5):

  • Cataloging layer — sources (Ed-Fi Data Standard, SEA spec documents) feed the metadata repository via extraction/enrichment services (JTBD 4, 6)
  • Metadata repository — central store (JTBD 7/3); must preserve provenance, confidence, and curation status
  • Analysis/usage layer — Scoring Engine (JTBD 1), Cluster Analysis (JTBD 2), Dashboards/Reports (JTBD 12/13), Opportunity Tracking (JTBD 11); read/write the repository

Key rules referenced in this review:

  1. JTBD 1 acceptance: evidence record per row; Score Context (Implementation Shape / Documentation Style / Documentation Gap); staff override with rationale; cross-state distributions; ≥ 80% tier alignment; ≥ 0.7 F1; ≥ 0.6 Cohen's Kappa
  2. JTBD 3: schema at (resource, property) grain; retained source narrative for cited-span validation; provenance and curation status per element
  3. JTBD 14: review queue with pending/approved/rejected/updated status model
  4. NFR-TECH-2: scoring/ETL backend must use Python (preferred), JS, or C#
  5. NFR-BACKEND-1: async, retry with exponential backoff, circuit breaker
  6. NFR-INFRA-1/2: Azure managed/container services preferred; ≤ $100/month rule of thumb
  7. NFR-CICD-1: GitHub Actions conformant with Ed-Fi Alliance guidelines
  8. NFR-CICD-2: deployable to localhost (Docker Compose), staging, production
  9. NFR-TEST-1: ≥ 80% unit test coverage on custom code
  10. NFR-LICENSE-1: Apache-2.0
  11. PRD OQ-1: OpenMetadata evaluation still open; bespoke system is a working assumption, not a settled decision

🔴 Critical

None found.

🟠 High

None found.

🟡 Medium

Terminology drift: "Integration Profile" vs. "Score Context"
Rule violated: Rule 1 (JTBD 1 — Score Context terminology) — docs/PRD.md, JTBD 1, "NACHOS Score Context (peer signal)"
Location: §2 (the alignment table), §1.1 stage table
Detail: The PRD defines the three mandatory peer-signal components as Score Context (Implementation Shape / Documentation Style / Documentation Gap). The design introduces Integration Profile as the POC term and maps it via a table in §2, but never resolves which label the product will use. Readers of JTBD 12 dashboards, JTBD 14 review queues, and future PRD amendments will encounter both terms. If the product adopts "Integration Profile," the PRD needs updating; if it adopts "Score Context," the design should commit to that convention going forward.
Fix: Add a sentence at the end of §2 stating the chosen product-facing name (suggest "Score Context," as that is what the PRD mandates), and note "Integration Profile" as the internal POC label being retired.

Note

Are "Integration Profile" and "Score Context" truly referring to the same thing? I need to read more carefully myself to validate. Either way, there might be some term confusion to address.

Review-queue routing labels not reconciled with JTBD 14
Rule violated: Rule 3 (JTBD 14 — status model: pending / approved / rejected / updated) — docs/PRD.md, JTBD 14
Location: §1.2 ("review routing") and §8.4
Detail: The design introduces a four-route taxonomy (POLICY / DATA_MODEL / SCORING / ANALYST) for flagging rows for human attention. JTBD 14 specifies a distinct status lifecycle (pending → approved / rejected / updated). It is unclear whether these are separate concerns (routing = why a row needs review; status = disposition after review) or whether the routing labels are expected to replace or extend JTBD 14's statuses. Left unreconciled, the two schemas could produce a UI where staff see inconsistent or redundant state labels.
Fix: Add a note in §8.4 stating how the routing label relates to the JTBD 14 status model — e.g., "routing determines which queue a row enters; the JTBD 14 status (pending/approved/rejected/updated) tracks its disposition within that queue."

🔵 Low

OQ-1 (OpenMetadata) not acknowledged as a predecessor decision
Rule violated: Rule 11 (PRD OQ-1 still open) — docs/PRD.md §7, OQ-1
Location: §0 (Purpose)
Detail: PRD OQ-1 asks whether OpenMetadata could satisfy any JTBDs; it remains Open. This design assumes a bespoke system throughout. That assumption is consistent with the PRD's own working assumption, but the document doesn't acknowledge that Approach A is contingent on OQ-1 being resolved in favor of bespoke. A reader unfamiliar with the PRD's open questions section could interpret this design as settled architecture.
Fix: Add a one-sentence note in §0 or §3 that this design presupposes the bespoke-system path and that OQ-1 should be closed before committing to Approach A.

Note

SF: OpenMetadata has now been rejected as an option.

GitHub Actions as production scheduler — NFR-CICD-1 conformance needs explicit sign-off
Rule violated: Rule 7 (NFR-CICD-1 — GitHub Actions conformant with Alliance guidelines) — docs/PRD.md, NFR-CICD-1
Location: §6 (GitHub Actions as runner row)
Detail: The design correctly flags "public-repo hygiene needs care" for the GitHub Actions approach. Running a production job scheduler in a public repo — handling API keys via OIDC federation — is meaningfully different from CI use, which is what the Alliance's GitHub Actions guidelines govern. The B1-on-Actions path should require explicit confirmation that the Alliance's guidelines permit this usage before it is selected, not just a technical assessment.
Fix: Add a note under the GitHub Actions row in §6 (or in §4 Phase B1) that this path requires a governance check against NFR-CICD-1 before adoption.

Note

SF: Interesting observation. GitHub Actions is likely to become a scheduling engine in some regard, but we haven't yet worked through the security implications and guardrails.

New adapter code coverage against NFR-TEST-1 not addressed
Rule violated: Rule 9 (NFR-TEST-1 — ≥ 80% unit test coverage on custom code) — docs/PRD.md, NFR-TEST-1
Location: §8.5
Detail: §8.5 promotes the existing ~1,700-test suite as the CI gate, which covers the unchanged scoring core. It does not address coverage requirements for the new repository adapter (the read-input-contract / write-sidecar code). This is net-new custom code and falls squarely under NFR-TEST-1.
Fix: Add a sentence in §8.5 committing the repository adapter to ≥ 80% unit test coverage per NFR-TEST-1, separate from the inherited POC test suite.

⚪ Nit

--- horizontal rules used before every ## section header — minor visual inconsistency with docs/PRD.md, which doesn't follow this pattern.

Note

SF: I'm always fighting back against the AI tendency to add horizontal lines before headings. For me, the heading is sufficient. But I'm not so concerned that I've put that into CLAUDE.md or AGENTS.md yet 😆 . This does NOT need to be changed in this PR.

§8.6 framing slightly ambiguous against the PRD's distributions mandate — The PRD's cross-state distribution requirement (JTBD 1 SHALL) is not negotiable; §8.6 could be read as leaving it open. Clarify that distributions are required regardless of how the primary display element is rendered.

D-4 (LLM provider ownership) should cross-reference NFR-CICD-3/4 — Both NFRs already assign system maintenance and monitoring to Ed-Fi Alliance Staff; citing them in D-4 ties the open question to existing obligations.


Verdict

Architectural fitness: ALIGNED

Dimension Assessment
Layering ✅ — Scoring pipeline correctly sits above the metadata repository; ingestion/extraction cleanly separated
Module boundaries ✅ — Repository adapter pattern explicitly scopes coupling to two contracts
Dependency direction ✅ — Scoring reads from JTBD 3/4/6 outputs; writes back to repository; no upward dependencies
Pattern consistency ✅ — Approaches A and B both conform to PRD's phased delivery model; Approach B's bridge condition is well-guarded
ADR adherence ✅ — No relevant ADRs in this repo; PRD NFRs are systematically cited throughout
Naming & conventions ⚠️ — "Integration Profile" / "Score Context" terminology needs resolution before implementation

Overall: This is a thorough, PRD-consistent candidate design that maps the POC's production-ready properties directly to JTBD 1 acceptance criteria, making the evidence-record and override gaps explicit and addressing them. The two Medium findings (terminology reconciliation, review-queue routing vs. JTBD 14 status model) are not blockers but should be resolved before the design is used to drive implementation work, so engineering doesn't build two conflicting naming systems.

🤖 Generated with Claude Code

touchdownllc and others added 2 commits June 25, 2026 12:57
…July status)

June revisions that never made it into the PR copy: PRD reference updated
to the merged docs/PRD.md; Integration Profile renamed to the PRD's Score
Context term; the D-1 schema-gap note against the ingestion PRD storage
table (PR Ed-Fi-Alliance-OSS#3 s2.1.5); semantic-fidelity modeled as enum + rationale.

July status updates (dated [2026-07 update] notes): the POC now encodes
the operator runbook in code (poc3 publish stage registry + freshness
manifest + fail-loudly readers - the hosted job wraps the registry);
the s8.4 human-override capture half exists (workbook round-trip into
committed curation records the repository migration must ingest); D-3
gained the consolidated deliverable surface and the workbook-as-write-
surface constraint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
@touchdownllc

Copy link
Copy Markdown
Author

Pushed a revision (2026-07-02) with two batches of updates:

June alignment edits that hadn't made it into this copy: the PRD reference now points at the merged docs/PRD.md; the POC's internal Integration Profile term is retired in favor of the PRD's Score Context; a note under the input contract names the concrete D-1 schema gap against the ingestion PRD's storage table (PR #3 §2.1.5 — structural-depth inputs missing); semantic fidelity modeled as a small enum + rationale rather than a boolean.

Dated [2026-07 update] status notes reflecting July work in the POC that changes this design's starting point:

  • The "human as the orchestrator" pain point is resolved POC-side: a poc3 publish command now encodes the full chain as a stage registry (produces/consumes declared as data, ordering constraints structurally tested — including one the runbooks had never documented), with an external freshness manifest and readers that fail loudly on stale inputs. The hosted job can wrap that registry rather than re-derive the sequence, which shrinks the §3/§4 build.
  • The §8.4 human-override gap in the JTBD-1 table is half-closed: analyst workbooks now carry approval + score-override columns that round-trip into committed curation records (value, author, timestamp; re-applied on every regeneration, never overwritten; disagreement surfaces at the top of the Review Queue). The design's remaining job is migrating those records into the repository's curation table.
  • D-3 (workbooks long-term) gained new facts: the deliverable surface consolidated to one workbook per state + one combined, and the workbook is now a write surface — any export story must stay paired with the ingest path.

No changes to the platform comparison (§6) or the open-questions table beyond the D-3 annotation.

…S access notes

- Renumber sections 0-9 to 1-10 to match the parent PRD convention (review: Minor)
- Schema-type callout in section 3: adjusted NACHOS is 0-4.5 in half-steps
  (base tier is the 0-3 integer); PR Ed-Fi-Alliance-OSS#3 section 2.2.1 currently types both as
  integer - the adjusted field needs a decimal type (review: Critical)
- TWEDS access reconciliation in section 5: the TWEDSAPI endpoint serves a
  server-rendered .NET app, not an XML/REST feed - fresh pulls are
  version-pinned portal scrapes with a headless browser for entity detail
  pages; folded into D-2 (review: Moderate; PR Ed-Fi-Alliance-OSS#3 section 2.1.1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkkWmnhF7LtSY2KUSY7Qrn
@touchdownllc

Copy link
Copy Markdown
Author

Thanks Stephen. Both reviews became the checklist for the last two pushes (7eaa417 and ab05082, both 2026-07-02). Everything is addressed, either in the doc or exported to PR #3 where the fix belongs on the PRD side. Point-by-point below.

Section numbers: per your last Minor finding the doc now numbers §1–§10 (was §0–§9), so every section shifted +1. I cite new numbers with the old in parentheses; the PR description is updated to match.

Design-doc review

  1. Review routing: four categories vs. binary PRD field (Critical). These are orthogonal axes, and §9.4 (was §8.4) now says so: the route (POLICY / DATA_MODEL / SCORING / ANALYST) is why a row is queued and to whom; JTBD 14's status (pending / approved / rejected / updated) tracks its disposition once reviewed. The PRD's binary Auto-accept | Route-for-review is the coarse form of the routing signal. The doc now says the PRD should adopt the four routes or define the collapse explicitly; that fix is flagged for PR MC-10-Draft PRD-Ing-Scoring #3 §2.2.1.

  2. NACHOS scalar: 0–4.5 vs. integer (Critical). The POC is the tiebreaker: the adjusted score is fractional. Extension and multi-entity adjustments apply in 0.5 steps, so adjusted_nachos spans 0–4.5 in half-step increments; the base complexity tier (nachos_score) is the 0–3 integer. New callout in §3 (ab05082); the type fix (adjusted field to decimal) belongs in PR MC-10-Draft PRD-Ing-Scoring #3 §2.2.1.

  3. "Integration Profile" vs. "Score Context" (Significant). Adopted Score Context as the product-facing term throughout; "Integration Profile" appears once in §2.1 as the retired internal POC label. To answer the note in your architecture review: yes, they were the same artifact, the two-axis structural-depth × documentation-style output with the documentation-gap flag. The rename has also landed product-side in the POC's July work: the analyst-facing workbook surfaces now say NACHOS Score Context / Implementation Shape.

  4. 22-field input contract vs. ~15-field storage schema (Significant). Agreed this is the substantive one: a real dependency, not wording. §3 now carries an explicit callout ("Schema gap to close (D-1)") naming the missing structural fields: FK-chain depth, reference fan-out, sub-collection shape, extension attribution. Approach A cannot compute its structural facts until the JTBD 3 schema carries them. The concrete field additions are proposed for PR MC-10-Draft PRD-Ing-Scoring #3 §2.1.5.

  5. D-1–D-6 not in a PRD backlog (Moderate). Agreed they belong in the PRD decision log, not just here. Question for you: would you rather these land as a small follow-up PR against docs/PRD.md's open-questions section, or in PR MC-10-Draft PRD-Ing-Scoring #3's backlog while it's still open? Happy to open either.

  6. Semantic fidelity: "justification string" vs. separate fields (Moderate). Reworded in §3: it's a semantic-fidelity class (aligned / divergent-explained / divergent-unclear) driving tiered adjustment magnitudes, plus a plain-text rationale. The PRD side needs a small enum plus rationale, not a boolean; also flagged for PR MC-10-Draft PRD-Ing-Scoring #3 §2.2.1.

  7. TWEDS: scrape vs. "versioned XML / REST API" (Moderate). Clarified from POC experience in §5 (ab05082): the TWEDSAPI endpoint serves a server-rendered .NET app (no consumable XML/JSON feed, despite the URL prefix), so fresh pulls are version-pinned portal scrapes, with a headless browser for the entity detail pages. The PRD's description of this source should reflect the scrape as the current access path; reconciling it is folded into D-2 and flagged for PR MC-10-Draft PRD-Ing-Scoring #3 §2.1.1.

  8. PRD link points at the PR (Moderate + your inline comment). Fixed. The metadata block points at the merged docs/PRD.md.

  9. NFR-TEST-1 coverage (Minor + architecture-review Low). §9.5 (was §8.5) now carries both halves: the inherited ~1,700-test suite's measured coverage against the 80% bar is an explicit verify-on-transfer item, not asserted; the repository adapter, as net-new custom code, commits to its own ≥80% unit coverage per NFR-TEST-1, with contract tests against the JTBD 3 schema.

  10. Quality-gate cross-reference (Minor). Added in §9.6, with one caveat: the POC's measured calibration (agreement %, tier confusion matrix, tier-delta distributions against single-reviewer human scores) maps directly onto the ≥80% tier-alignment gate, but F1 and Cohen's Kappa don't. F1 needs a defined positive class and ground truth; kappa needs two independent raters. Those two are flagged as an open methodology item rather than gates the POC already clears.

  11. Section numbering (Minor). Renumbered to §1–§10 (ab05082).

Architecture review

  • Terminology drift (Medium) — see MC-10-Draft PRD-Ing-Scoring #3 above.
  • Routing vs. JTBD 14 (Medium) — see Initial PRD #1; the orthogonality framing you suggested is in §9.4 nearly verbatim.
  • OQ-1 / OpenMetadata (Low) — thanks for the update. §7 now states the bespoke-repository path is settled, with schema readiness (D-1) as the remaining open piece.
  • GitHub Actions as scheduler (Low) — §7 now makes the B1-on-Actions path contingent on an Alliance security/governance review against NFR-CICD-1 (OIDC-federated secrets, log and audit hygiene) before it is selected.
  • Adapter coverage (Low) — covered under #9.
  • Nits — cross-state distributions are now stated as required regardless of how the primary display element renders (§9.6); D-4 cross-references NFR-CICD-3/4; horizontal rules left alone per your note.

PRD-side fixes for PR #3

PR #3 is still open, so it can absorb these directly: (a) adjusted_nachos integer → decimal (0–4.5 in 0.5 steps), §2.2.1; (b) review_routing binary → the four routes, or define the collapse, §2.2.1; (c) semantic fidelity boolean → small enum + rationale, §2.2.1; (d) add the structural-depth fields to the §2.1.5 storage table; (e) TWEDS access description, §2.1.1. I'll raise these on PR #3 unless you'd rather batch them differently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants