[DMS-1129] Design SQL Server foreign-key pruning for identity-update cascades

[samuellugo-ship-it]

Summary

Spike DMS-1129 — Design foreign key pruning. Design/analysis only; no production code. This PR is documentation: it replaces the SQL Server "every reference composite FK uses ON UPDATE NO ACTION + MssqlIdentityPropagationTrigger" strategy with ODS-style foreign-key pruning, and fixes the flaw ODS has (silently pruning when no edge is safe to prune).

Design note: reference/design/backend-redesign/design-docs/mssql-cascading.md (includes diagrams of the objects, propagation direction, and each pruning/fail-fast decision point).

The approach

• The cascade graph is analyzed in propagation direction — referenced/parent table → referrer/child table. SQL Server error 1785 forbids a table from appearing more than once in a single UPDATE/DELETE's cascade action tree (the legal graph is a multitree): the conflict is a diamond (a receiver reached by two distinct paths from one origin) or a cycle — not raw cascade in-degree. Direct in-degree > 1 is only a candidate signal: independent parents into one receiver are legal and are never pruned (pruning an independent live edge would only reintroduce error 547). Detection is per-origin duplicate reachability (shared cascade ancestry), not per referenced parent.
• Cascade-eligibility and liveness use transitive identity mutability (IsAbstract || TransitivelyAllowIdentityUpdates), matching TransitiveIdentityMutabilityPass / ReferenceConstraintPass.
• Every emitted SQL Server reference FK keeps the full composite key (identity columns restored — this closes the DMS-1002 referential-integrity weakness). Per-edge outcome (MssqlPropagationMode):
  • NativeCascade — an eligible edge (an independent edge, the surviving path of a diamond, or the sole edge into a receiver); ON UPDATE CASCADE.
  • NoPropagation — a reconverging edge of a diamond that is pruned but covered by the surviving path for the same origin and canonical columns; ON UPDATE NO ACTION.
  • ImmutableNoAction — a reference to a genuinely immutable target (not in the cascade graph); ON UPDATE NO ACTION. (So NO ACTION no longer implies "pruned/covered"; the mode is carried explicitly and is not derivable from the action alone.)
• There is no TriggerFallback and no DocumentId-only FK shape. The retired MssqlIdentityPropagationTrigger is gone for identity-value propagation; native ON UPDATE CASCADE (which follows FKs on child/_ext binding tables) replaces it.
• Fail derivation fast (with a diagnostic) in two fundamental conditions: (1) a cascade cycle / SCC (or self-loop); and (2) diamonds that cannot be jointly broken — either a single uncovered diamond (a receiver reached twice from one origin whose reconverging edge cannot be covered — the case ODS prunes silently and incorrectly), or globally infeasible overlapping diamonds (each breakable in isolation, but no global survivor assignment satisfies the retained-NativeCascade invariant because overlapping diamonds share edges). Diagnostics name the origin/root(s), receiver(s), distinct paths, candidate/pruned edges, and coverage columns — plus, for the overlapping case, the shared edge/mode conflict and the invariant clause that failed. No permissive fallback remains, so the MetaEd authoring guard (METAED-1667) is load-bearing and the DMS fail-fast is the backstop.
• Physical FK pruning, unsafe-graph detection, and fail-fast are SQL Server only — a team-decided engine-specific policy (where the two engines can't behave identically, each does what works for it rather than being forced into parity). PostgreSQL keeps full-composite ON UPDATE CASCADE on every eligible edge and is left as-is: never pruned or failed by DMS (it has no SQL Server 1785 DDL restriction; no claim is made about PostgreSQL runtime behavior for a cascade cycle). Cross-engine safety for authors is handled at authoring time by MetaEd (METAED-1667), which flags a cascade graph SQL Server cannot represent — rather than forcing PostgreSQL derivation to reject graphs it can represent.
• Canonical FK / *_RefKey column order is identity storage columns first, DocumentId last (matches ReferenceConstraintPass and the change-queries /deletes seek-path rationale; local and target lists are positionally aligned). No column-order code change is required.
• Derived contract DMS-1258 must add: per-edge MssqlPropagationMode (three values above), coverage/carrier diagnostics for NoPropagation edges only, and hard fail-fast derivation errors. The MssqlFkShape axis is dropped (every FK is full composite).

Empirical validation (SQL Server 2022)

A throwaway container and a real populated ODS database (EdFi_Ods_Populated_Template, all writes rolled back) confirmed the behaviors the design rests on:

• Msg 1785 — a table reached by multiple cascade paths (or a cascade cycle) rejected at DDL time; pruning one edge into the receiver to NO ACTION makes the graph legal.
• Msg 547 — a NO ACTION composite FK that includes identity columns blocks a parent identity update, and an AFTER trigger cannot rescue it (the FK check precedes the trigger). This is why pruned edges must be covered, and why a DocumentId-only + trigger fallback cannot preserve value-level RI — this design removes that fallback.
• Msg 2113 — no INSTEAD OF UPDATE on a table with a cascading FK (forecloses the "reorder children-first" alternative).
• Kept ON UPDATE CASCADE composite FKs propagate identity changes natively; redundant pruning is safe when a key-unified column is covered by a surviving cascade.
• Real-data note: the stock Ed-Fi cascade cluster is already acyclic, so pruning/fail-fast matter chiefly for key-unified extensions; a single edfi.Session rename cascaded to hundreds of rows in ~1.2 s (identity-update fan-out is the real operational cost).

Changes

• New: mssql-cascading.md — consolidated design note with a "Resolved design decisions" section and a Mermaid Diagrams section (object map, propagation-direction diamond, safe covered pruning, unsafe fail-fast, cycle/SCC fail-fast, before/after RI improvement).
• Revised to point at it and drop the old blanket trigger/DocumentId-first claims: overview, strengths-risks, transactions-and-concurrency, key-unification, data-model, ddl-generation, summary, compiled-mapping-set, update-tracking, flattening-reconstitution, the 01-relational-model/07-index-and-trigger-inventory and 09-identity-concurrency/03-identity-propagation epic docs, and the two presentations. Added "superseded" banners to key-unification-children-problem and key-unification-design-change-proposal (their non-root-referrer concern is solved natively by cascade).

Follow-ups

• DMS-1258 — implement pruning + fail-fast (blocked by this spike). Its ticket wording was corrected off the old "per referenced table" / TriggerFallback framing to the receiver / propagation-direction / three-mode model.
• METAED-1667 — disallow allow primary key updates when no safe pruning strategy exists (now load-bearing, since there is no permissive fallback).
• DMS-1128 — superseded by DMS-1258.

Acceptance criteria

• Analyze ODS implementation and update the design documents with a DMS approach.
• Decide MSSQL-only vs both → MSSQL-only for pruning, detection, and fail-fast (engine-specific; PostgreSQL left as-is; MetaEd is the cross-engine authoring guard — rationale in the note).
• Create a MetaEd ticket to disallow allow primary key updates with no safe pruning → METAED-1667.
• Create a DMS ticket to implement pruning + fail-fast → DMS-1258.
• Update DMS-1128 with the new approach → superseded by DMS-1258.

🤖 Generated with Claude Code

[samuellugo-ship-it]

Rebased onto main (picked up #1086).

[samuellugo-ship-it]

Addressed the review findings in 9c49e3c: rewrote mssql-cascading.md (propagation-direction graph, cycle/SCC + convergence fail-fast, TriggerFallback removed, transitive mutability, portability-guard PG rationale, canonical FK order, MssqlPropagationMode contract + Resolved design decisions) and swept the stale trigger-only/DocumentId-first wording across the design docs. Also corrected the DMS-1258 ticket wording.

[samuellugo-ship-it]

Round-2 review findings addressed in a3358f9: added ImmutableNoAction as a third MssqlPropagationMode (immutable NO ACTION FKs no longer conflated with pruned/covered), swept remaining direct→transitive mutability and local-FK column-order wording, replaced stale "SQL Server propagation trigger" mentions, and added a Mermaid diagrams section (objects, propagation direction, safe/unsafe pruning, cycle/SCC, before/after). PR body and DMS-1258 updated to the three-mode contract.

[samuellugo-ship-it]

Addressed the PASS-WITH-NITS finding: the NativeCascade contract definitions in mssql-cascading.md and compiled-mapping-set.md now explicitly enumerate independent, diamond-survivor, and sole edges (matching the algorithm and outcome table). Docs-only, commit 34db8f6.

[samuellugo-ship-it]

Addressed the re-review findings in 533d21e3 (docs-only; no code changes).

Two things not visible in the diff:

• PostgreSQL fail-fast policy (the one you asked me to decide): adopted fail derivation on BOTH dialects for a SQL-Server-unrepresentable graph, as an explicit cross-engine portability/authoring policy — not a PostgreSQL correctness requirement (PG alone would accept the graph). This resolves the contradiction with the old "future decision to also fail on PostgreSQL" hedge, which is removed. If you'd prefer the alternative (PG records a diagnostic, only SQL Server fails), say so and I'll flip it.
• Jira updated via MCP: METAED-1667 rewritten to per-origin duplicate reachability (dropped "one surviving edge per referenced table"); DMS-1127 retargeted off the removed fallback/nested-trigger model to native-cascade update-tracking validation; DMS-1258 aligned to the both-dialect policy + the new global retained-graph invariant.

Also fixed: column order (identity-first, DocumentId-last), precise "origin" definition, global invariant for overlapping diamonds, pass-ordering/metadata home, softened PG cycle wording, abstract-FK qualifier, expanded NoPropagation/fail-fast diagnostics, historical side-doc subsections marked superseded, and swept the "surviving edge" shorthand to accurate wording repo-wide.

[samuellugo-ship-it]

Addressed the FAIL findings in 81b2e66a (docs-only; no code).

• Overlapping-diamond survivor selection (Major): went with direction A. mssql-cascading.md §5 now specifies a real deterministic global search — backtracking over overlapping diamonds in stable order, accepting the first complete assignment whose retained NativeCascade graph satisfies the invariant (one mode/FK, acyclic, multitree, every NoPropagation edge covered by the final graph), and failing fast only when no global assignment exists. This replaces the "local first-fit + post-hoc fail" that could falsely reject representable schemas.
• Stale PG subsection (Major): key-unification.md "PostgreSQL (no special mitigation required)" rewritten to match mssql-cascading.md — full-composite CASCADE on eligible edges, never pruned, both-dialect fail-fast as a portability policy, no runtime-cycle claim.

Off-diff:

• PR body bullet 6 corrected (dropped "PostgreSQL cascades multiple paths and cycles natively").
• Jira: DMS-1128 supersession banner rewritten to drop TriggerFallback/MssqlFkShape/DocumentId-only/"one surviving edge per referenced table"; DMS-1258 AC + tasks aligned to the deterministic global search (incl. an overlapping-diamond backtrack test case).

Verified the remaining cycles or multiple cascade paths hits are quotes of SQL Server's own 1785 error text, not PG claims; mssql-cascading.md and key-unification.md now agree on the PostgreSQL policy.

[samuellugo-ship-it]

Open decision for the team — PostgreSQL fail-fast policy. The recurring PG-wording findings trace back to one unratified policy call: should DMS fail derivation on both dialects for a SQL-Server-unrepresentable graph (current design, a portability policy), or only on SQL Server with PostgreSQL recording a diagnostic? I've written up the context, both options with trade-offs, and a recommendation (Option A) as a decision record on DMS-1258 for team ratification: https://edfi.atlassian.net/browse/DMS-1258?focusedCommentId=94649 — the docs, DMS-1258 ACs, and METAED-1667 will be aligned to whichever the team picks.

[samuellugo-ship-it]

Latest review findings addressed in af3981cd (docs-only). The recurring open item — the PostgreSQL fail-fast policy — is now resolved by team decision: engine-specific.

• Pruning, unsafe-graph detection, and fail-fast are SQL Server only. PostgreSQL is left as-is (full-composite ON UPDATE CASCADE, never pruned or failed by DMS). Cross-engine safety for authors lives in the MetaEd authoring guard (METAED-1667) — we do not force PostgreSQL to reject graphs it can represent. Replaced the earlier "fail derivation on both dialects / portability policy" (Option A) wording everywhere, and removed the last PG runtime-cycle phrasings.
• Overlapping-diamond fail flavor ("globally infeasible overlapping diamonds") propagated into the implementation-facing fail-fast shorthands (data-model, ddl-generation, overview, summary, transactions, flattening-reconstitution, key-unification, epics 07 + 09) and the MetaEd guard, so no summary hides it.
• Column-order straggler fixed in key-unification-design-change-proposal.md.

Off-diff: METAED-1667 + DMS-1258 aligned to engine-specific + the overlapping flavor; the DMS-1258 decision record (comment 94649) is marked resolved; DMS-1129 AC-bullet-2 decision recorded; this PR body updated.

Verified no regressions: no TriggerFallback / MssqlFkShape / DocumentId-only / per-referenced-table / raw-in-degree as live behavior; no both-dialect overstatement; no PG runtime-cycle claims.

[samuellugo-ship-it]

Rebased onto latest main (docs-only, no conflicts).