ACE-072: cross-datasource bridge as a validated model type by ashwin-agami · Pull Request #156 · AgamiAI/agami-core

ashwin-agami · 2026-07-28T04:47:27Z

Spec: ACE-072

Summary

First slice of F16 — cross-datasource metrics. Promotes the cross-datasource "bridge" (the shared-key link that says a customer in one datasource is the same entity in another, e.g. accounts.account_key = customers.account_key) from a skill-side file into a first-class, validated semantic-model type on the deployment-level OrgRecord. This is the foundation the cross-datasource metric (ACE-073) will reconcile through. Previously the bridge lived only in <artifacts_dir>/local/cross_profile_relationships.yaml — unvalidated, and lost on a machine re-onboard (the confirmed Northpeak v2 regression).

Scope is the foundation only. The idempotent "link your databases" offer + the governed sm bridges CLI are deliberately out of scope → ACE-078; the metric type/execution/browse are ACE-073/074/076.

Changes

models.py — new standalone CrossDatasourceRelationship (datasource+schema.table+column-list endpoints; executable restricted to split/informational, rejecting same_engine; optional cardinality; trust block; endpoint_key for dedup). Adds the cross_datasource_relationships bucket to OrgRecord.
org_record.py — load_org_record now merges bridges from three sources (inline · sidecar cross_datasource_relationships.yaml · legacy cross_profile_relationships.yaml migration), deduped by endpoint tuple, in-memory / read-only, and lenient (a malformed optional file is skipped per-entry, never raised — preserving the runtime-path contract).
validator.py — new deployment-level validate_deployment() pass: loads every attached datasource model and rejects a bridge that is same_engine or whose datasource/dataset/column endpoint doesn't resolve (cross_datasource_executable_mismatch, cross_datasource_endpoint_unresolved), fail-closed.
cli.py — sm validate also runs the deployment pass when an OrgRecord is present (no new subcommand; a no-op on pre-F16 layouts).
Tests — new tests/test_cross_datasource_bridge.py (model, validation reject/accept, idempotent + precedence-correct migration, malformed-file leniency, distinct unreadable-profile message, CLI wiring).

Test plan

uv run dev.py check green (1576 passed / 1 skipped; ruff + gitleaks clean). uv run dev.py cover = 100% patch coverage.
Reviewed by the Agami review panel (correctness + security + structural, high lane): no stop-the-line issues; all findings addressed (leniency contract, endpoint-message accuracy, exact-schema preference, test rigor).

Checklist

Spec-linked (Spec: ACE-072), scope-contained (no ACE-073/074/078 work)
Public-safe: synthetic acme/demo/account_key fixtures only — no customer or internal naming
New behavior covered by tests; CI-equivalent green locally
Fail-closed validation; no SQL-guard surface touched (model + validation only)

Add CrossDatasourceRelationship (deployment-level bridge) to the semantic model, merge it into the OrgRecord from inline/sidecar/legacy sources on load (deduped, idempotent), and validate its endpoints in a new validate_deployment pass wired into sm validate. Legacy cross_profile_relationships.yaml is migrated in-memory.

…et schema match (ACE-072)

Copilot

Pull request overview

This PR implements the ACE-072 foundation for F16 cross-datasource metrics by promoting the “bridge” concept (shared-key link across two datasources) into a first-class, validated deployment-level model on OrgRecord, plus a deployment-wide validation pass and CLI wiring.

Changes:

Adds a new CrossDatasourceRelationship model type and stores a list of them on OrgRecord.
Adds deployment-level validation (validate_deployment) to resolve bridge endpoints across all attached datasources and reject invalid bridges.
Updates sm validate to run the deployment-level pass when invoked within a deployment layout, and adds comprehensive tests for model/merge/migration/validation behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_cross_datasource_bridge.py	New tests covering the bridge model, deployment validation, merge/migration behavior, and CLI integration.
packages/agami-core/src/semantic_model/validator.py	Adds `validate_deployment()` and supporting helpers to validate cross-datasource bridges across all datasource models.
packages/agami-core/src/semantic_model/org_record.py	Loads/merges bridges from inline + sidecar + legacy sources and dedupes them by endpoint identity.
packages/agami-core/src/semantic_model/models.py	Introduces `CrossDatasourceRelationship` and adds `cross_datasource_relationships` to `OrgRecord`.
packages/agami-core/src/semantic_model/cli.py	Extends `sm validate` to also run the deployment-level validation pass when applicable.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    if sidecar.exists():
+        doc = yaml.safe_load(sidecar.read_text(encoding="utf-8")) or {}
+        for r in doc.get("relationships", doc if isinstance(doc, list) else []):
+            try:
+                out.append(CrossDatasourceRelationship(**r))
+            except Exception:
+                # load_org_record is lenient by contract (never raises — it's on runtime paths). A
+                # malformed entry is skipped so its valid siblings in the same file still load.
+                continue


+    if legacy.exists():
+        doc = yaml.safe_load(legacy.read_text(encoding="utf-8")) or {}
+        for r in doc.get("relationships", doc if isinstance(doc, list) else []):
+            try:
+                out.append(_migrate_legacy_bridge(r, str(legacy)))
+            except Exception:
+                # Same leniency: a legacy entry missing a required key is skipped, not raised, so a
+                # single malformed row can't take down a runtime load_org_record call.
+                continue


…ecord (ACE-072) The per-entry parse was already lenient; wrap the sidecar/legacy yaml.safe_load too so a corrupt bridge FILE degrades to no bridges rather than raising out of the runtime-path load_org_record. Adds test_corrupt_bridge_yaml_file_does_not_raise.

_load_bridges claimed to support a mapping OR a bare YAML list, but doc.get(...) raises AttributeError on a bare list before the default is used. Extract entries via an explicit shape check (_bridge_entries); add a bare-list test.

ashwin-agami · 2026-07-28T04:59:25Z

Thanks @copilot — both findings were valid and are fixed in 2ade01a.

_load_bridges now extracts entries via an explicit shape check (_bridge_entries): a relationships: mapping, a bare YAML list, or [] — so a bare-list bridge file no longer calls .get on a list. The read/parse of the optional sidecar + legacy files was also wrapped in an earlier commit (bc87c50), so a corrupt/unreadable file degrades to no bridges rather than raising on the runtime-path load_org_record.

Both are locked by new tests: test_bare_list_sidecar_loads and test_corrupt_bridge_yaml_file_does_not_raise.

Copilot · 2026-07-28T05:01:53Z

Thanks @copilot — both findings were valid and are fixed in 2ade01a.

_load_bridges now extracts entries via an explicit shape check (_bridge_entries): a relationships: mapping, a bare YAML li...

Confirmed — both issues are addressed in the current code. _bridge_entries now handles bare-list YAML shapes without calling .get on a list, and the read_text/safe_load calls for both the sidecar and legacy file are wrapped in try/except so corrupt content degrades to an empty list rather than raising. The two new tests lock both contracts. CI is green on 2ade01a.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

…ts (ACE-072) write/load_organization_record hand-listed 4 fields, silently dropping new OrgRecord buckets on a hosted deploy — the cross_datasource_relationships bridge among them. Dump/validate the whole record (its own stated contract), so the bridge (and future buckets) round-trip. Adds a persistence regression test.

ashwin-agami added 2 commits July 28, 2026 09:58

fix(core): address ACE-072 review — leniency, endpoint message, datas…

288aeff

…et schema match (ACE-072)

ashwin-agami requested a review from sandeep-agami as a code owner July 28, 2026 04:47

Copilot AI review requested due to automatic review settings July 28, 2026 04:47

Copilot started reviewing on behalf of ashwin-agami July 28, 2026 04:47 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

ashwin-agami added 2 commits July 28, 2026 10:23

Copilot started work on behalf of ashwin-agami July 28, 2026 05:00 View session

Copilot finished work on behalf of ashwin-agami July 28, 2026 05:02

ashwin-agami requested a review from Copilot July 28, 2026 09:33

Copilot started reviewing on behalf of ashwin-agami July 28, 2026 09:34 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ACE-072: cross-datasource bridge as a validated model type - #156

ACE-072: cross-datasource bridge as a validated model type#156
ashwin-agami wants to merge 5 commits into
mainfrom
ACE-072-bridge-as-model-type

ashwin-agami commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ashwin-agami commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ashwin-agami commented Jul 28, 2026

Summary

Changes

Test plan

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

ashwin-agami commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants