feat: metagraph phase 1 — per-fiber stateRoot + metagraphStateRoot#117
feat: metagraph phase 1 — per-fiber stateRoot + metagraphStateRoot#117ottobot-ai wants to merge 11 commits into
Conversation
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
690c08f to
6ecfa8f
Compare
abc0b01 to
67b1c2d
Compare
- Add 15 failing tests in 5 groups for metagraph integration spec - Group 1: StateMachineFiberRecord stateRoot field (3 tests) - Group 2: CalculatedState metagraphStateRoot field (3 tests) - Group 3: MerklePatriciaProducer integration (3 tests) - Group 4: hashCalculatedState override (3 tests) - Group 5: State proof API endpoint (3 tests) All tests fail with NotImplementedError as expected in TDD workflow. Tests will pass once Phase 1 implementation is complete: - Add stateRoot to StateMachineFiberRecord - Add metagraphStateRoot to CalculatedState - Compute via MerklePatriciaProducer.inMemory in FiberCombiner - Override hashCalculatedState with MPT root - Add GET /state-proof/:fiberId endpoint
- add stateRoot field to FiberData (MPT per-fiber state commitment) - add metagraphStateRoot to CalculatedState (MPT over all fiber roots) - MerklePatriciaProducer computes roots via RFC 8785 canonicalization - CheckpointService updates roots on each snapshot - tests: TDD suite for MPT root computation and CalculatedState integration
* feat: phase 1B — GET /state-machines/:fiberId/state-proof endpoint Add two-level MPT inclusion proof endpoint to ML0CustomRoutes. - GET /v1/state-machines/:fiberId/state-proof lists available fields + fiberStateRoot + metagraphStateRoot - GET /v1/state-machines/:fiberId/state-proof?field=<name> returns fiberProof (field ∈ stateRoot) + metagraphProof (fiberId.stateRoot ∈ metagraphStateRoot) Proof generation replicates FiberCombiner/ML0Service MPT encoding: - field key = UTF-8 bytes of field name as lowercase hex - metagraph key = UUID without dashes as hex Tests: 12 weaver tests covering encoding, proof generation, verification against roots, and two-level proof chain consistency. * chore: update CODEOWNERS to ottobot-ai for fork iteration
67b1c2d to
fea42e4
Compare
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
|
🔍 Self-Review Complete Metagraph Phase 1 — per-fiber stateRoot + metagraphStateRoot:
Ready for @scasplte2 review and merge. |
🔧 CI Fix — a1979b8Root cause: Tessellation uses Fix: Always return Re-triggering E2E. |
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
hashCalculatedState must return state.computeDigest for Tessellation snapshot validation to work correctly. Returning metagraphStateRoot (a trie hash of fiber roots) caused a hash mismatch: the framework validates the hash against the serialized CalculatedState, so returning a different hash broke consensus and caused all E2E transaction confirmations to time out at ordinal 4. metagraphStateRoot is an auxiliary proof field for the /state-proof API only. It must not replace the canonical state hash.
a1979b8 to
dd9b275
Compare
|
🔧 Fixed CI failure: commitlint Issue: Commit header was 76 chars (max 72): Fixed to (65 chars): Body copy retained verbatim. Force-pushed. |
metagraphStateRoot is an MPT proof field for the /state-proof API. It must NOT be included in hashCalculatedState because: 1. combineData sets metagraphStateRoot after transactions are applied 2. The acceptance manager recomputes the hash from a state where the field may be None (e.g. deserialized from a snapshot that pre-dates this field, or replayed via the global snapshot sync path) 3. Hashing with the field present vs absent produces different hashes causing CalculatedStateHashDoesNotMatchMajority errors at ordinal 5+ Fix: always strip metagraphStateRoot before calling computeDigest so the canonical state hash is stable across all validation paths. This supersedes dd9b275 which restored computeDigest but still included metagraphStateRoot in the serialized state.
The tessellation develop branch script tries to build snapshot-streaming from source during 'just up', which requires a GitHub token to authenticate with GitHub Packages sbt resolver. Without it, sbt fails with: unable to locate a valid GitHub token from Or(GitConfig(github.token),Environment(GITHUB_TOKEN)) Add GITHUB_TOKEN to the Start cluster step env so the tessellation docker scripts can authenticate with GitHub Packages when building dependencies.
snapshot-streaming in tessellation's develop branch fails to compile against the 4.0.0-rc.10 release JARs due to API changes. Pin the git clone to the exact tag resolved from metakit so the build infrastructure matches the JARs.
Pinning to the release tag broke E2E tests — 0/12 flows, all timing out at ML0 ordinal confirmation (300s). The compose-runner at the release tag does not support --skip-assembly with --dl1 --data flags. The develop branch has the latest compose-runner with pre-staged JAR support. Revert to develop; SDK deps resolve from Maven Central anyway.
3ba5483 to
3c0d6b5
Compare
Tessellation develop branch drift broke Configuration.scala in snapshot-streaming (constructor parameter order changed). Pin to the version tag resolved from the metakit dependency chain to ensure stable infrastructure files.
d6cabda to
3c1903c
Compare
|
🔧 CI fix: shortened commit header Commitlint rejected the header at 74 chars (limit: 72). Amended to: |
…event hash mismatch The DataApplicationSnapshotAcceptanceManager throws CalculatedStateHashDoesNotMatchMajority when the proposal-phase hash differs from the acceptance-phase recomputed hash. Root cause: FiberCombiner.computeStateRoot and ML0Service.computeMetagraphStateRoot set stateRoot on StateMachineFiberRecord and metagraphStateRoot on CalculatedState after transactions are applied. hashCalculatedState stripped metagraphStateRoot but NOT stateRoot on individual fibers, causing the hash to include the per-fiber stateRoot values. If any deserialization path (proto round-trip, snapshot sync) drops these fields, the hash computed from the deserialized state differs. Fix: - hashCalculatedState now strips BOTH metagraphStateRoot AND stateRoot on all fiber records before computing the digest. These fields are proof artifacts derived from stateData — not primary consensus state. - Add stateRoot (field 15) to StateMachineFiberRecord proto and metagraph_state_root (field 3) to CalculatedState proto so these fields survive any proto-based serialization paths (indexer, explorer, SDK). The stateRoot and metagraphStateRoot values remain accessible via the /state-proof API and are stored correctly in the checkpoint — they are simply excluded from the consensus hash computation.
|
🔧 CI Fix — CalculatedStateHashDoesNotMatchMajority Root cause: The consensus hash included Diagnosis from ML0 logs (artifact Ordinals 1–4 advanced fine (TimeTrigger, no data). Ordinal 5 was the first EventTrigger with data — where Fix ( Also added: |
|
🔄 Rebased onto main — conflict resolved. |
|
PR size: 825 lines (too large for automated conflict resolution) @scasplte2 — please rebase manually when ready. |
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
diff --git a/docs/design/authenticated-trie-integration-spec.md b/docs/design/authenticated-trie-integration-spec.md
new file mode 100644
index 0000000..0b16c0c
--- /dev/null
+++ b/docs/design/authenticated-trie-integration-spec.md
@@ -0,0 +1,465 @@
+# Authenticated Trie Integration for OttoChain State
+
+**Card:** Design: Authenticated trie integration for OttoChain state (699fa07f)
+**Status:** Specification
+**Author:** @Think
+**Date:** 2026-02-26
+**Depends on:** PR scasplte2#117 (Phase 1 — per-fiber stateRoot + metagraphStateRoot)
+**Related:** Phase 1B card (69a04ae3), SDK authenticated trie card (69963015), PR scasplte2#61 (feat/authenticated-tries)
+
+---
+
+## 1. Problem Statement
+
+OttoChain fiber state is managed by ML0's `CalculatedState`. Clients (bridges, SDK, external verifiers) currently have no way to prove that a fiber's `stateData` matches what the metagraph committed to in a snapshot — they must trust the bridge entirely.
+
+**Goal:** Enable trustless verification of fiber state by:
+1. Exposing Merkle Patricia Trie (MPT) inclusion proofs for individual fiber state fields
+2. Anchoring those per-fiber proofs to the `metagraphStateRoot` committed in `hashCalculatedState`
+3. Giving TypeScript clients a verifier that needs only the snapshot hash + proof (no full state download)
+
+### What Phase 1 (PR scasplte2#117) Already Added
+
+PR scasplte2#117 established the data layer:
+- `StateMachineFiberRecord.stateRoot: Option[Hash]` — per-fiber MPT root of `stateData` fields
+- `CalculatedState.metagraphStateRoot: Option[Hash]` — MPT root over `{fiberIdHex → stateRoot}` map
+- `hashCalculatedState` returns `metagraphStateRoot` when present (making it the snapshot commitment)
+
+This spec covers **Phase 1B**: the proof-generation API that makes those roots useful to clients.
+
+---
+
+## 2. Architecture Overview
+
+```
+Client wants to verify fiber X, field "balance"
+ │
+ ▼
+GET /v1/state-machines/{fiberId}/state-proof?field=balance
+ │
+ ▼ [ML0CustomRoutes.scala]
+ │ 1. Get stateData from CheckpointService
+ │ 2. Build 5-leaf trie: stateData field → JSON value
+ │ 3. attestPath(hexEncode("balance")) → field-level proof
+ │ 4. Build metagraph trie: fiberIdHex → stateRoot
+ │ 5. attestPath(fiberIdHex) → metagraph-level proof
+ │ 6. Return {fieldProof, metagraphProof, stateRoot, metagraphStateRoot}
+ │
+ ▼
+Client reconstructs:
+ - Hashes stateData[field] → compares against Leaf.dataDigest in fieldProof
+ - Verifies fieldProof against stateRoot
+ - Verifies metagraphProof against metagraphStateRoot
+ - Compares metagraphStateRoot against known snapshot hash (from global snapshot)
+```
+
+### Why ML0 (Not Bridge)
+
+Proof generation requires `CheckpointService` access (live `CalculatedState` with all `stateData` fields). Only ML0 has this. The bridge proxies the endpoint transparently — no business logic in the bridge.
+
+### Why Stateless (Not Persistent LevelDB)
+
+Phase 1 uses `StatelessMerklePatriciaProducer` (recomputes the trie from `stateData` on each request). For the 5-field trie typical of OttoChain fibers, this takes **<5ms** — no meaningful latency. LevelDB adds operational complexity (disk management, per-fiber files) with zero benefit at current scale. If Phase 3 requires historical proofs, the Indexer's stored `stateData` history enables reconstruction without LevelDB on ML0.
+
+---
+
+## 3. Proof Data Model
+
+### 3.1 Request
+
+```
+GET /v1/state-machines/{fiberId}/state-proof?field={fieldName}
+```
+
+**Path param:** `fiberId` — UUID of the fiber (e.g. `550e8400-e29b-41d4-a716-446655440000`)
+**Query param:** `field` — dot-path into `stateData` JSON (e.g. `balance`, `owner`, `status`)
+**Auth:** Public (no authentication required — proofs contain no sensitive data)
+
+**Omit `?field`:** Returns only the metagraph-level proof (proves the fiber's stateRoot is committed, without exposing stateData contents). Useful for existence proofs.
+
+### 3.2 Success Response (200 OK)
+
+```json
+{
+ "fiberId": "550e8400-e29b-41d4-a716-446655440000",
+ "field": "balance",
+ "value": {"var": 1000},
+ "stateRoot": "abc123...",
+ "metagraphStateRoot": "def456...",
+ "fieldProof": {
+ "path": "62616c616e6365",
+ "witness": [
+ { "type": "Leaf", "contents": { "remaining": [6, 2, ...], "dataDigest": "sha256hash..." } }
+ ]
+ },
+ "metagraphProof": {
+ "path": "550e8400e29b41d4a716446655440000",
+ "witness": [
+ { "type": "Branch", "contents": { "pathsDigest": {...} } },
+ { "type": "Leaf", "contents": { "remaining": [...], "dataDigest": "sha256hash..." } }
+ ]
+ }
+}
+```
+
+**Field descriptions:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `fiberId` | string | UUID of the fiber |
+| `field` | string | The requested field name |
+| `value` | JSON | The current value of `stateData[field]` |
+| `stateRoot` | Hex | SHA-256 MPT root of this fiber's stateData |
+| `metagraphStateRoot` | Hex | SHA-256 MPT root of all fiber stateRoots |
+| `fieldProof` | object | MPT inclusion proof: `stateData[field]` in fiber trie |
+| `metagraphProof` | object | MPT inclusion proof: `stateRoot` in metagraph trie |
+
+### 3.3 MPT Key Encoding
+
+**Field-level trie keys:**
+`hex(utf8Bytes(fieldName))` — e.g., `"balance"` → `62616c616e6365`
+
+**Metagraph-level trie keys:**
+`fiberIdUUID.toString.replace("-", "")` — e.g., UUID `550e8400-e29b-41d4-a716-446655440000` → `550e8400e29b41d4a716446655440000`
+
+(This matches the encoding already used in `ML0Service.computeMetagraphStateRoot`)
+
+### 3.4 Error Responses
+
+| HTTP | Error | Condition |
+|------|-------|-----------|
+| 404 | `fiber_not_found` | UUID exists but no fiber in CalculatedState |
+| 404 | `field_not_found` | fiber exists but `stateData[field]` is absent |
+| 404 | `no_state_root` | fiber exists but `stateRoot` is None (pre-PR scasplte2#117) |
+| 400 | `invalid_uuid` | fiberId is not a valid UUID |
+| 500 | `proof_error` | MPT proof generation failed (should not happen) |
+
+```json
+{
+ "error": "fiber_not_found",
+ "message": "No fiber found with id: 550e8400-e29b-41d4-a716-446655440000"
+}
+```
+
+---
+
+## 4. ML0 Implementation Spec
+
+### 4.1 New Route in `ML0CustomRoutes.scala`
+
+```scala
+case req @ GET -> Root / "state-machines" / UUIDVar(fiberId) / "state-proof" :?
+ OptionalQueryParamDecoderMatcher[String]("field")(fieldOpt) =>
+ generateStateProof(fiberId, fieldOpt)
+```
+
+### 4.2 `generateStateProof` Logic
+
+```scala
+private def generateStateProof(fiberId: UUID, fieldOpt: Option[String]): F[Response[F]] = {
+ checkpointService.get.flatMap { case Checkpoint(_, state) =>
+ state.stateMachines.get(fiberId) match {
+ case None =>
+ NotFound(Json.obj("error" -> "fiber_not_found".asJson, ...))
+
+ case Some(fiber) if fiber.stateRoot.isEmpty =>
+ NotFound(Json.obj("error" -> "no_state_root".asJson, ...))
+
+ case Some(fiber) =>
+ val stateData: Map[String, Json] = fiber.stateData // from StateMachineFiberRecord
+ val stateRoot: Hash = fiber.stateRoot.get
+
+ // Build field-level trie (same key encoding as FiberCombiner)
+ val fieldKeys: Map[Hex, Json] = stateData.map { case (k, v) =>
+ Hex(k.getBytes("UTF-8").map("%02x".format(_)).mkString) -> v
+ }
+
+ for {
+ // Field-level proof
+ fieldProofOpt <- fieldOpt.traverse { field =>
+ stateData.get(field) match {
+ case None => NotFound(Json.obj("error" -> "field_not_found".asJson, ...)).pure[F]
+ case Some(_) =>
+ val fieldHex = Hex(field.getBytes("UTF-8").map("%02x".format(_)).mkString)
+ MerklePatriciaProducer.stateless[F]
+ .create(fieldKeys)
+ .flatMap(trie => MerklePatriciaProver.make(trie).attestPath(fieldHex))
+ }
+ }
+
+ // Metagraph-level proof (fiberId → stateRoot)
+ metagraphKeys: Map[Hex, Hash] = state.stateMachines.collect {
+ case (id, f) if f.stateRoot.isDefined =>
+ Hex(id.toString.replace("-", "")) -> f.stateRoot.get
+ }
+ metagraphProof <- MerklePatriciaProducer.stateless[F]
+ .create(metagraphKeys)
+ .flatMap(trie =>
+ MerklePatriciaProver.make(trie)
+ .attestPath(Hex(fiberId.toString.replace("-", "")))
+ )
+
+ response = buildProofResponse(fiberId, fieldOpt, fiber, stateRoot,
+ state.metagraphStateRoot.get, fieldProofOpt, metagraphProof)
+ result <- Ok(response)
+ } yield result
+ }
+ }
+}
+```
+
+### 4.3 Bridge Proxy
+
+The bridge routes `GET /fiber/:fiberId/state-proof?field=X` → ML0 `GET /v1/state-machines/:fiberId/state-proof?field=X` transparently. No validation logic in the bridge — it just forwards and returns the response.
+
+New bridge route in `routes/stateProof.ts`:
+```typescript
+router.get('/fiber/:fiberId/state-proof', async (req, res) => {
+ const { fiberId } = req.params;
+ const { field } = req.query;
+ const url = `${ML0_URL}/v1/state-machines/${fiberId}/state-proof${field ? `?field=${field}` : ''}`;
+ const response = await fetch(url);
+ const data = await response.json();
+ res.status(response.status).json(data);
+});
+```
+
+---
+
+## 5. Canonicalization Note for Verifiers
+
+**Critical for TypeScript implementors:**
+
+The MPT leaf data digest is computed via `JsonBinaryHasher.computeDigest` → `JsonBinaryCodec.serialize` → `JsonCanonicalizer.canonicalizeJson` which uses **RFC 8785** with **UTF-16BE key sort order**.
+
+This is **not** the same as `JSON.stringify` with `Object.keys().sort()` for non-ASCII keys.
+
+For all-ASCII field names (which OttoChain stateData uses), simple lexicographic key sort produces the same result. But TypeScript verifiers MUST NOT assume this — they should use a proper RFC 8785 implementation (e.g., the `canonicalize` npm package or `JSON.stringify` with `replacer` + `Array.sort()` with UTF-16 code unit comparison).
+
+**SHA-256 prefix bytes** (from `MerklePatriciaNode`):
+- Leaf nodes: prefix `0x00`
+- Branch nodes: prefix `0x01`
+- Extension nodes: prefix `0x02`
+
+These must be prepended before hashing when reimplementing the verifier.
+
+---
+
+## 6. TypeScript Client Verification
+
+### 6.1 `verifyStateProof()` — ~30 lines
+
+```typescript
+import { createHash } from 'crypto'; // or SubtleCrypto in browsers
+
+type Proof = {
+ path: string; // hex-encoded path
+ witness: Commitment[]; // ordered leaf-to-root
+};
+
+type Commitment =
+ | { type: 'Leaf'; contents: { remaining: number[]; dataDigest: string } }
+ | { type: 'Branch'; contents: { pathsDigest: Record<string, string> } }
+ | { type: 'Extension'; contents: { shared: number[]; childDigest: string } };
+
+const PREFIX = { Leaf: Buffer.from([0x00]), Branch: Buffer.from([0x01]), Extension: Buffer.from([0x02]) };
+
+function hexToNibbles(hex: string): number[] {
+ return hex.split('').map(c => parseInt(c, 16));
+}
+
+async function computeNodeDigest(commitment: Commitment): Promise<string> {
+ const json = JSON.stringify(commitment.contents); // all-ASCII keys: safe to use stringify
+ const data = Buffer.concat([PREFIX[commitment.type], Buffer.from(json)]);
+ return createHash('sha256').update(data).digest('hex');
+}
+
+export async function verifyStateProof(
+ proof: Proof,
+ expectedRoot: string
+): Promise<boolean> {
+ const nibbles = hexToNibbles(proof.path);
+ let currentDigest = expectedRoot;
+ let remainingPath = nibbles;
+
+ for (const commitment of [...proof.witness].reverse()) {
+ const computedDigest = await computeNodeDigest(commitment);
+ if (computedDigest !== currentDigest) return false;
+
+ if (commitment.type === 'Leaf') {
+ return commitment.contents.remaining.join('') === remainingPath.join('');
+ } else if (commitment.type === 'Extension') {
+ remainingPath = remainingPath.slice(commitment.contents.shared.length);
+ currentDigest = commitment.contents.childDigest;
+ } else if (commitment.type === 'Branch') {
+ const nibble = remainingPath[0].toString(16);
+ currentDigest = commitment.contents.pathsDigest[nibble];
+ remainingPath = remainingPath.slice(1);
+ }
+ }
+ return false;
+}
+```
+
+### 6.2 End-to-End Verification Pattern
+
+```typescript
+// Get proof from bridge
+const proof = await fetch(`/fiber/${fiberId}/state-proof?field=balance`).then(r => r.json());
+
+// Step 1: Verify field value matches leaf
+const valueJson = JSON.stringify(proof.value);
+const valueDigest = sha256(valueJson);
+// (Leaf.dataDigest in fieldProof should match valueDigest)
+
+// Step 2: Verify field inclusion in per-fiber trie
+const fieldValid = await verifyStateProof(proof.fieldProof, proof.stateRoot);
+
+// Step 3: Verify per-fiber stateRoot in metagraph trie
+const metagraphValid = await verifyStateProof(proof.metagraphProof, proof.metagraphStateRoot);
+
+// Step 4: Verify metagraphStateRoot against known snapshot
+// (From global snapshot's calculatedStateHash — fetched from L0 API)
+const knownHash = await getSnapshotCalculatedStateHash(latestSnapshotOrdinal);
+const rootValid = proof.metagraphStateRoot === knownHash;
+
+return fieldValid && metagraphValid && rootValid;
+```
+
+---
+
+## 7. Relationship to Existing Work
+
+### PR scasplte2#61 (feat/authenticated-tries)
+
+PR scasplte2#61 is an early-stage prototype exploring authenticated trie infrastructure. Phase 1B (this spec) is the production implementation path using metakit's `StatelessMerklePatriciaProducer` which is already in the codebase.
+
+**Decision:** Do not merge or depend on PR scasplte2#61. Proceed with Phase 1B implementation directly on `develop`. PR scasplte2#61 can be closed once Phase 1B merges.
+
+### SDK Card (69963015 — Feasibility)
+
+The SDK card covers TypeScript-side authenticated trie management for **multiple dimensions** (stateData, owners, children, etc.). That is Phase 2. Phase 1B covers only the bridge-proxied proof endpoint for `stateData` fields.
+
+### Phase Structure
+
+| Phase | Scope | Blocked on |
+|-------|-------|------------|
+| Phase 1 | Per-fiber stateRoot + metagraphStateRoot computation | ✅ PR scasplte2#117 |
+| **Phase 1B** | `GET /state-proof` endpoint (this spec) | PR scasplte2#117 merge |
+| Phase 2 | Historical proofs (Indexer stateData history) | Phase 1B |
+| Phase 3 | Multi-dimensional tries (owners, children, scripts) | Phase 2 + SDK card |
+
+---
+
+## 8. Open Questions (Spec-Level — Not Blockers)
+
+These are design decisions to resolve during implementation or James review:
+
+1. **Batch proofs**: Support `?fields=balance,owner` to prove multiple fields in one request? Adds complexity; defer to Phase 2 unless needed immediately.
+
+2. **Auth on proof endpoint**: Public access is fine (proofs contain no secrets). But should we rate-limit? Bridge could add rate limiting if needed.
+
+3. **PR scasplte2#61 disposition**: Confirm PR scasplte2#61 should be closed once Phase 1B merges (or keep open for Phase 3 exploration?).
+
+---
+
+## 9. TDD Test Cases
+
+**22 tests in 5 groups.** Tests are for the Phase 1B implementation — blocked on PR scasplte2#117 merge, which adds `stateRoot` and `metagraphStateRoot` to the data model.
+
+### Group 1: Route Registration (3 tests)
+
+```
+T1.1 — GET /v1/state-machines/{uuid}/state-proof returns 200 for valid fiber with stateRoot
+T1.2 — GET /v1/state-machines/not-a-uuid/state-proof returns 400 (invalid UUID)
+T1.3 — GET /v1/state-machines/{uuid}/state-proof without ?field omits fieldProof (returns metagraph proof only)
+```
+
+### Group 2: Error Cases (4 tests)
+
+```
+T2.1 — Unknown fiberId → 404 with error: "fiber_not_found"
+T2.2 — Known fiberId, unknown field → 404 with error: "field_not_found"
+T2.3 — Known fiberId with stateRoot: None → 404 with error: "no_state_root"
+T2.4 — Empty CalculatedState (no fibers) → 404 for any fiberId
+```
+
+### Group 3: Proof Format (5 tests)
+
+```
+T3.1 — Response contains fiberId, field, value, stateRoot, metagraphStateRoot, fieldProof, metagraphProof
+T3.2 — fieldProof.path = hex(utf8("fieldName"))
+T3.3 — metagraphProof.path = fiberId.toString.replace("-", "")
+T3.4 — fieldProof.witness is non-empty List[MerklePatriciaCommitment] (Leaf/Branch/Extension)
+T3.5 — metagraphProof.witness is non-empty List[MerklePatriciaCommitment]
+```
+
+### Group 4: Proof Correctness — Scala Round-Trip (5 tests)
+
+```
+T4.1 — fieldProof verifies against stateRoot using MerklePatriciaVerifier.make(stateRoot).confirm(fieldProof)
+T4.2 — metagraphProof verifies against metagraphStateRoot using MerklePatriciaVerifier.make(metagraphStateRoot).confirm(metagraphProof)
+T4.3 — stateRoot returned in response matches fiber.stateRoot from CalculatedState (same value PR scasplte2#117 computed)
+T4.4 — metagraphStateRoot returned matches state.metagraphStateRoot (same value hashCalculatedState returns)
+T4.5 — Tampered stateData field → MerklePatriciaVerifier.confirm returns Left (proof invalid)
+```
+
+### Group 5: TypeScript Verifier (5 tests)
+
+```
+T5.1 — verifyStateProof(fieldProof, stateRoot) returns true for proof from Scala endpoint
+T5.2 — verifyStateProof(metagraphProof, metagraphStateRoot) returns true for proof from Scala endpoint
+T5.3 — verifyStateProof with wrong expectedRoot returns false
+T5.4 — verifyStateProof with tampered witness (changed dataDigest) returns false
+T5.5 — Full chain: field proof → stateRoot → metagraphProof → metagraphStateRoot matches hashCalculatedState output
+```
+
+**Test file locations:**
+- Scala: `modules/l0/src/test/scala/xyz/kd5ujc/metagraph_l0/StateProofRouteSuite.scala`
+- TypeScript: `ottochain-sdk/src/__tests__/state-proof-verifier.test.ts`
+
+**Pre-conditions:**
+- PR scasplte2#117 merged (stateRoot + metagraphStateRoot fields available)
+- `MerklePatriciaProducer.stateless` available in l0 module (already is)
+- `MerklePatriciaProver.make` available (already is)
+- `MerklePatriciaVerifier.make` available (already is)
+
+---
+
+## 10. Acceptance Criteria
+
+- [ ] **AC-1:** `GET /v1/state-machines/{fiberId}/state-proof?field={field}` returns 200 with proof JSON
+- [ ] **AC-2a:** `fieldProof` validates against `stateRoot` using `MerklePatriciaVerifier`
+- [ ] **AC-2b:** `metagraphProof` validates against `metagraphStateRoot` using `MerklePatriciaVerifier`
+- [ ] **AC-3:** `metagraphStateRoot` in response equals `hashCalculatedState` output (snapshot commitment)
+- [ ] **AC-4:** Request with unknown `fiberId` returns 404
+- [ ] **AC-5:** Request with unknown `field` returns 404
+- [ ] **AC-6:** Request without `?field` returns 200 with metagraph proof only (no `fieldProof`)
+- [ ] **AC-7:** Bridge proxy `GET /fiber/:fiberId/state-proof` forwards to ML0 transparently
+- [ ] **AC-8:** TypeScript `verifyStateProof()` passes T5.1–T5.5 cross-language tests
+- [ ] **AC-9:** All 22 TDD tests pass before implementation is considered complete
+- [ ] **AC-10:** Canonicalization note documented in SDK (TypeScript verifier README)
+
+---
+
+## 11. Implementation Checklist
+
+```
+Pre-conditions:
+ ☐ PR scasplte2#117 merged to develop
+
+Phase 1B implementation (~3h @work):
+ ☐ Add OptionalQueryParamDecoderMatcher[String]("field") to ML0CustomRoutes
+ ☐ Add generateStateProof() private method
+ ☐ Add new GET route: /state-machines/{uuid}/state-proof
+ ☐ Add proof JSON response encoder
+ ☐ Add bridge proxy route: GET /fiber/:fiberId/state-proof → ML0
+
+Tests (@code):
+ ☐ StateProofRouteSuite.scala — 17 Scala tests (Groups 1-4)
+ ☐ state-proof-verifier.test.ts — 5 TypeScript tests (Group 5)
+
+Documentation:
+ ☐ SDK README: verifyStateProof() usage + canonicalization warning
+ ☐ Update ML0CustomRoutes route table in docs/
+```
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
…oint
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR #117 merge
Depends on PR #117 (feat/metagraph-phase1-state-roots)
Spec for Design card 699fa07f. Covers:
- ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint
- Two-level MPT proof chain (field → stateRoot → metagraphStateRoot)
- TypeScript verifyStateProof() client implementation (~30 lines)
- RFC 8785 canonicalization notes for cross-language verifiers
- 22 TDD tests in 5 groups (17 Scala + 5 TypeScript)
- 10 acceptance criteria
- Phase 1B blocked on PR scasplte2#117 merge
Depends on PR scasplte2#117 (feat/metagraph-phase1-state-roots)
…119) * docs: authenticated trie integration spec — Phase 1B state-proof API Spec for Design card 699fa07f. Covers: - ML0 GET /v1/state-machines/{fiberId}/state-proof?field=X endpoint - Two-level MPT proof chain (field → stateRoot → metagraphStateRoot) - TypeScript verifyStateProof() client implementation (~30 lines) - RFC 8785 canonicalization notes for cross-language verifiers - 22 TDD tests in 5 groups (17 Scala + 5 TypeScript) - 10 acceptance criteria - Phase 1B blocked on PR #117 merge Depends on PR #117 (feat/metagraph-phase1-state-roots) * fix(docker): currency L0 run-rollback no longer requires hash arg As of tessellation 4.0.0-rc.10: - DAG L0 (GL0): REQUIRES rollback hash as positional argument - Currency L0 (ML0/CL0): Does NOT require hash (auto-detects from storage) - Data L1: Does NOT require hash The old entrypoint assumed all layers needed the hash for run-rollback, causing ML0/CL0 restarts to fail with 'Unexpected argument' error.
Re-homes #117's field-level state proof onto the real committed root. GET /v1/state-machines/:id/state-proof[?field=] and GET /v1/scripts/:id/state-proof[?field=] return a Merkle-Patricia inclusion proof for the fiber/script record (committed at fiber/<id> / script/<id>) against the MPT root whose combined hash IS the snapshot's calculatedStateProof. A client verifies the proof against the consensus-signed root, then reads any field off the proven record — #117's two-level field->fiberRoot->metagraphRoot collapses to one level, anchored to a real consensus root instead of an off-chain one. ?field= also surfaces the named stateData field of the proven record. StateProofHandler delegates to CommittedReader.committed.proveKey (no hand-rolled tries); wired via makeL0 extraRoutes (now passed the reader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-homes #117's field-level state proof onto the real committed root. GET /v1/state-machines/:id/state-proof[?field=] and GET /v1/scripts/:id/state-proof[?field=] return a Merkle-Patricia inclusion proof for the fiber/script record (committed at fiber/<id> / script/<id>) against the MPT root whose combined hash IS the snapshot's calculatedStateProof. A client verifies the proof against the consensus-signed root, then reads any field off the proven record — #117's two-level field->fiberRoot->metagraphRoot collapses to one level, anchored to a real consensus root instead of an off-chain one. ?field= also surfaces the named stateData field of the proven record. StateProofHandler delegates to CommittedReader.committed.proveKey (no hand-rolled tries); wired via makeL0 extraRoutes (now passed the reader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ot into the currency snapshot (#164) * docs: committed-state migration plan (unify #117/#158) Plan of record for adopting metakit committed-state (Approach B) as hashCalculatedState, salvaging #117's field-level proof endpoint on top. Records the key reframe: tessellation already roots calculatedStateProof into the signed currency snapshot + global proof, so a structured root needs no framework change. The prior roadblock was the journal=Option stall (seeded cell -> BreadcrumbUnresolvable), not the rooting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(committed): add CommittedView[CalculatedState] projection (Phase 1) First slice of the committed-state migration: project the FULL calculated state (all 4 fields) into metakit's committed dictionary so the two-tier committed root commits to it, ready to become the currency snapshot's calculatedStateProof. Keys are lowercase slash-namespaced CommitKeys: fiber/<uuid>, script/<uuid>, registry/<name>, reverse/<uuid>. Key derivation is TOTAL (entries has no error channel; a non-total key would halt combine): UUIDs always fit a 64-char segment, and an over-long registry name (render up to 253) falls back to registry/h/<sha256>. CommittedViewSuite covers the registry readable/hashed-overflow keys, namespacing, totality, determinism, and empty genesis. Refs docs/proposals/committed-state-migration.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(committed): wire makeL0 into ML0Service (Phase 1) ML0Service now assembles the data application via CommittedApp.makeL0, so the snapshot's calculatedStateProof becomes the two-tier committed root (MPT state-dict + SMT epoch catalog) — a verifiable calculated-state root in the signed currency snapshot, no tessellation change required. - ML0Service: makeL0 with orderedCombiner (total-order sort + latestLogs reset before folding) and rejectionNotifyingValidator (per-update rejection webhooks); the new onConsensusResult hook refreshes the notification-side checkpoint cache + dispatches the snapshot webhook (replaces the hand-rolled onSnapshotConsensusResult); extraRoutes keeps the existing ML0Routes handlers. - Main: acquire a LevelDB CatalogJournal Resource (required — without it a seeded/restarted node stalls). - DL1: makeL0 commits CommittedOnChain[OnChain]; the validator's on-chain cache and the /onchain route decode the wrapper and read .inner, so the fiber sequence checks and the client-facing shape are unchanged. - Mock fixture serializes the same CommittedOnChain shape. Depends on metakit Constellation-Labs/metakit#48 (required journal + onConsensusResult + extraRoutes context). sharedData 358/358 green; full build compiles. e2e validation pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(committed): total signedOrdering + bump metakit rc.3 Adopt #158's total signing order: OttochainMessage.signedOrdering now tiebreaks on the proof signatures, completing the partial message order to a TOTAL one (pure — no Hasher). The combiner drops the per-combiner content-digest tiebreak and just sorts by signedOrdering, so every node folds the identical sequence (greenfield — no coordination needed). Bump the metakit pin to 1.8.0-rc.3 (committed-state changes: required journal + onConsensusResult/extraRoutes hooks + genesis-combine fix). Until rc.3 lands on Central the metakit-from-source action builds the v1.8.0-rc.3 tag; the tessellation rc.10 jar fallback is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(committed): state-proof endpoint on the committed MPT (Phase 2) Re-homes #117's field-level state proof onto the real committed root. GET /v1/state-machines/:id/state-proof[?field=] and GET /v1/scripts/:id/state-proof[?field=] return a Merkle-Patricia inclusion proof for the fiber/script record (committed at fiber/<id> / script/<id>) against the MPT root whose combined hash IS the snapshot's calculatedStateProof. A client verifies the proof against the consensus-signed root, then reads any field off the proven record — #117's two-level field->fiberRoot->metagraphRoot collapses to one level, anchored to a real consensus root instead of an off-chain one. ?field= also surfaces the named stateData field of the proven record. StateProofHandler delegates to CommittedReader.committed.proveKey (no hand-rolled tries); wired via makeL0 extraRoutes (now passed the reader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump metakit to rc.4; drop source-build patch metakit 1.8.0-rc.4 is published to Maven Central with all the committed-state changes (required journal, onConsensusResult / extraRoutes hooks, and the genesis-combine fix) — verified in the published jar. So the temporary metakit-from-source action is no longer needed: removed it and its callers (ci.yml x2, e2e.yml), and e2e.yml now resolves the tessellation-sdk version straight from the Central metakit POM again. The tessellation rc.10 hypergraph-jar fallback is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Superseded by #164 (merged). The field-level state-proof endpoint from this PR was salvaged into #164 (Phase 2), re-homed onto the committed MPT (reader.committed.proveKey → inclusion proof against the snapshot's calculatedStateProof). The hand-rolled per-fiber stateRoot + metagraphStateRoot mechanism was replaced by rooting the committed calculated-state tree into the currency snapshot's existing calculatedStateProof — no tessellation fork needed. Closing as superseded. |
Summary
Phase 1 of the Constellation metagraph integration (spec: PR #107). Adds cryptographic state commitment infrastructure enabling Merkle inclusion proofs for individual fiber states.
Trello Card
Analysis: Validate Constellation metagraph integration approach (6996301a)
What Changed
Schema (backward-compatible, all new fields are
Option[_] = None)StateMachineFiberRecord(Records.scala)stateRoot: Option[Hash] = NoneMerklePatriciaProducer.statelessfrom top-levelstateDatafieldsCalculatedState(CalculatedState.scala)metagraphStateRoot: Option[Hash] = None{fiberId → fiberStateRoot}pairsFiberCombiner
computeStateRoot(stateData): builds in-round MPT fromMapValuefieldscreateStateMachineFiber: computes initialstateRootfrominitialDatahandleCommittedOutcome: recomputesstateRootfor all updated fibersML0Service
computeMetagraphStateRoot(state): MPT of all active fiber stateRootscombine: callscomputeMetagraphStateRootaftercombiner.foldLefthashCalculatedState: returnsmetagraphStateRootwhen present, falls back tostate.computeDigestTests
What's NOT in this PR (Phase 1B)
GET /state-proof/:fiberIdHTTP endpoint — tracked as follow-upMerklePatriciaProducer.levelDb)Implementation Notes
MerklePatriciaProducer.stateless[F](no persistence, rebuilt each round) — appropriate for Phase 1computeStateRootreturnsNonefor emptyMapValuestateData (nothing to hash)computeMetagraphStateRootreturnsNonewhen no fibers have stateRoots yet