Skip to content

feat(store): tier-2 value ledger — encrypted-at-rest per-step inputs + monitor detail (M0) - #230

Merged
rhCat merged 3 commits into
mainfrom
feat/pg-provenance-ledger
Jul 22, 2026
Merged

feat(store): tier-2 value ledger — encrypted-at-rest per-step inputs + monitor detail (M0)#230
rhCat merged 3 commits into
mainfrom
feat/pg-provenance-ledger

Conversation

@rhCat

@rhCat rhCat commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

M0 of the two-tier provenance ledger (docs/pg-provenance-ledger.md). Makes each governed tool-use inspectable and reproducible — you can now review the actual parameters of a run in the fleet + local monitor — without weakening the value-free wire/monitor.

What lands

  • infra/store/valuecrypt.py — envelope encryption (X25519 + HKDF + AES-256-GCM, no new dependency). Fresh per-blob DEK wrapped to a recipient set (node standalone; +mothership at fleet join via rewrap, no re-encryption). values_sha commits the plaintext, never ciphertext.
  • backendrun_values table (sqlite live; PG inert-parallel with raw BYTEA + JSONB) + record_values/get_values on both adapters; store_selftest interface-conformance covers them.
  • mirror — a new value-free values queue op (single drain writer; backend only, never the chain) + values_sha added to _SAFE_EVENT_KEYS.
  • govd Store — node recipient key at boot (0600, default-on, inert if disabled); record_values encrypts + enqueues at the one WS point where plaintext exists and binds values_sha into the tier-1 chain step event; decrypt_values for the operator view. New operator-gated GET /monitor/values/<run_id> (node-local decrypt).
  • monitors — local dashboard run tab + fleetdash run page each gain a "tool-use detail" reveal that decrypts step inputs on request (textContent-only, no inline-onclick sink); fleetdash _EVENT_KEYS + proxy allowlist gain values_sha / monitor/values.
  • teststest_value_ledger.py (9) + 3 integrated delegated cases; full suite green locally (194 passed, 1 PG skip).

Guarantees preserved

Secrets are never recorded (declared-subset, secret-filtered; *_FILE pointers stay node-local). Off-node at rest = ciphertext only. The tier-1 chain and the :5773 wire stay value-free — only the values_sha hash crosses into them.

Ouroboros self-inspection

Ran the transmutation mechanical pass (alembic putrefactio shape-extraction + LVP audit) and citrinitas intent-alignment on the new code. Intent-alignment on encrypt confirms the blueprint matches the documented contract (empty-recipient guard gates before sealing; fresh DEK/nonce; per-recipient wrap). All mechanical leads adjudicated to false-positive or pre-existing (Rust .unwrap() heuristic on Python; bounded-loop misreads on the daemon drain) — no real defect in the new code.

⚠️ Touches the enforcement surface (govd.py, mirror.py) — needs the CI gates + multi-agent adversarial review before merge, per repo discipline. Opening for review, not auto-merge.

🤖 Generated with Claude Code

rhCat and others added 2 commits July 22, 2026 02:19
…+ monitor detail (M0)

Makes each governed tool-use inspectable and reproducible WITHOUT weakening the
value-free wire/monitor (docs/pg-provenance-ledger.md, M0).

- infra/store/valuecrypt.py: envelope encryption (X25519+HKDF+AES-GCM, no new dep).
  Fresh per-blob DEK wrapped to a recipient set (node standalone; +mothership at
  fleet join via rewrap without re-encrypting). values_sha commits the PLAINTEXT.
- backend: run_values table (sqlite live; PG inert-parallel, raw BYTEA + JSONB) +
  record_values/get_values on both adapters; store_selftest interface-conformance
  covers them.
- mirror: new value-free `values` queue op (single drain writer; backend only,
  never the chain) + values_sha added to _SAFE_EVENT_KEYS.
- govd Store: node recipient key at boot (0600, default-on, inert if disabled);
  record_values encrypts + enqueues at the ONE WS point plaintext exists, binds
  values_sha into the tier-1 chain step event; decrypt_values for the operator view.
  New operator-gated endpoint GET /monitor/values/<run_id> (node-local decrypt).
- monitors: local dashboard run tab + fleetdash run page each gain a "tool-use
  detail" reveal that decrypts step inputs on request (textContent-only, no
  inline-onclick sink); fleetdash _EVENT_KEYS + proxy allowlist gain values_sha /
  monitor/values.
- tests: test_value_ledger.py (9) + 3 integrated delegated cases. Full suite green.

Secrets never recorded (declared-subset, secret-filtered, *_FILE pointers stay
node-local). Off-node at rest = ciphertext only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r-run, PG-safe read

Four-lens adversarial review (crypto / value-free-boundary / web-surface /
enforcement-path) found four real defects; this fixes all of them.

- [crypto Medium] values_sha was an UNSALTED hash of low-entropy plaintext,
  living in the value-free chain/index — the same DB as the ciphertext — so a
  backup/replica/over-granted role could brute-force the values without touching
  the envelope (contradicting the module's own at-rest claim). Now SALTED:
  sha256(salt ‖ plaintext) with a fresh salt sealed INSIDE the AEAD blob; an
  authorized decryptor recovers salt+plaintext and re-verifies. Full-width keyid
  too (no rewrap truncation-collision/evict).
- [enforcement #2/#3] record_values ran BEFORE execute_step with a first-wins
  upsert, so a refused/retried step orphaned a value row and a retry with
  corrected values desynced the chain sha from the stored blob. Now records ONLY
  on a terminal step_result (ok/error), at-most-once, so the stored blob and the
  chain-bound sha always agree; a refusal records nothing.
- [enforcement #1] decrypt_values read the drain worker's shared (unguarded
  psycopg) connection — a monitor read racing a drain write could drop a tier-1
  chain-index row. Now reads through a FRESH backend connection (reconciler
  discipline).
- [key siting] the node key defaulted into the replicated record root
  (backup+key = broken at-rest); govd now warns loudly and the doc requires
  value_ledger.node_key_file off the backup path.

Residuals documented (accepted): name-based secret filter, plaintext view behind
the shared monitor token, values queue pressure. Tests updated for the salted
commitment + a retry-desync regression. infra/ ruff clean; store selftest green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rhCat

rhCat commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

Adversarial review — 4 lenses, 4 real fixes (commit 24bba89)

Ran a four-agent adversarial review (each prompted to refute): crypto correctness, value-free boundary, web/monitor surface, enforcement-path integrity. Four real defects surfaced and are now fixed; residuals documented.

Fixed

  1. Unsalted commitment = preimage oracle (crypto, Medium). values_sha was sha256(plaintext) over low-entropy values, sitting in the value-free chain/index — the same DB as the ciphertext. A backup / replica / over-granted DB role could brute-force the values without touching the envelope, defeating encrypt-at-rest for exactly the values the ledger records. → Now sha256(salt ‖ plaintext) with a fresh salt sealed inside the AEAD blob; an authorized decryptor recovers salt+plaintext and re-verifies. Full-width keyid too (no rewrap truncation-collision/evict).
  2. Retry desync + orphan rows (enforcement). record_values ran before execute_step with a first-wins upsert, so a refused/retried step orphaned a value row and a retry with corrected values left the chain attesting one sha while the stored blob held the old values. → Now records only on a terminal step_result (at-most-once), so the stored blob and the chain-bound sha always agree; a refusal records nothing.
  3. Monitor read on the unguarded PG connection (enforcement). decrypt_values read the drain worker's shared psycopg connection (not thread-guarded) — a monitor read racing a drain write could drop a tier-1 chain-index row. → Now reads through a fresh backend connection, the same discipline the reconciler uses.
  4. Key co-located with ciphertext (key siting). The node key defaulted into the replicated record root (a backup holding both key and ciphertext breaks at-rest). → govd warns loudly; the doc now requires value_ledger.node_key_file off the backup path.

Accepted residuals (documented, no code change)

  • Secret filter is name-based (a sensitive value under a benign key would be recorded, encrypted). Same set exod already receives.
  • The plaintext /monitor/values view is gated by the same monitor token as the value-free /monitor/run — an operator carve-out, bounded by "authenticated and on a node holding the key."
  • values jobs share the bounded mirror queue (≈2× enqueue volume); large queue makes drop rare.

Crypto and web lenses found no exploitable bug (all LOW/INFO residuals). infra/ ruff clean, store selftest green, full affected suite green.

…-monitor plaintext path

The fleet monitor reviews plaintext via the LIVE PROXY: fleetdash → /proxy/<node>/
monitor/values/<run_id> → the node decrypts with its own recipient key → plaintext
back. This adds the first over-HTTP test of that node endpoint (unit tests covered
the Store; this covers the wire):
  (1) 403 without the monitor token (fail-closed gate),
  (2) 200 with the decrypted per-step values when authed,
  (3) the value-free /monitor/run for the SAME run carries only values_sha — no
      plaintext leaks into the value-free view.

Confirms decryption stays node-side (no central key, no central ciphertext) — the
live-proxy oversight model.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rhCat
rhCat merged commit 7de8559 into main Jul 22, 2026
2 checks passed
@rhCat
rhCat deleted the feat/pg-provenance-ledger branch July 22, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant