diff --git a/docs/rfcs/0002-graph-visualization.md b/docs/rfcs/0002-graph-visualization.md new file mode 100644 index 00000000..98b1c579 --- /dev/null +++ b/docs/rfcs/0002-graph-visualization.md @@ -0,0 +1,195 @@ +# RFC 0002: Graph visualization export (`export --format graph-json` + `omnigraph viz`) + +| | | +|---|---| +| **Status** | Proposed | +| **Author(s)** | Shreya Sharma (@shreya-sharma_data) | +| **Discussion** | None yet — opened directly on the public RFC track (open to external contributors per [README](README.md)) | +| **Implementation** | TBD | + +## Summary + +Give operators a first-class way to *see* a graph. Add a stable +`export --format graph-json` that emits a documented `{ nodes, edges, schema }` +interchange shape, and a convenience `omnigraph viz` that renders it as a single +**self-contained, dependency-free interactive HTML file** (force-directed +node-link diagram; filter by type, hover to highlight a node's neighborhood, +click for its properties and every relationship in/out). Both are read-only, +honor existing branch/snapshot addressing and Cedar `read` policy, and add no +on-disk or wire-format surface. + +## Motivation + +Today there is no way to look at a graph. `export` streams row-oriented JSONL +(one record per node/edge), which is perfect for reload/bridging but useless for +comprehension — you cannot answer "what does this graph *look like*, and how is +this node connected?" without hand-writing a converter and wiring up an external +tool every time. Every operator building a company-brain / agent-memory graph +hits this on day one. + +Concretely, the current gap forces a multi-step ritual for something that should +be one command: + +1. `omnigraph export` → JSONL (and see the sharp edge below). +2. Hand-write a script to reshape JSONL into some tool's node/edge model. +3. Import into Cytoscape/Gephi/a bespoke D3 page, re-doing type coloring and + layout each time. + +This RFC argues the long-run liability of *not* having this: visualization is a +recurring, universal need for a knowledge-graph product, and leaving it to +ad-hoc per-user scripts means everyone re-solves it, inconsistently, and the +graph model's own semantics (typed nodes, directed edges, branches) never show +up in the picture. + +**A sharp edge this also fixes.** `omnigraph export` currently fails on a graph +created by `init` + `load` with: + +``` +storage read failed for '.../graph.omni/_schema.pg': + Object ... /_schema.pg not found (os error 2) +``` + +`export` reads schema from a `_schema.pg` sidecar that `init` does not write, so +export is unusable out of the box unless the operator manually drops the schema +file next to the datasets. The schema already lives in the catalog/`__manifest`; +the read path proposed here reconstructs types from there, removing the sidecar +dependency (see Reference-level design). This is the originating pain point. + +**Prototype (proof of concept).** A working prototype exists: a self-contained +HTML renderer fed by `export` output, exercised on a 27-node / 46-edge +"company-brain" graph (People, Teams, Projects, Technologies, Documents, +Decisions). It renders fully offline (no CDN — a hard requirement, see below), +runs a force layout in vanilla JS/SVG, colors nodes by type and sizes them by +degree, highlights a node's neighborhood on hover, and shows a node's full +in/out relationship list on click. It confirms the approach is viable in a +single ~25 KB file with zero runtime dependencies. This RFC proposes folding +that capability into the CLI as a supported, tested surface. + +## Guide-level explanation + +Two new read-only surfaces on the existing addressing model (`--store` / a +positional URI / `--server --graph` / `--branch` / `--snapshot`): + +**1. A stable graph-shaped export.** + +```bash +# emit a { nodes, edges, schema } document for any branch or snapshot +omnigraph export --store ./graph.omni --format graph-json > graph.json +omnigraph export --server intel-dev --graph spike --branch review --format graph-json +``` + +`graph-json` is a documented interchange shape that external tools (Cytoscape, +Gephi via a thin adapter, D3/Sigma, or the bundled renderer) can consume without +re-deriving node/edge structure. Shape: + +```json +{ + "graph": { "branch": "main", "snapshot": "01K…", "node_count": 27, "edge_count": 46 }, + "schema": { "node_types": ["Person", "Team", …], "edge_types": ["MemberOf", …] }, + "nodes": [ { "id": "carol@nw.io", "type": "Person", "props": { "name": "Carol Diaz", … } } ], + "edges": [ { "type": "MemberOf", "source": "carol@nw.io", "target": "platform", "props": {} } ] +} +``` + +**2. A one-command interactive picture.** + +```bash +omnigraph viz --store ./graph.omni --out graph.html # writes a self-contained HTML file +omnigraph viz --server intel-dev --graph spike --branch agent/ingest-42 --out review.html +``` + +`viz` reads the same `graph-json` internally and writes one self-contained HTML +file (no external requests — CDN-free, works on an air-gapped host, matching +the project's on-prem posture). Opening it: nodes colored by type with a legend +(click a type to filter it in/out), hover to highlight a node and its immediate +neighbors, click to pin a details panel listing the node's properties and every +edge in and out (`→ Uses Rust`, `← WorksOn Carol Diaz`), plus drag / zoom / pan. + +Because it addresses through the normal read path, you can visualize **any +branch** or a **time-travel snapshot** — e.g. render an agent's isolated branch +before merge, or diff two branches by eye. + +## Reference-level design + +- **Crate / placement:** `omnigraph-cli`. `graph-json` is a new `--format` on the + existing `export` command (alongside `jsonl`); `viz` is a new sibling command + in the same command family. No engine (`omnigraph`) API change is required — + both consume the existing snapshot read + catalog surface the current `export` + already uses. +- **Read path:** open a read-only `Snapshot` at the resolved branch/snapshot + (existing behavior). Enumerate node/edge types from the **catalog derived from + `__manifest`** (not a `_schema.pg` sidecar), scan each type's dataset, and + project rows to `{id, type, props}` / `{type, source, target, props}`. This + removes the current sidecar dependency that breaks `export` on freshly + `init`'d graphs. Everything runs under one snapshot, so the picture is a + consistent point-in-time view (no torn reads). +- **`graph-json` shape:** as above; versioned implicitly by a top-level + `"format": "omnigraph.graph-json/v1"` field so the interchange contract can + evolve without breaking consumers. Blob columns are emitted as a size/omitted + marker rather than inline bytes (a picture doesn't need the payload). +- **`viz` rendering:** a template embedded in the binary (`include_str!`) with + the `graph-json` injected at a single placeholder; force-directed layout, + interaction, and styling are vanilla JS/SVG with **no external dependencies** + and **no network requests** (strict-CSP / offline friendly). Output is one + `.html` file. Large graphs: `viz` warns and suggests `--format graph-json` + into a dedicated tool above a node/edge threshold (rendering budget, not a + hard cap). +- **Policy:** on a served graph, `viz`/`graph-json` are the `read` action and go + through the same Cedar gate as any read; no new action is introduced. On a + direct `--store` graph they behave like `export` does today. +- **Errors:** unreadable/again-quarantined graph surfaces the existing typed + read errors; `--out` path collisions follow the same overwrite/`--force` + convention as other file-writing commands. + +## Invariants & deny-list check + +No Hard Invariant in [../dev/invariants.md](../dev/invariants.md) is weakened. + +- **Read-only:** no mutation, no HEAD advance, no manifest write; a single + snapshot is held for the read (snapshot-isolation preserved). +- **No format/substrate change:** no on-disk or wire format changes; no new + substrate dependency. `graph-json` is an *output* interchange shape, not a + storage format. +- **Policy is engine-wide:** the read goes through the same Cedar `read` gate; + no new action or bypass is added. +- **Deny-list:** touches none of it — this is additive, read-only CLI surface. + +## Drawbacks & alternatives + +- **Do nothing / external tools only.** Operators keep hand-rolling converters + per tool. Rejected: it's a universal day-one need and the graph's own + semantics (types, direction, branches) never make it into the view. A stable + `graph-json` at minimum removes the reshape tax even if `viz` is dropped. +- **Ship a web UI in `omnigraph-server`.** A live server-rendered graph browser. + Rejected for now: much larger surface, conflicts with the cluster-only / + policy-scoped serving model, and pulls a front-end stack into a Rust-only + substrate. A static file from the CLI is far cheaper and covers the core need; + a server UI could be a later RFC that builds on `graph-json`. +- **Embed a JS graph library** (Cytoscape.js/Sigma) in the template. Rejected as + the default: it either needs a CDN (violates offline/on-prem) or vendoring a + large bundle. A compact vanilla renderer keeps the file small and + dependency-free; the `graph-json` output remains the escape hatch for anyone + who wants a heavyweight tool. +- **`graph-json` scope.** Keeping it a stable, documented shape is a small + ongoing compatibility commitment; the `format` version field bounds that cost. + +## Reversibility + +High. `viz` (HTML output) is a pure convenience and trivially removable. The one +semi-durable surface is the `graph-json` interchange shape — versioned via its +`format` field so it can evolve or be deprecated without breaking existing +consumers. No on-disk, wire, or substrate commitment, so evidence demand is low +(a CLI feature, not a format decision). The `_schema.pg`→catalog read change is +strictly a bug fix that makes `export` work as documented. + +## Unresolved questions + +- Should `graph-json` be a new `--format` on `export`, or its own `graph export` + subcommand? (This RFC proposes the former to minimize surface.) +- The node/edge threshold at which `viz` should refuse/warn and defer to an + external tool — pick a default from real large-graph rendering behavior. +- Should the sidecar-removal (`_schema.pg` → catalog read) ship independently as + a bug fix ahead of this RFC, since `export` is broken out of the box today? +- Optional follow-ups (out of scope here): edge-label rendering density controls, + a two-branch visual diff mode, and a GEXF/Cytoscape adapter alongside + `graph-json`.