diff --git a/README.md b/README.md index 49c01a8..9327b5d 100644 --- a/README.md +++ b/README.md @@ -1,224 +1,277 @@ -# Layered Nockchain Indexer and Query CLI +# iris-blocks -`iris-blocks` incrementally syncs nockchain data into a local SQLite database and exposes query commands over that indexed state. +### Layered Nockchain Indexer, Accounting Engine, and Query CLI -The database is built in layers (`l0` -> `l4`) where each layer depends on the previous one. Reorgs are handled by cascading invalidation from `l0` upwards. +A fast, self-hosted Nockchain indexer that syncs chain data into a local SQLite database and gives you full SQL access to blocks, transactions, notes, ownership, and double-entry accounting — all queryable in under a second. -## CLI +## What can you do with iris-blocks? -```bash -iris-blocks --help -``` +- **Time-travel to any block** — query chain state at any height: block metadata, transactions, coinbase rewards. +- **Track owners of complex notes** — resolves ownership across V0 public keys, V1 PKHs, multi-sig, and lock trees, continuously enriched as spend conditions are revealed. +- **Reveal public keys behind PKHs** — captures reverse mappings when PKH holders sign transactions. +- **Debit/credit tracking with instant CSV export** — full double-entry accounting ledger, export a wallet's complete history. +- **Extend with custom SQL** — plain SQLite database, build your own queries and APIs on top. Schema documented in [docs/SCHEMA.md](docs/SCHEMA.md). +- **Self-hosted and private** — your data stays on your machine. No third-party APIs, no rate limits. +- **Sub-second block parsing** — each new block parsed and all five layers derived in under a second. +- **Runs in the browser** — compiles to WebAssembly and runs entirely client-side. -### From source (Cargo) +--- -The CLI binary target is feature-gated. Use `--features binary` when running from source: +## Table of Contents -```bash -# Show CLI help -cargo +nightly run --features binary -- --help +- [Getting Started](#getting-started) +- [CLI Reference](#cli-reference) +- [Web Interface](#web-interface) +- [Using iris-blocks in JavaScript/TypeScript](#using-iris-blocks-in-javascripttypescript) +- [Architecture and Schema](#architecture-and-schema) +- [Reference](#reference) +- [License](#license) + +--- -# Example query -cargo +nightly run --features binary -- \ - --db nockchain.sqlite \ - balance
+## Getting Started + +The CLI is feature-gated — run all commands with: + +```bash +cargo run --features binary -- ``` -Commands: +There are two ways to get a populated database: download a pre-built snapshot, or sync directly from a Nockchain node. -- `sync`: connect to a node and update local DB state -- `balance
`: wallet balance view (ground truth from unspent notes) -- `tx `: transaction drilldown (spends, outputs, signers, credits, debits) -- `block `: block metadata, tx list, coinbase credits -- `status`: layer cursors + key table row counts -- `audit
`: wallet audit with selectable text/json view (`--view summary|notes|both`), plus CSV exports for summary accounting view (`--csv`) and detailed note ledger view (`--csv-notes`) +### Option A: Download a Chain Snapshot -### Data sources +We publish a new chain snapshot every 24 hours so you can start querying immediately without running a node. -- Local file mode (query commands): `--db ` points to a SQLite file -- Node mode (sync): `sync --connect ` +**1. Download the latest snapshot** from the release page: -Examples: +> [github.com/nockbox/iris-blocks/releases/tag/sample-db](https://github.com/nockbox/iris-blocks/releases/tag/sample-db) + +**2. Query it:** ```bash -# Initialize/upgrade schema locally (no node sync) -iris-blocks --db nockchain.sqlite sync --run-migrations +cargo run --features binary -- --db nockchain.sqlite status +cargo run --features binary -- --db nockchain.sqlite balance
+cargo run --features binary -- --db nockchain.sqlite tx +cargo run --features binary -- --db nockchain.sqlite audit
--csv +``` -# Sync from node -iris-blocks --db nockchain.sqlite sync --connect http://localhost:5555 --run-migrations +The snapshot is a standard SQLite file — you can also open it with any SQLite client (`sqlite3`, DB Browser, DBeaver, etc.) and write your own queries against the [documented schema](docs/SCHEMA.md). -# Query by PKH or legacy V0 key -iris-blocks --db nockchain.sqlite balance
-iris-blocks --db nockchain.sqlite audit
--csv wallet_flow_summary.csv +### Option B: Sync from a Nockchain Node (Private gRPC) -# Audit text/json view selection: -iris-blocks --db nockchain.sqlite audit
--view summary -iris-blocks --db nockchain.sqlite audit
--format json --view notes -iris-blocks --db nockchain.sqlite audit
--view both +For real-time data, sync iris-blocks directly from a Nockchain node's private gRPC interface. -# Auto-name summary CSV in current directory: -# nockchain_transactions_
.csv -iris-blocks --db nockchain.sqlite audit
--csv +> **Note:** NockBox does not offer hosted gRPC services. You need access to your own Nockchain node's private gRPC endpoint. -# Auto-name detailed notes CSV: -# nockchain_notes_
.csv -iris-blocks --db nockchain.sqlite audit
--csv-notes +**1. Start your Nockchain node** with the private gRPC API enabled (default port `5555`). -# Write auto-named summary CSV into a directory: -iris-blocks --db nockchain.sqlite audit
--csv /path/to/output-dir/ +**2. Initialize and sync:** -# Write both summary and detailed CSV in one run: -iris-blocks --db nockchain.sqlite audit
\ - --csv /path/to/output-dir/ \ - --csv-notes /path/to/output-dir/ +```bash +cargo run --features binary -- --db nockchain.sqlite sync \ + --connect http://localhost:5555 \ + --run-migrations ``` -## Units +This creates the database, runs migrations, connects to the node, and begins fetching blocks through all layers (L0–L4). It keeps running and syncing new blocks as they appear. -- All amounts are represented in **nicks**. -- This repository does not convert values to NOCK in CLI output. +**3. Query while syncing** from a second terminal: -## Address Input Semantics - -Address input type controls the query scope: - -- PKH input (`AddressType::Pkh`) -> V1-only accounting and note views. -- Public key input (`AddressType::DbPublicKey`) -> V0-only accounting and note views. +```bash +cargo run --features binary -- --db nockchain.sqlite balance
+``` -This split is intentional and prevents mixed V0/V1 balances when a PK and PKH are related. +**Selective layer syncing** — restrict which layers are derived if you don't need all of them: -## Audit/CSV Behavior +```bash +cargo run --features binary -- --db nockchain.sqlite sync \ + --connect http://localhost:5555 \ + --run-migrations \ + --only-enable-layers l1,l2 +``` -`audit` exposes two views: +**Re-derive without re-syncing** — when no `--connect` is provided, iris-blocks processes existing L0 blocks through the enabled upper layers, then exits: -- **Summary view** (`--view summary`, default `--csv`): - - recipient-level accounting rows (`incoming` / `outgoing` / `coinbase`) - - self-refund/change incoming rows are omitted from summary output - - transaction fees are always represented (including fee-only rows when needed) - - `txid` is always populated; coinbase rows use synthetic ids (`coinbase@`) -- **Notes view** (`--view notes`, `--csv-notes`): - - raw note-level ledger rows (`credit` / `debit` / `coinbase`) - - includes refund/change entries and counterparties +```bash +cargo run --features binary -- --db nockchain.sqlite sync --run-migrations +``` -## Wasm +--- -You may build this as a wasm module with: +## CLI Reference ``` -wasm-pack build --features=wasm --target web --out-dir pkg --scope nockbox +cargo run --features binary -- [--db ] ``` -If you run a direct wasm target check (`cargo +nightly check --features wasm --target wasm32-unknown-unknown`), -`sqlite-wasm-rs` requires a wasm-capable clang toolchain (Apple clang alone is not enough). -One working approach: +`--db` defaults to `nockchain.sqlite`. All query commands support `--format text` (default) or `--format json`. + +| Command | Description | +|---------|-------------| +| `sync` | Connect to a node and sync chain data into the local database | +| `balance
` | Wallet balance (ground truth from unspent notes) | +| `tx ` | Transaction drilldown: spends, outputs, signers, credits, debits | +| `block ` | Block metadata, transaction list, coinbase credits | +| `status` | Layer sync cursors and table row counts | +| `audit
` | Wallet audit with text/JSON views and CSV export | + +### Audit and CSV Export ```bash -nix shell nixpkgs#llvmPackages_18.clang nixpkgs#llvmPackages_18.llvm -c sh -lc ' - export CC_wasm32_unknown_unknown="$(command -v clang)" - export AR_wasm32_unknown_unknown="$(command -v llvm-ar)" - cargo +nightly check --features wasm --target wasm32-unknown-unknown -' -``` +# Summary CSV (auto-named: nockchain_transactions_
.csv) +cargo run --features binary -- --db nockchain.sqlite audit
--csv -And then use it with: +# Detailed note-level CSV (auto-named: nockchain_notes_
.csv) +cargo run --features binary -- --db nockchain.sqlite audit
--csv-notes -``` -import { BlockExporter, setLogging } from "@nockbox/iris-blocks"; -setLogging(); -const e = await new BlockExporter(["l0", "l1"], ":memory:", true, "http://localhost:8080"); +# Both CSVs into a directory +cargo run --features binary -- --db nockchain.sqlite audit
\ + --csv /path/to/output/ \ + --csv-notes /path/to/output/ + +# JSON output with both views +cargo run --features binary -- --db nockchain.sqlite audit
--format json --view both ``` -Data is not persisted. +**Summary view** (`--csv`) produces recipient-level accounting rows: `incoming`, `outgoing`, `coinbase` entries with running balance. Self-refund/change rows are excluded, fees are always represented. Coinbase rows use synthetic txids (`coinbase@`). -## Layer Summary +**Notes view** (`--csv-notes`) produces note-level ledger rows: `credit`, `debit`, `coinbase` entries including counterparties. -### L0 +### Sync Flags -Canonical block/transaction storage. +| Flag | Description | +|------|-------------| +| `--connect ` | gRPC URI of the Nockchain node (e.g. `http://localhost:5555`) | +| `--run-migrations` | Run schema migrations before syncing | +| `--rederive-layer ` | Reset a layer's cursor to 0 to re-derive it (l1–l4) | +| `--remove-layer ` | Drop and recreate a layer's tables (l1–l4) | +| `--only-enable-layers ` | Restrict which layers are derived | -- `blocks`: PK (`id`), UNIQUE (`height`), `version`, UNIQUE (`parent`), `timestamp`, `msg`, `jam` -- `transactions`: PK (`id`), `block_id`, `height`, `version`, `fee`, `total_size`, `jam` +--- -### L1 +## Web Interface -Note lifecycle (created/spent/unspent notes). +A hosted version is available at **[nockbox.github.io/iris-blocks](https://nockbox.github.io/iris-blocks/)** — no installation required, runs entirely in your browser via WebAssembly. -- `notes`: PK (`first`, `last`), `version`, `assets`, `coinbase`, `created_*`, `spent_*`, `jam` +- **Load a snapshot** — download a `.sqlite` file from the [daily snapshots](https://github.com/nockbox/iris-blocks/releases/tag/sample-db) and drag it into the interface to start querying. +- **Connect to a gRPC-Web endpoint** — enter your node's gRPC-Web URL and hit Start to sync blocks live. +- **Run SQL queries** — the built-in `nocksql>` terminal accepts arbitrary SQL against the full [schema](docs/SCHEMA.md). +- **JavaScript mode** — type `.js` to get an `iris>` REPL with access to `iris-wasm`, `sqlQuery()`, and `nounRows()` for decoding JAM blobs into nouns. +- **Export the database** — download the in-memory database as a `.sqlite` file for offline use. -### L2 +Data in the web interface is held in-memory and is not persisted across page reloads — use Export DB to save your work. -#### L2.1 +--- -Transaction internals and ordering. +## Using iris-blocks in JavaScript/TypeScript -- `tx_spends`: PK (`txid`, `z`), `version`, UNIQUE (`first`, `last`), `fee`, `height` -- `tx_seeds`: PK (`txid`, `z`, `idx`), `amount`, `first`, `height` -- `tx_outputs`: PK (`txid`, `idx`), UNIQUE (`first`, `last`), `assets`, `height` -- `tx_signers`: PK (`txid`, `z`, `pk`), `height` +Integrate iris-blocks directly into your own web application as a WASM library: -#### L2.2 +```javascript +import { BlockExporter, setLogging } from "@nockbox/iris-blocks"; -Hash reversals. +setLogging(); -- `name_to_lock`: PK (`first`), UNIQUE (`root`), `height`, `block_id` -- `pkh_to_pk`: PK (`pkh`), UNIQUE (`pk`), `height`, `block_id` +const exporter = await new BlockExporter({ + layers: ["l0", "l1", "l2", "l3", "l4"], + db_connect: ":memory:", + db_run_migrations: true, + remove_layer: null, + private_grpc_connect: "http://localhost:8080", + scry_no_pow: true, + verify_outputs: false, +}); + +const result = await exporter.query("SELECT COUNT(*) as cnt FROM blocks"); +const dbBytes = await exporter.exportDb(); +await exporter.stop(); +``` -#### L2.3 +### Building the WASM Package -Spend condition retrieval. +```bash +wasm-pack build --features=wasm --target web --out-dir pkg --scope nockbox +``` -- `lock_tree`: PK (`root`), `height`, `axis`, `hash` -- `spend_conditions`: PK (`hash`), UNIQUE (`txid`, `z`), `height`, `jam`, nullable `z` +`sqlite-wasm-rs` requires a WASM-capable clang toolchain. If Apple clang alone isn't sufficient: -_Null z implies non-witness derived spend condition (%lock note-data)_ +```bash +nix shell nixpkgs#llvmPackages_18.clang nixpkgs#llvmPackages_18.llvm -c sh -lc ' + export CC_wasm32_unknown_unknown="$(command -v clang)" + export AR_wasm32_unknown_unknown="$(command -v llvm-ar)" + cargo check --features wasm --target wasm32-unknown-unknown +' +``` + +--- -### L3 +## Architecture and Schema -Double entry accounting ledger. +iris-blocks builds its database in five layers. Each layer depends on the previous one; reorgs cascade invalidation from L0 upward. -- `credits`: PK (NULLABLE (`txid`), `first`, `height`), `block_id`, `amount` -- `debits`: PK (NULLABLE (`txid`), NULLABLE (`first`), `height`), `block_id`, `amount`, `fee` +``` +L0 Blocks & Transactions ← canonical chain data from node + └─ L1 Note Lifecycle ← created / spent / unspent notes + └─ L2 Transaction Detail ← spends, outputs, signers, hash reversals, spend conditions + └─ L3 Accounting ← double-entry credits & debits + └─ L4 Ownership ← resolved owner per note (pk / pkh / lock / musig) +``` -_Null txid/first imply coinbase_ +### Tables -### L4 +| Table | Layer | Purpose | +|-------|-------|---------| +| `blocks` | L0 | Block headers and metadata | +| `transactions` | L0 | Transaction data with fees and JAM blobs | +| `notes` | L1 | Note lifecycle (created, spent, unspent) | +| `tx_spends`, `tx_seeds`, `tx_outputs`, `tx_signers` | L2 | Transaction internals | +| `name_to_lock` | L2 | Note name to lock tree root mapping | +| `pkh_to_pk` | L2 | Public key hash to public key reverse lookup | +| `lock_tree`, `spend_conditions` | L2 | Lock trees and spend condition payloads | +| `credits` | L3 | Incoming value (including coinbase) | +| `debits` | L3 | Outgoing value with fees | +| `name_info` | L4 | Resolved ownership per note name | -Additional accounting information, more frequently recomputed. +Since this is plain SQLite, you can query it with any SQL-compatible tool and build custom APIs, dashboards, or analytics on top. For column types, indexes, join patterns, and query examples, see [docs/SCHEMA.md](docs/SCHEMA.md). -- `name_info`: PK (`first`, `height`), `version`, `owner_type`, `owner` +--- -_Reset behavior is height-based: delete `name_info` rows with `height >= next_block_height`._ +## Reference -_Update behavior is two-phase per block:_ -- _process newly revealed `spend_conditions` + `lock_tree` roots (resolve owner as `pkh` / `musig` / `lock`), then insert `version=1` name rows;_ -- _process newly created notes missing name metadata (V0 decode to `pk` / `musig`; V1 defaults to `lock`, with V1 coinbase first names resolved to `pkh`) and insert versioned rows._ +### Units -_Reporting/query behavior uses the latest `name_info` row per `first` (`ORDER BY height DESC LIMIT 1`)._ +All amounts are in **nicks** (native chain unit). `65536 nicks = 1 NOCK`. The CLI does not convert to NOCK. -### Upgrading existing databases +### Address Input -If your SQLite DB was created before `name_info` replaced `credit_info`, run: +The type of address you pass determines the query scope: -```bash -iris-blocks --db sync --remove-layer l4 -iris-blocks --db sync --run-migrations -``` +- **PKH** (public key hash) — queries V1 notes only (standard wallet queries) +- **Public key** — queries V0 notes only (legacy wallet queries) -This recreates only L4 with the new schema and avoids `no such table: name_info` runtime errors. +This split prevents mixed V0/V1 balances when a public key and its derived PKH are related. -## Recipients +### Recipient Types -- `pk`: resolved or v0 public key recipient -- `pkh`: public key hash recipient -- `lock`: unresolved or multi-owner lock-level recipient -- `musig`: multi-sig recipient +| Type | Meaning | +|------|---------| +| `pk` | Resolved or V0 public key | +| `pkh` | Public key hash (V1) | +| `lock` | Unresolved or multi-owner lock tree | +| `musig` | Multi-signature | -## Address mapping model +### Upgrading Existing Databases -Observed mapping chain: +If your database was created before `name_info` replaced `credit_info`: -```text -pk -> pkh -> lock -> name(first) +```bash +cargo run --features binary -- --db sync --remove-layer l4 +cargo run --features binary -- --db sync --run-migrations ``` -For full schema details, joins, and query patterns, see [docs/SCHEMA.md](docs/SCHEMA.md). \ No newline at end of file +--- + +## License + +[MIT](LICENSE)