diff --git a/.env.testing-artifacts b/.env.testing-artifacts index 92b2fbae2..e1e14d228 100644 --- a/.env.testing-artifacts +++ b/.env.testing-artifacts @@ -5,7 +5,7 @@ # zcashd Git tag (https://github.com/zcash/zcash/releases) ZCASH_VERSION=6.20.0 -ZEBRA_VERSION=6.0.0-rc.0 +ZEBRA_VERSION=6.0.0 # zcash-devtool git ref (https://github.com/zingolabs/zcash-devtool) — # the wallet client driven by wallet-tests via zcash_local_net, built diff --git a/.github/workflows/publish-dry-run.yml b/.github/workflows/publish-dry-run.yml index 403de3465..f2fb2e948 100644 --- a/.github/workflows/publish-dry-run.yml +++ b/.github/workflows/publish-dry-run.yml @@ -13,20 +13,11 @@ jobs: publish-dry-run: name: cargo publish --dry-run runs-on: ubuntu-latest - # Blocking context (pushes to rc/** or stable, and PRs targeting them): - # this job MUST pass. Advisory context (everything else): findings inform - # but the run stays green. Keep this predicate the exact inverse of the - # "Compute check mode" step below — job-level continue-on-error cannot - # read step outputs, so the two encode the same rule twice. - continue-on-error: >- - ${{ - github.event_name == 'pull_request' - && !startsWith(github.base_ref, 'rc/') - && github.base_ref != 'stable' - || github.event_name != 'pull_request' - && !startsWith(github.ref_name, 'rc/') - && github.ref_name != 'stable' - }} + # Blocking context: direct pushes to rc/** or stable — the branch is in + # release-prep state and must publish cleanly. Advisory context: everything + # else, including PRs that *target* rc/** or stable. Those PRs carry + # unbumped dev code; version bumps happen on the RC branch after merge, so + # a publish failure at PR time is expected, not actionable. steps: - name: Checkout repository uses: actions/checkout@v7 @@ -37,11 +28,12 @@ jobs: - name: Compute check mode id: mode run: | - # Inverse of the job's continue-on-error predicate above. - if [[ "${{ github.event_name }}" == "pull_request" ]]; then - target="${{ github.base_ref }}" - else + if [[ "${{ github.event_name }}" == "push" ]]; then target="${{ github.ref_name }}" + else + # PRs (including those targeting rc/** or stable) are always + # advisory — they carry pre-bump code by design. + target="" fi if [[ "$target" == rc/* || "$target" == stable ]]; then echo "mode=blocking" >> "$GITHUB_OUTPUT" @@ -96,13 +88,18 @@ jobs: echo "============================================" echo "" echo "The workspace does not publish cleanly as a set. Common causes:" - echo " - a [patch.crates-io] override pulling in unreleased upstream code" + echo " - a [patch.crates-io] override pulling in unreleased upstream" + echo " code (published crates cannot depend on unpublished revisions)" echo " - a dependency on a git revision or path outside the workspace" echo " - a crate that does not compile from its packaged form" + echo " - new cross-crate API or features that exist in the workspace but" + echo " not in the published dependency; resolved by the version bump +" + echo " publish of that dependency at the next release" + echo "" + if [[ "${{ steps.mode.outputs.mode }}" == "advisory" ]]; then + echo "::warning::publish dry-run failed — the workspace would not release cleanly. Advisory on this branch, but it must be resolved before the next release." + exit 0 + fi + echo "::error::publish dry-run failed — the workspace would not release cleanly." exit 1 fi - - - name: Annotate on failure (advisory context only) - if: failure() && steps.mode.outputs.mode == 'advisory' - run: | - echo "::warning::publish dry-run failed — the workspace would not release cleanly. This must be resolved before the next release." diff --git a/CHANGELOG.md b/CHANGELOG.md index 621a927ce..e07073cae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,46 @@ and this library adheres to Rust's notion of ## Unreleased ### Changed +- `zaino-state`: `FetchService` and `StateService` are merged into a single + generic `NodeBackedIndexerService` (module + `zaino_state::indexer::node_backed_indexer`; the former `backends` module is + gone). The validator connection is now selected at runtime rather than by type: + `NodeBackedIndexerServiceConfig { common, connection }` carries a + `ValidatorConnectionType` of either `Rpc` (JSON-RPC, formerly `Fetch`) or + `Direct(DirectConnectionConfig)` (Zebra `ReadStateService`, formerly `State`). + The per-backend `Fetch/StateServiceConfig`, `Fetch/StateServiceError`, and + `BackendConfig` types are replaced by `NodeBackedIndexerServiceConfig`, + `NodeBackedIndexerServiceError`, and `ValidatorConnectionType`. +- **Breaking** — config: `zainod.toml`'s `backend` selector is renamed + `state` → `direct` and `fetch` → `rpc`. The legacy `"state"` / `"fetch"` + values are still accepted as aliases, so existing config files keep working. +- `zaino-state`: the `ChainIndex` trait is split into `ChainIndex` (the + wallet-essential core: chain/tx/address/mempool access) and a + `ChainIndexRpcExt: ChainIndex` extension (compact-block serving, subtree + roots, and the block-explorer / mining / node-passthrough RPCs). The split is + a provisional first pass to be refined into finer capability traits later. +- `zaino-state`: all remaining backend-split RPC functionality has moved out of + the `FetchService` (`JsonRpSeeConnector`) and `StateService` + (`ReadStateService`) backends and into `BlockchainSource` / + `ChainIndex`. Both backends now resolve every fetch through their `ChainIndex` + indexer — building responses from Zaino's own indexed state where possible and + delegating to the `ValidatorConnector` (`BlockchainSource`) only for + validator-only or passthrough data. Validator connection/syncer spawning also + moves into `ValidatorConnector::spawn_fetch` / `spawn_state`, so each + service/subscriber now holds only `{ indexer, data, config }`. This readies + the two backends for their eventual merge into a single + `ValidatorBackedIndexerService`. No behaviour change. +- TLS: zaino now installs rustls's **aws-lc-rs** CryptoProvider as its + preferred process-level default (was ring) and enables rustls's + `prefer-post-quantum` feature, so the X25519MLKEM768 hybrid key exchange + leads zaino's outbound handshakes (ADR-0006). Installation remains + first-install-wins: an embedder that installs a provider before zaino + keeps its choice. + +### Deprecated +- Classical TLS key exchange (X25519, SECP256R1, SECP384R1) is deprecated: + still offered and accepted for wallet compatibility, slated for refusal + once major wallet stacks negotiate hybrid key exchange (ADR-0006). - **Breaking** — config: `storage.database.sync_write_batch_bytes` (bytes) is renamed to `sync_write_batch_size` and given in **GiB** (default raised from 4 GiB to 32 GiB); this budget now also bounds the txout-set accumulator diff --git a/CONTEXT.md b/CONTEXT.md index 4d40dd72f..6246105d7 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -29,3 +29,70 @@ its packaged content differs. The tree cannot be released until that crate's version is bumped. An unchanged crate keeping its published version is not a violation. _Avoid_: stale version, forgotten bump + +### Chains and networks + +**The Public Testnet**: +The public Zcash test network, and only that. Its regimes are +non-hermetic — state is shared with other participants, and an epoch +the public chain has left (e.g. pre-NU6.3 once NU6.3 activates there) +cannot be re-entered. Always this exact phrase in prose; identifiers +use `the_pub_testnet` and types use `PubTestnet`. +_Avoid_: bare "Testnet"/"testnet", and especially "testnet" for any +locally-launched chain, even one launched under a testnet network kind + +**Regtest net**: +A hermetic, locally-launched chain whose activation heights the +launcher chooses. Every hermetic local net is a regtest net, whatever +network-kind flag it runs under. +_Avoid_: local testnet, custom testnet + +### Pools and upgrades + +**Ironwood / Orchard (era naming)**: +Eras, fixtures, and predicates that speak of shielded pools are named by +pool — Orchard, Ironwood — and a name that mentions one pool pairs with +the other pool's name, never with the upgrade's. **NU6.3** names only the +network upgrade itself: activation heights, consensus branch ID, +consensus rules. +_Avoid_: mixing vocabularies in one name or one sibling set (e.g. an +`ORCHARD_ONLY_*` fixture whose sibling is `NU6_3_ACTIVE_*` — the sibling +is `IRONWOOD_ONLY_*`) + +**Cross-address restriction**: +The post-NU6.3 rule the Orchard Action circuit enforces: "(g_d, pk_d) +of the output note must equal (g_d, pk_d) of the spent note" — the +output note must carry the same expanded receiver (diversified base +g_d, diversified transmission key pk_d) as its spent note, so each +Orchard action is either change to the spent note's own address or a +withdrawal (positive value balance). Orchard-to-Orchard transfers to +any other address — including another address of the same wallet — are +prohibited. A companion transaction-level rule forbids new value +entering the pool. Source: + +_Avoid_: "exit-only" (overclaims — same-receiver change still lands in +the pool and its commitment tree still grows) + +### TLS and cryptography + +**Preferred CryptoProvider**: +The TLS cryptography provider zaino installs as the process-wide default +when no provider is installed yet. A preference, not a mandate: an +embedder (e.g. zallet) that installs a provider before zaino keeps its +choice, and zaino handshakes through it. +_Avoid_: "enforced provider" (implies zaino overrides an embedder's +already-installed provider; it never does) + +**Hybrid key exchange**: +A TLS 1.3 key-exchange group combining a classical curve with a +post-quantum KEM (key-encapsulation mechanism), e.g. X25519MLKEM768. +Secure if either component holds. Hybrids are the post-quantum +deployment vehicle, not a classical fallback. +_Avoid_: calling hybrids "classical" because they contain a classical +component + +**Classical key exchange**: +A key-exchange group with no post-quantum component (X25519, SECP256R1, +SECP384R1). Deprecated in zaino: still accepted for client +compatibility, slated for refusal once major wallet stacks negotiate +hybrids. diff --git a/Cargo.lock b/Cargo.lock index f725865bb..798fd33aa 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -173,9 +173,9 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "aws-lc-rs" -version = "1.17.0" +version = "1.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" +checksum = "4342d8937fc7e5dd9b1c60292261c0670c882a2cd1719cfc11b1af41731e32ad" dependencies = [ "aws-lc-sys", "zeroize", @@ -183,14 +183,15 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.41.0" +version = "0.42.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" +checksum = "6d9ceb1da931507a12f4fccea479dccd00da1943e1b4ae72d8e502d707361444" dependencies = [ "cc", "cmake", "dunce", "fs_extra", + "pkg-config", ] [[package]] @@ -305,26 +306,6 @@ dependencies = [ "serde", ] -[[package]] -name = "bindgen" -version = "0.69.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088" -dependencies = [ - "bitflags 2.13.0", - "cexpr", - "clang-sys", - "itertools 0.12.1", - "lazy_static", - "lazycell", - "proc-macro2", - "quote", - "regex", - "rustc-hash 1.1.0", - "shlex 1.3.0", - "syn 2.0.118", -] - [[package]] name = "bindgen" version = "0.72.1" @@ -338,7 +319,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "rustc-hash 2.1.2", + "rustc-hash", "shlex 1.3.0", "syn 2.0.118", ] @@ -688,14 +669,9 @@ name = "clientless" version = "0.2.0" dependencies = [ "anyhow", - "corez", "futures", "hex", - "serde_json", - "tempfile", "tokio", - "tower 0.4.13", - "tracing", "wire_serialized_transaction_test_data", "zaino-common", "zaino-fetch", @@ -707,7 +683,6 @@ dependencies = [ "zebra-chain", "zebra-rpc", "zebra-state", - "zip32", ] [[package]] @@ -2246,15 +2221,6 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" -[[package]] -name = "itertools" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.13.0" @@ -2373,7 +2339,7 @@ dependencies = [ "jsonrpsee-types", "parking_lot", "rand 0.8.6", - "rustc-hash 2.1.2", + "rustc-hash", "serde", "serde_json", "thiserror 1.0.69", @@ -2480,12 +2446,6 @@ dependencies = [ "spin", ] -[[package]] -name = "lazycell" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55" - [[package]] name = "left-right" version = "0.11.7" @@ -2530,14 +2490,13 @@ dependencies = [ [[package]] name = "librocksdb-sys" -version = "0.16.0+8.10.0" +version = "0.17.3+10.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce3d60bc059831dc1c83903fb45c103f75db65c5a7bf22272764d9cc683e348c" +checksum = "cef2a00ee60fe526157c9023edab23943fae1ce2ab6f4abb2a807c1746835de9" dependencies = [ - "bindgen 0.69.5", + "bindgen", "bzip2-sys", "cc", - "glob", "libc", "libz-sys", "lz4-sys", @@ -2560,7 +2519,7 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f8ce05b56f3cbc65ec7d0908adb308ed91281e022f61c8c3a0c9388b5380b17" dependencies = [ - "bindgen 0.72.1", + "bindgen", "cc", "thiserror 2.0.18", "tracing", @@ -2808,12 +2767,6 @@ version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "549e471b99ccaf2f89101bec68f4d244457d5a95a9c3d0672e9564124397741d" -[[package]] -name = "nonempty" -version = "0.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9737e026353e5cd0736f98eddae28665118eb6f6600902a7f50db585621fecb6" - [[package]] name = "nu-ansi-term" version = "0.50.3" @@ -2952,9 +2905,9 @@ checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" [[package]] name = "orchard" -version = "0.15.0-pre.1" +version = "0.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8e277dd4b46f5d06deae3ffb8af1a951e8622368f028c2a4d6fe59339566403" +checksum = "cca2ede6a4a77bd729eed3be9303d98a08b217edc85190dcf4dbe004086d5a65" dependencies = [ "aes", "bitvec", @@ -2971,7 +2924,7 @@ dependencies = [ "incrementalmerkletree", "lazy_static", "memuse", - "nonempty 0.11.0", + "nonempty", "pasta_curves", "rand 0.8.6", "rand_core 0.6.4", @@ -3472,7 +3425,7 @@ dependencies = [ "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash 2.1.2", + "rustc-hash", "rustls", "socket2", "thiserror 2.0.18", @@ -3487,13 +3440,12 @@ version = "0.11.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e" dependencies = [ - "aws-lc-rs", "bytes", "getrandom 0.3.4", "lru-slab", "rand 0.9.4", "ring", - "rustc-hash 2.1.2", + "rustc-hash", "rustls", "rustls-pki-types", "slab", @@ -3877,7 +3829,6 @@ dependencies = [ "log", "percent-encoding", "pin-project-lite", - "quinn", "rustls", "rustls-pki-types", "rustls-platform-verifier", @@ -3937,9 +3888,9 @@ dependencies = [ [[package]] name = "rocksdb" -version = "0.22.0" +version = "0.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6bd13e55d6d7b8cd0ea569161127567cd587676c99f4472f779a0279aa60a7a7" +checksum = "ddb7af00d2b17dbd07d82c0063e25411959748ff03e8d4f96134c2ff41fce34f" dependencies = [ "libc", "librocksdb-sys", @@ -3957,12 +3908,6 @@ version = "0.1.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d" -[[package]] -name = "rustc-hash" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" - [[package]] name = "rustc-hash" version = "2.1.2" @@ -5017,9 +4962,9 @@ dependencies = [ [[package]] name = "tower-batch-control" -version = "1.0.1" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e6cf52578f98b4da47335c26c4f883f7993b1a9b9d2f5420eb8dbfd5dd19a28" +checksum = "0a49cdaf2b00c1150c93a8dbbb63cae193573903a3c9a07b5d4d61251b9e32f0" dependencies = [ "futures", "futures-core", @@ -5034,9 +4979,9 @@ dependencies = [ [[package]] name = "tower-fallback" -version = "0.2.41" +version = "0.2.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4434e19ee996ee5c6aa42f11463a355138452592e5c5b5b73b6f0f19534556af" +checksum = "700cd6732c9c9e2c77501cc663708b2229abc9e8fa27d3deae15126330a1ed87" dependencies = [ "futures-core", "pin-project", @@ -5942,10 +5887,8 @@ dependencies = [ name = "zaino-common" version = "0.2.0" dependencies = [ - "hex", - "nu-ansi-term", + "rustls", "serde", - "thiserror 2.0.18", "time", "tracing", "tracing-subscriber", @@ -6006,6 +5949,7 @@ dependencies = [ "tonic", "tower 0.4.13", "tracing", + "zaino-common", "zaino-fetch", "zaino-proto", "zaino-state", @@ -6020,11 +5964,9 @@ dependencies = [ "arc-swap", "bitflags 2.13.0", "blake2", - "bs58", "chrono", "corez", "dashmap", - "derive_more", "futures", "hex", "incrementalmerkletree", @@ -6032,11 +5974,8 @@ dependencies = [ "lmdb", "lmdb-sys", "metrics", - "nonempty 0.12.0", - "once_cell", "primitive-types 0.14.0", "proptest", - "prost", "rand 0.10.1", "reqwest 0.13.1", "sapling-crypto", @@ -6105,22 +6044,20 @@ dependencies = [ "tokio", "toml", "tracing", - "tracing-subscriber", "zaino-common", "zaino-fetch", "zaino-serve", "zaino-state", "zcash_address", "zcash_protocol", - "zebra-chain", "zebra-state", ] [[package]] name = "zcash_address" -version = "0.13.0-pre.0" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8cf2918e73eff76388cda87695a6e7398f96a2e3383a0de7b77729a204ec00a0" +checksum = "5a854b28c07dba372f4410ea8ad62b4bf7d5c2bf8be32fc4b31bc0db6521a975" dependencies = [ "bech32", "bs58", @@ -6138,14 +6075,14 @@ checksum = "1440921903cdb86133fb9e2fe800be488015db2939a30bedb413078a1acb0306" dependencies = [ "corez", "hex", - "nonempty 0.11.0", + "nonempty", ] [[package]] name = "zcash_history" -version = "0.5.0-pre.0" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82e8634d011026cb181cb67b2a412e601dc344a2827b9c576fe3a727fdebf441" +checksum = "69d2ed9a9fa7b466a7adedab1ad5a21f8330e4943756912fc776cf7f3beae32b" dependencies = [ "blake2b_simd", "byteorder", @@ -6154,9 +6091,9 @@ dependencies = [ [[package]] name = "zcash_keys" -version = "0.15.0-pre.0" +version = "0.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a0bac3a9e5b0d954684ba1f07386d55b5ae4588b3ba2d93cad05ee09f3e0947" +checksum = "36391ebfce7df2510c6564f79e608ed93f1c0692c6464e2397b248e06dae3912" dependencies = [ "bech32", "blake2b_simd", @@ -6166,7 +6103,7 @@ dependencies = [ "document-features", "group", "memuse", - "nonempty 0.11.0", + "nonempty", "orchard", "rand_core 0.6.4", "sapling-crypto", @@ -6183,7 +6120,7 @@ dependencies = [ [[package]] name = "zcash_local_net" version = "0.7.0" -source = "git+https://github.com/zingolabs/infrastructure.git?rev=0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646#0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646" +source = "git+https://github.com/zingolabs/infrastructure.git?rev=85502cd0b5faf9308e590865dae51bf31fdd342e#85502cd0b5faf9308e590865dae51bf31fdd342e" dependencies = [ "hex", "reqwest 0.12.28", @@ -6213,9 +6150,9 @@ dependencies = [ [[package]] name = "zcash_primitives" -version = "0.29.0-pre.0" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba7bfe66975658f44dba87d535f9ebb9fb4c9c0c4fecca3f5c40cbe1583dece6" +checksum = "bdbeccc05bfe63b6dee9e989c6ff5027421741562a4853c36504dd621f6a81b1" dependencies = [ "blake2b_simd", "block-buffer 0.11.0-rc.3", @@ -6228,7 +6165,7 @@ dependencies = [ "incrementalmerkletree", "jubjub", "memuse", - "nonempty 0.11.0", + "nonempty", "orchard", "rand_core 0.6.4", "redjubjub", @@ -6244,9 +6181,9 @@ dependencies = [ [[package]] name = "zcash_proofs" -version = "0.29.0-pre.0" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39b90964ffe6bdc314368c7b849aaa6dcc2b495249a3bf1b9bfbdc92ba4e4f6" +checksum = "b91fb0b420fb66a765f1fa92ab7a6b631aa54c08415415ef6185256826e74fbd" dependencies = [ "bellman", "blake2b_simd", @@ -6267,9 +6204,9 @@ dependencies = [ [[package]] name = "zcash_protocol" -version = "0.10.0-pre.0" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97f339a9801c7f70295732a5cf822346f194202cea6425fc2fc14a5af679b004" +checksum = "2973d210e74ebd81adc50828fbbb284ab6aaa099308097c42c47dc63cf3420fc" dependencies = [ "corez", "document-features", @@ -6306,9 +6243,9 @@ dependencies = [ [[package]] name = "zcash_transparent" -version = "0.9.0-pre.0" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e941230f67056aad41c8fa9b926f9cc4d9d1a321f32e95c39c0b0e38e85f79b" +checksum = "72e0f8c142bed366ed5886dcfa8772ec90b5420cc127e6cdb76b6744c53a287b" dependencies = [ "bip32", "bs58", @@ -6316,7 +6253,7 @@ dependencies = [ "document-features", "getset", "hex", - "nonempty 0.11.0", + "nonempty", "ripemd 0.1.3", "secp256k1", "sha2 0.10.9", @@ -6331,9 +6268,9 @@ dependencies = [ [[package]] name = "zebra-chain" -version = "11.0.0" +version = "11.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "600a4c35ee81350384226f8178fab2b088e38d5e0a9f6cb0b1051caba30702d7" +checksum = "014adde138ea00b1cf6c873aaf56aacb5aa0d55b8e5451a3e3b1b06c461bd107" dependencies = [ "bech32", "bitflags 2.13.0", @@ -6400,9 +6337,9 @@ dependencies = [ [[package]] name = "zebra-consensus" -version = "10.0.0" +version = "11.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0aa2b2678c4ce6d18172bad73a9b72b24a4b3a65af4c2a5e9120a1470c3fc402" +checksum = "4481ced3223226db18e33b9fb2e8eab7f7cefeae1681f4e8d8d7e3c2285a5c4d" dependencies = [ "bellman", "blake2b_simd", @@ -6443,9 +6380,9 @@ dependencies = [ [[package]] name = "zebra-network" -version = "10.0.0" +version = "10.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36b1e6ca4687bc8d128553ca04bb82cc38701c4dac7107f3e56717b39d4b18da" +checksum = "9a59dc9e9ac723c4c01d985a5ecf3ddd85c0f1681574457d52173c43263905f2" dependencies = [ "bitflags 2.13.0", "byteorder", @@ -6481,9 +6418,9 @@ dependencies = [ [[package]] name = "zebra-node-services" -version = "9.0.0" +version = "9.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4446db9b576c0de14ff91c7fd7b38a59d340c1969a1604787185a52b8b7f53a6" +checksum = "006992c62b9c4742a13a306f2e7443646fa0b5b884aac63a6fba5b0be273f864" dependencies = [ "color-eyre", "jsonrpsee-types", @@ -6497,9 +6434,9 @@ dependencies = [ [[package]] name = "zebra-rpc" -version = "11.0.0" +version = "11.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb180542f1ffc0f66c20e1cd66c7741662053ccda588e3847a8ed2df8c83dae7" +checksum = "4d071b177740cdc1107f90dd50fac67e1b440a84c8509ae1529b3609fb6f801d" dependencies = [ "base64", "chrono", @@ -6558,9 +6495,9 @@ dependencies = [ [[package]] name = "zebra-script" -version = "10.0.0" +version = "10.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8648c201e3dadb1c95022eb7605d528de4bf94b6c0cebf76537805849aee76c5" +checksum = "f7c967badb1bb99573dba430948c98fbca940daf23915f2e9736e1a1d2a166f3" dependencies = [ "libzcash_script", "rand 0.8.6", @@ -6572,9 +6509,9 @@ dependencies = [ [[package]] name = "zebra-state" -version = "10.0.0" +version = "10.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b95b81181253a46885e3254cc5a20e4d82b5258116c9e43db5c8e9660c6b973" +checksum = "39295d344c1cc56ed725646cfc7bb7e16f8edae7518ea5a9e599665f979e9cf6" dependencies = [ "bincode", "chrono", @@ -6734,12 +6671,12 @@ dependencies = [ [[package]] name = "zingo-consensus" version = "0.1.0" -source = "git+https://github.com/zingolabs/infrastructure.git?rev=0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646#0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646" +source = "git+https://github.com/zingolabs/infrastructure.git?rev=85502cd0b5faf9308e590865dae51bf31fdd342e#85502cd0b5faf9308e590865dae51bf31fdd342e" [[package]] name = "zingo_test_vectors" version = "0.0.1" -source = "git+https://github.com/zingolabs/infrastructure.git?rev=0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646#0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646" +source = "git+https://github.com/zingolabs/infrastructure.git?rev=85502cd0b5faf9308e590865dae51bf31fdd342e#85502cd0b5faf9308e590865dae51bf31fdd342e" [[package]] name = "zip32" diff --git a/Cargo.toml b/Cargo.toml index 0bf07dc94..e13a29ac6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -40,17 +40,16 @@ license = "Apache-2.0" # Librustzcash incrementalmerkletree = "0.8" -zcash_address = "0.13.0-pre.0" -zcash_keys = "0.15.0-pre.0" -zcash_protocol = "0.10.0-pre.0" -zcash_primitives = "0.29.0-pre.0" -zcash_transparent = "0.9.0-pre.0" -zcash_client_backend = "0.24.0-pre.0" +zcash_address = "0.13" +zcash_keys = "0.15" +zcash_protocol = "0.10" +zcash_primitives = "0.29" +zcash_transparent = "0.9" # Zebra -zebra-chain = "11.0" -zebra-state = "10.0" -zebra-rpc = "11.0" +zebra-chain = "11.1" +zebra-state = "10.1" +zebra-rpc = "11.1" # Runtime tokio = { version = "1.38", features = ["full"] } @@ -77,7 +76,7 @@ http = "1.1" url = "2.5" reqwest = { version = "0.13", default-features = false, features = [ "cookies", - "rustls", + "rustls-no-provider", "webpki-roots", ] } tower = { version = "0.4", features = ["buffer", "util"] } @@ -115,7 +114,6 @@ blake2 = "0.10" hex = "0.4" toml = "1.1" primitive-types = "0.14" -bs58 = "0.5" bitflags = "2.9" # Test @@ -138,15 +136,16 @@ zaino-testutils = { path = "live-tests/zaino-testutils" } # Zingo-infra (validator launcher driving the live tests; not a prod dep). # Pinned to an exact commit past zcash_local_net_v0.7.0: the v0.7.0 launcher's # zebrad config writer silently drops NU6.3 activation heights -# (https://github.com/zingolabs/zaino/issues/1368); this rev carries the -# "NU6.3" TOML plumbing. Restore a release tag once one is cut upstream. -zcash_local_net = { git = "https://github.com/zingolabs/infrastructure.git", rev = "0dc4a51f7d9666a3df7eaf28ad4f95cacb86d646" } +# (https://github.com/zingolabs/zaino/issues/1368). This rev additionally +# makes the Validator the sole compile-time source of activation-height +# truth (infrastructure ADR 0003, PR #278): wallet clients derive their +# heights via `WalletNetwork::from_validator`, and `ZainodConfig` carries a +# payload-free network kind. Restore a release tag once one is cut upstream. +zcash_local_net = { git = "https://github.com/zingolabs/infrastructure.git", rev = "85502cd0b5faf9308e590865dae51bf31fdd342e" } wire_serialized_transaction_test_data = "1.0.0" config = { version = "0.15", default-features = false, features = ["toml"] } -nonempty = "0.12.0" proptest = "~1.11" -zip32 = "0.2.1" anyhow = "1.0" arc-swap = "1.7.1" derive_more = "2.0.1" diff --git a/Makefile.toml b/Makefile.toml index f38a60907..2bcaec624 100644 --- a/Makefile.toml +++ b/Makefile.toml @@ -93,7 +93,7 @@ script.main = 'source "./tools/scripts/push-image.sh"' [tasks.container-test] clear = true -description = "Engine: run nextest in the local CI container, forwarding args (zcashd_support OFF by default; --with-zcashd adds --features zcashd_support). Devs use `makers test [package|e2e|clientless|live|all]`; live-clientless / live-e2e and the `test` front door call this." +description = "Engine: run nextest in the local CI container, forwarding args (zcashd_support OFF by default; --with-zcashd adds --features zcashd_support). Devs use `makers test [packages|e2e|clientless|live|all]`; live-clientless / live-e2e and the `test` front door call this." # This task runs tests inside the container built from live-tests/test_environment. # The entrypoint.sh script in the container sets up test binaries (zcashd, zebrad, zcash-cli) # by creating symlinks from /home/container_user/artifacts to the expected live-tests/test_binaries/bins location. @@ -132,16 +132,16 @@ script.main = 'source "./tools/scripts/container-nextest-workspace.sh"' # =================================================================== # Developer-facing test front door: a single `makers test [SET]` task. # -# makers test packaged (the default) -# makers test package packages/* production-crate tests (no validator) +# makers test packages (the default) +# makers test packages packages/* production-crate tests (no validator) # makers test e2e the e2e live partition # makers test clientless the clientless live partition # makers test live both live partitions + combined summary -# makers test all package then live (everything) +# makers test all packages then live (everything) # -# SET names mirror the crate/dir names (`e2e`, `clientless`); `package`, -# `live`, and `all` are the groupings. `live` = all non-package tests; `all` -# = `package` + `live` (see live-tests/CONTEXT.md). Each set delegates to an +# SET names mirror the crate/dir names (`e2e`, `clientless`); `packages`, +# `live`, and `all` are the groupings. `live` = all non-packages tests; `all` +# = `packages` + `live` (see live-tests/CONTEXT.md). Each set delegates to an # existing engine rather than re-implementing a run. `--with-zcashd` (handled # by the engines) adds the zcashd-backed tests to any set. # =================================================================== @@ -150,42 +150,56 @@ script.main = 'source "./tools/scripts/container-nextest-workspace.sh"' # clear = true fully replaces cargo-make's built-in `test` task (which runs # `cargo test`); without it our `script` would merge with the builtin `command`. clear = true -description = "Run a test set: `makers test [package|e2e|clientless|live|all]` (default: package). Pass --with-zcashd for the zcashd-backed tests." +description = "Run a test set: `makers test [packages|e2e|clientless|live|all|ironwood]` (default: packages). Pass --with-zcashd for the zcashd-backed tests." script_runner = "bash" script = ''' set -uo pipefail usage() { - echo "usage: makers test [package|e2e|clientless|live|all] [-- nextest args]" >&2 - echo " package (default) packages/* tests, no live validator" >&2 + echo "usage: makers test [packages|e2e|clientless|live|all|ironwood] [-- nextest args]" >&2 + echo " packages (default) packages/* tests, no live validator" >&2 echo " e2e | clientless that single live partition" >&2 echo " live both live partitions + combined summary" >&2 - echo " all package then live (everything)" >&2 + echo " all packages then live (everything)" >&2 + echo " ironwood the ironwood tests of every set, packages then clientless then e2e" >&2 } # A non-flag first arg selects the set and must be a known name; a flag first -# arg (or none) keeps the default `package` and is forwarded to nextest. So -# `makers test --with-zcashd` runs package with the flag, while a mistyped set +# arg (or none) keeps the default `packages` and is forwarded to nextest. So +# `makers test --with-zcashd` runs packages with the flag, while a mistyped set # name errors instead of silently running the default. -set="package" +set="packages" if [ "$#" -gt 0 ] && [ "${1#-}" = "$1" ]; then case "$1" in - package|e2e|clientless|live|all) set="$1"; shift ;; + packages|e2e|clientless|live|all|ironwood) set="$1"; shift ;; *) echo "makers test: unknown set '$1'" >&2; usage; exit 2 ;; esac fi case "$set" in - package) exec makers container-test "$@" ;; + packages) exec makers container-test "$@" ;; e2e) exec makers live-e2e "$@" ;; clientless) exec makers live-clientless "$@" ;; live) exec cargo run -q --manifest-path tools/test-runner/Cargo.toml --bin live-summary -- "$@" ;; all) - # Run both even if the first fails, then fail if either did, so the - # omnibus reflects the whole suite. + # The summary runner's --all mode runs the packages set and both live + # partitions (each set runs even when an earlier one fails, and the exit + # code reflects any failure), so the combined table covers everything. + exec cargo run -q --manifest-path tools/test-runner/Cargo.toml --bin live-summary -- --all "$@" ;; + ironwood) + # The one definition of the ironwood selection: the two dedicated test + # binaries, plus every test whose name carries the pool's name (the + # CONTEXT.md era-naming convention keeps new ironwood tests inside this + # net). The same expression serves every engine — a selector naming a + # binary a partition lacks simply matches nothing there. The engines are + # invoked directly rather than through the `live` summary runner, which + # forwards no nextest args. Run every set even if an earlier one fails, + # as in `all`. + ironwood_filter='binary(ironwood_activation) | binary(the_pub_testnet_ironwood_boundary) | test(ironwood)' rc=0 - makers container-test "$@" || rc=1 - cargo run -q --manifest-path tools/test-runner/Cargo.toml --bin live-summary -- "$@" || rc=1 + makers container-test -E "$ironwood_filter" "$@" || rc=1 + makers live-clientless -E "$ironwood_filter" "$@" || rc=1 + makers live-e2e -E "$ironwood_filter" "$@" || rc=1 exit "$rc" ;; esac ''' diff --git a/README.md b/README.md index a72a9a495..0b8ca92f4 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ The test suites run inside a **podman** container via `makers` (cargo-make): ```sh makers test # packages/* tests that need no live validator (default) makers test live # both live partitions (clientless + e2e) + combined summary -makers test all # everything: package then live +makers test all # everything: packages then live ``` zcashd-backed tests are **off by default**; add `--with-zcashd` to include them diff --git a/docs/adr/0004-rename-integration-partition-to-clientless.md b/docs/adr/0004-rename-integration-partition-to-clientless.md index c98de87d4..cb80fbfa7 100644 --- a/docs/adr/0004-rename-integration-partition-to-clientless.md +++ b/docs/adr/0004-rename-integration-partition-to-clientless.md @@ -50,3 +50,6 @@ unchanged. "clientless" is no longer avoided. - ADR-0003's naming bullet and its three-front-door task surface are superseded; its two-crate split and the `live` umbrella decision still stand. + +> **Later note:** the `package` set was renamed `packages` (plural, matching the +> `packages/` dir); `makers test package` now errors. See `live-tests/CONTEXT.md`. diff --git a/docs/adr/0006-aws-lc-rs-preferred-crypto-provider.md b/docs/adr/0006-aws-lc-rs-preferred-crypto-provider.md new file mode 100644 index 000000000..13306503c --- /dev/null +++ b/docs/adr/0006-aws-lc-rs-preferred-crypto-provider.md @@ -0,0 +1,90 @@ +# aws-lc-rs is the preferred CryptoProvider; classical key exchange is deprecating + +## Status + +accepted (reverses the ring choice made on zingolabs/zaino#1366 before it +merged) + +## Context and decision + +Zaino installs a rustls `CryptoProvider` as the process-wide default at +its two TLS boundaries (`zaino-serve`'s gRPC server, `zaino-fetch`'s +jsonrpsee connector) via `zaino_common::crypto::ensure_default_crypto_provider` +(zingolabs/zaino#1360). The provider was initially pinned to **ring** for +two reasons: zebra used ring (shared attack surface across the deployed +stack), and ring avoided the `aws-lc-sys` + `cmake` build dependencies. + +Daira Emma argued for **aws-lc-rs** instead: post-quantum TLS key +exchange in the rustls ecosystem requires the aws-lc-rs provider — the +ring provider has no ML-KEM support at all — and the provider +architecture makes the swap cheap +(, +). + +We decide: + +1. **aws-lc-rs is zaino's preferred CryptoProvider** — a *preference, + not a mandate*: installation stays first-install-wins, so an embedder + (e.g. zallet) that installs a provider before zaino keeps its choice. + Enforcement lives in the feature graph (`rustls` `aws_lc_rs` feature, + tonic `tls-aws-lc`), not at runtime. +2. **rustls's `prefer-post-quantum` feature is enabled**, reordering the + provider's default key-exchange groups so the hybrid X25519MLKEM768 + leads (same membership; in rustls 0.23 the hybrid is in the default + set either way). Verified effect in rustls 0.23.41: on zaino's + *outbound* (client-role) TLS the hybrid share is offered upfront, and + rustls adds the hybrid's X25519 component as a free second key share, + so classical-only servers negotiate without a HelloRetryRequest — the + cost is ~1.2 KB of ClientHello, not a round trip. On the *server* + side rustls follows the client's group preference, so inbound + negotiation outcomes are unchanged by the feature; hybrid uptake on + inbound connections is entirely client-driven, which is why the + deprecation below is gated on client stacks. +3. **Classical key exchange (X25519, SECP256R1, SECP384R1) remains + accepted but is deprecated.** Refusing it today would break both + flagship wallets: zingolib's TLS stack is feature-pinned to ring (no + ML-KEM possible), and ZODL rides mobile platform TLS where hybrid + support is still rolling out. The refusal precondition — zingolib on + aws-lc-rs with hybrid preferred, and ZODL's Android/iOS platform + stacks negotiating X25519MLKEM768 — is tracked in a dedicated issue; + deprecation is documented in the changelog rather than signalled at + runtime. +4. **Ring stays in `Cargo.lock`, tolerated and fenced, until zebra drops + it.** `zebra-rpc` hard-enables `zebra-node-services/rpc-client`, + whose reqwest 0.12 `rustls-tls` feature compiles ring in — reqwest + 0.12 offers no aws-lc alternative, only `*-no-provider` variants. This + is dormant code, not a live handshake path: reqwest consults the + process-default provider first and falls back to ring only when none + is installed, so both zaino (which installs aws-lc-rs early) and + embedders (whose earlier install wins) preempt it. A CI lint pins + `cargo tree --invert ring` to exactly this one known path and fails on + drift in *either* direction: a new ring consumer is a regression to + investigate; the path vanishing means zebra dropped ring, the + allowlist gets deleted, and the guard tightens to "ring absent". + +## Considered options + +- **Stay on ring, aligned with zebra.** Rejected: forecloses post-quantum + key exchange entirely; the harvest-now-decrypt-later argument applies + to wallet↔zaino TLS traffic today. +- **Refuse classical key exchange (hybrid-only).** Rejected for now: + hard-locks out every wallet whose TLS stack lacks X25519MLKEM768, + which today is all of them. Revisit when the tracked precondition is + met. +- **Fork-pin zebra to purge ring from the lock immediately.** Rejected: + the fork tax (pin churn here and in the wallet-tests workspace) buys + only the removal of dormant code; downstream is already protected by + provider preemption. Instead we upstream the fix to zebra — + `zebra-node-services` switches reqwest to a `*-no-provider` feature + and zebrad installs its own default provider (the same pattern as + zaino's #1360 fix) — and pick it up at the next zebra bump. + +## Consequences + +- `aws-lc-sys` and `cmake` return to the build graph (both build images + already install cmake). +- Zaino diverges from zebra's provider until zebra moves; the CI lint + makes the eventual convergence loud instead of silent. +- TLS 1.2 remains available only as long as classical key exchange + does; the refusal that ends the classical deprecation also implies + TLS 1.3-only. diff --git a/docs/notes/ironwood-activation-plan.md b/docs/notes/ironwood-activation-plan.md new file mode 100644 index 000000000..abb2d3b97 --- /dev/null +++ b/docs/notes/ironwood-activation-plan.md @@ -0,0 +1,227 @@ +# Ironwood activation: test design and heights source-of-truth roadmap + +State capture, 2026-07-06. Design session artifacts live in this tree +(uncommitted at time of writing); coordination artifacts live in +`zingolabs/infrastructure` (PR #278 worktree). This note is the durable +record of the decisions and the facts they rest on. + +## Domain facts, with sources + +- **Ironwood is a new shielded pool** activated by NU6.3 + (). It shares the Action-circuit shape + with Orchard but is a distinct pool. +- **Public testnet activated NU6.3 at height 4,134,000, ~2026-07-04.** + Defined identically in zebra-chain 11.0.0 + (`parameters/constants.rs::activation_heights::testnet::NU6_3`) and + zcash_protocol 0.10.0-pre.0 (`consensus.rs`), consensus branch ID + `0x37a5165b` in both. Wallet stack and validator stack therefore flip eras + at the same height with no drift. **Mainnet has no NU6.3 height** in those + pins. (Gotcha: public explorers can mislabel testnet/mainnet heights — + verify against a synced node.) +- **ZIP 318 (zcash/zips PR #1317), "Orchard to Ironwood migration"**: wallet + best-practices for a two-phase scheduled migration. Normative anchor: "The + user MUST be able to migrate Orchard-pool funds to the Ironwood pool, and + MUST be informed that doing so is necessary to retain access to those + funds." The wallet-visible migration transaction is an Orchard spend with + Ironwood outputs. +- **Cross-address restriction** + (): + post-NU6.3 the Orchard Action circuit requires "(g_d, pk_d) of the output + note must equal (g_d, pk_d) of the spent note" — every Orchard action is + either change to the spent note's own address or a withdrawal (positive + value balance). Cross-address Orchard transfers are circuit-impossible, + including between two addresses of one wallet. A companion + transaction-level rule forbids new value entering the pool. + - Do **not** describe Orchard as "exit-only": same-receiver change still + lands in the pool, so the Orchard note-commitment tree keeps growing + after activation. There is **no frozen-finalRoot predicate**; the + correct chain-walk predicate is **Orchard pool value non-increasing from + the boundary** (observable via `valuePools`). +- **No Orchard TAZ is obtainable** (and the testnet pre-epoch is closed): + post-activation nothing new can enter Orchard, so pre-activation Orchard + notes exist only in wallets that held them before the flip. Zaino holds + none. Consequence: the migration cell is exercisable **only on hermetic + regtest transition chains** — permanently. + +## Language decisions (canonical in CONTEXT.md) + +- **Testnet** names only the public test network. Every hermetic local net + is a **regtest net**, whatever network-kind flag it runs under. +- **Pool names pair with pool names** (Orchard ↔ Ironwood); **NU6.3** names + only the upgrade itself (activation heights, branch ID, consensus rules). + Applied renames: `NU6_3_ACTIVE_ACTIVATION_HEIGHTS` → + `IRONWOOD_ONLY_ACTIVATION_HEIGHTS` (zaino-testutils), + `NU6_3_ACTIVE_HEIGHTS` → `IRONWOOD_ONLY_HEIGHTS` + (`proptest_blockgen.rs`). `NU6_3_TRANSITION_BOUNDARY` keeps the upgrade + vocabulary: an activation height is the upgrade's concept. + +## Design model: predicates × regimes + +A **regime** is a (network, era) pair; a **predicate** is a +wallet-observable claim whose truth is fixed per regime. The suite is the +truth table; tests instantiate predicates in every reachable regime. + +| predicate | Orchard era | Ironwood era | +|---|---|---| +| unified-address receipt lands in Orchard | true | false | +| unified-address receipt lands in Ironwood | false (pool inactive) | true | +| shielded-receiver coinbase pays the era pool | Orchard | Ironwood | +| Orchard note spends into an Ironwood receipt (ZIP 318 migration) | n/a | true | +| `shield` deposits into the era pool | Orchard | Ironwood | +| anything can land in Orchard from outside | true | false (no-new-value rule) | +| Orchard pool value non-increasing | n/a | true (cross-address restriction) | + +Regime venues: + +- **Regtest, Ironwood-only** (`IRONWOOD_ONLY_ACTIVATION_HEIGHTS`, canonical + NU6.3-at-2): the existing devtool wallet tests (`send_to_ironwood`, + `shield_for_validator`, `receives_mining_reward`, `send_to_all` in + `live-tests/e2e/tests/devtool.rs`). +- **Regtest, Orchard-only** (`ORCHARD_ONLY_ACTIVATION_HEIGHTS`): clientless + / wire mining fixtures only (devtool wallet cannot run there today). +- **Regtest, transition** (`ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS`, + NU6.3 at `NU6_3_TRANSITION_BOUNDARY` = 6): coinbase-routing and era + composition covered clientless + (`clientless/tests/compact_block_consistency.rs`) and over the wire + (`e2e/tests/compact_block_wire.rs`); **wallet cells live in + `e2e/tests/ironwood_activation.rs`** (see below). +- **Public testnet**: observational boundary walk across height 4,134,000 — + durable forever, historical blocks don't expire; third-party ZIP 318 + migrations supply the Orchard-spend traffic zaino must index but cannot + author. Active cells post-flip only, Orchard-free: faucet TAZ → + transparent receipt → `shield` → Ironwood send/receive. Env-gated manual + runs; record txids/heights in a dated evidence log + (`docs/notes/ironwood-activation-evidence.md`, created on first run). + `ZEBRAD_TESTNET_CACHE_DIR` (zaino-testutils) anticipates the cached sync. + +## Landed artifacts (this tree) + +- **`live-tests/e2e/tests/ironwood_activation.rs`** — the transition-regime + wallet cells, full real bodies, gated + `#[should_panic(expected = "UnsupportedActivationHeights")]`: + - `unified_receipt_lands_in_orchard_before_boundary` + - `orchard_note_spends_to_ironwood_across_boundary` — the migration cell; + recipient unified address generated *after* activation; asserts the + faucet's Orchard balance strictly shrinks, guarding against devtool note + selection dodging the migration by spending the boundary-height Ironwood + coinbase instead (sound under the cross-address restriction: change + returns only to the spent note's receiver, so a genuine Orchard spend + nets sent-plus-fee out of the pool). + - Verified: both pass as should_panic in ~5 s each + (`cargo nextest run -p e2e -E 'binary(ironwood_activation)'`). + - Registered on one backend (FetchService) while known-red: the panic + fires before the backend is exercised; mirror the fetch/state matrix + when the cells go live. +- **`e2e::devtool::build_clients_at(port, &heights)`** — devtool wallets on + caller-chosen regtest heights; `build_clients` delegates with the + canonical set. +- CONTEXT.md glossary entries; the renames above. + +## The blocker, and the infra handoff + +`zcash_local_net`'s devtool client rejects every regtest heights set except +its canonical NU6.3-at-2 one at wallet launch +(`ClientError::UnsupportedActivationHeights`; the guard exists because +heights drift between wallet and validator produces "incorrect consensus +branch id" — see zaino#1368). The guard predates the current devtool, which +reads heights at `init` from the `--activation-heights` TOML the client +already writes. + +Handoff: **`~/src/zingolabs/infras/dev/zaino-ironwood-activation-infra-spec.md`** +(for zingolabs/infrastructure PR #278, branch `bump_to_NU6.3`; zaino pins +rev `0dc4a51f...`). Asks: lift the equality guard (canonical set stays the +default), retire the error variant (zaino's known-red tests key on its name +so the pin bump flips them loudly), keep the caller-responsibility doc +contract, plus an infra-side integration test proving era-correct scanning +and branch-ID selection from file heights. Known unknown flagged: devtool +note-selection policy across pools. + +Scope addition (confirmed): infra also narrows `ZainodConfig.network` from +`NetworkType` to a payload-free `NetworkKind` in the same release — the +heights payload there is dead code (discarded at +`network_type_to_string()`; the Indexer never received heights that way). +Validator configs keep full `NetworkType`: they configure the source of +truth. + +## Roadmap: source of truth first + +Current directive (supersedes "unblock the suite next"): **first** make the +Validator the single source of truth for activation heights, with +regression tests that keep that source unique, and fix zaino#1076. + +Invariant: *the Indexer learns activation heights from the Validator at +startup and accepts them from nowhere else; configs carry network kind +only.* + +Audit facts grounding the implementation: + +- zainod's TOML is already kind-only: `NetworkSerde` serializes + "Mainnet"/"Testnet"/"Regtest", and deserializing "Regtest" injects the + compiled-in `ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS` + (`zaino-common/src/config/network.rs`). The real second channel is the + library seam: `Network::Regtest(ActivationHeights)` constructed + programmatically (e.g. by `TestManager`) and consumed via + `config.get_network()` → `to_zebra_network()`. +- The validator channel already exists in zaino's types: + `GetBlockchainInfoResponse.upgrades` is parsed (zaino-fetch + `jsonrpsee/response.rs`) and consumed (`backends/fetch.rs`, + `latest_network_upgrade`). The work is deriving the runtime + `zebra_chain::parameters::Network` from that response at startup and + narrowing the config type to kind-only. +- Regression tests that maintain uniqueness: launch the validator at + non-canonical heights (the transition fixture) with a kind-only-configured + zainod and assert era-correct serving (fails if anyone re-couples zainod + to a compiled-in set); the narrowed config type itself is the compile-time + regression test; a serde test pins the TOML shape. +- Payoff: the `InvalidData("Block commitment could not be computed")` + config-drift failure class becomes structurally impossible — the same + invariant the infra guard lift expresses on the wallet side (wallets get + heights from a file matched to the validator; the indexer gets them from + the validator directly). +- **#1076 residue** (issue text predates the all-at-2 canonical set, which + already replaced the NU6.1-at-1000 sentinel): wire + `lockbox_disbursements` into zebrad-launching fixtures (plumbing landed + at pinned rev, infra#243); add one NU6.1 boundary-crossing test (same + transition-fixture pattern as the Ironwood suite); watch for + block-commitment regressions. + +## Sequencing + +1. Zaino: heights source-of-truth implementation + uniqueness regression + tests (**landed in this tree**: `regtest_network_from_upgrades` + + spawn-time adoption in the zaino-state backends, de-mirrored testutils, + unit tests beside the mapping, live regressions in + `clientless/tests/validator_heights.rs`); remaining from this item: the + config-type split (kind-only config type, heights only post-handshake) + and the #1076 residue (lockbox fixtures, NU6.1 boundary test). +2. Infra: `NetworkKind` narrowing plus the strict form of the wallet-side + change on `bump_to_NU6.3`. The standing directive (2026-07-06) is that + the Validator is the *only* source of truth, enforced at compile time: + the harness derives the wallet's `activation-heights.toml` from + `Validator::get_activation_heights()`, and no client API accepts + caller-supplied heights (the spec's original "lift the guard, accept + caller heights" shape is superseded — a caller heights parameter is + itself a second source). +3. Zaino pin bump: adapt to the narrowed `ZainodConfig`; the two + `ironwood_activation` tests flip red (their expected + `UnsupportedActivationHeights` panic disappears along with the error + variant); replace `e2e::devtool::build_clients_at(port, &heights)` — + which exists only to express the caller-supplied shape — with a + derivation-based builder, so a fixture's heights are typed in exactly + one place: the zebrad launch config. First live runs settle devtool + note selection and the compact form of Orchard-spend data. +4. Deferred cells: testnet observational walk + post-flip Ironwood active + cells (env-gated, evidence log); chain-walk cells for the cross-address + restriction (Orchard pool-value monotonicity, same-receiver-change-only + commitments); the Orchard spend-window/freeze sub-regime waits on a + consensus source. + +## Unrelated finding from the same session + +The red `Integration tests / RPC tests shard-2/3` statuses on zaino commits +(zcash/integration-tests runs) are an upstream zallet break, not zaino's: +every run since 2026-07-03 ~23:25 UTC fails `zcashd_key_import{,_db}.py` in +zallet's `migrate-zcashd-wallet` with `UNIQUE constraint failed: +addresses.cached_transparent_receiver_address`, immediately after zallet +main's librustzcash/NU6.3 dependency bumps. No zallet issue existed for it +as of 2026-07-06. diff --git a/docs/testing.md b/docs/testing.md index f5c6d8f10..9bfd24575 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -26,14 +26,14 @@ The `makers` tasks below build and run the test suites inside a **podman** container, so you don't need the validator binaries on your host `$PATH`. The container image is built or pulled automatically on first run. -The single front door is `makers test [SET]`, where `SET` defaults to `package`: +The single front door is `makers test [SET]`, where `SET` defaults to `packages`: -- `makers test` (or `makers test package`) — the **package** set: the +- `makers test` (or `makers test packages`) — the **packages** set: the `packages/*` production-crate tests that need no live validator. - `makers test e2e` / `makers test clientless` — one live partition. - `makers test live` — both live partitions (`clientless` then `e2e`) against a live validator, with a combined pass/fail summary. -- `makers test all` — the whole suite: package then live. +- `makers test all` — the whole suite: packages then live. (`container-test`, `live-clientless`, and `live-e2e` are the internal engines the `test` front door delegates to; invoke them directly only when you need to diff --git a/live-tests/CONTEXT.md b/live-tests/CONTEXT.md index d00e0f064..1bef4076e 100644 --- a/live-tests/CONTEXT.md +++ b/live-tests/CONTEXT.md @@ -40,15 +40,17 @@ owns shared fixtures and the feature-forwarding surface (`zcashd_support`, `transparent_address_history_experimental`). _Avoid_: test helpers, testlib. -**package (test set)**: +**packages (test set)**: The tests of the `packages/*` production crates — the workspace `default-members`, exercised by bare `cargo nextest run` and by `makers test` -(the default with no argument, equivalently `makers test package`). Named for +(the default with no argument, equivalently `makers test packages`). Named for where the crates live (`packages/`); the functional reason they run apart from the live partitions is that they need **no** live validator. They are *not* network-free, though: e.g. zaino-serve's gRPC `spawn` regression test binds a loopback socket and stands up a tonic server. The absent ingredient is a validator, not the network. -_Avoid_: offline (overclaims "no network" — these tests bind loopback sockets); -unit test (the set includes crate-level integration tests); container test -(names the run mechanism, which the live suite shares). +_Avoid_: package (singular — the set is plural, matching the `packages/` dir; +`makers test package` now errors with `unknown set`); offline (overclaims "no +network" — these tests bind loopback sockets); unit test (the set includes +crate-level integration tests); container test (names the run mechanism, which +the live suite shares). diff --git a/live-tests/clientless/Cargo.toml b/live-tests/clientless/Cargo.toml index ba912e690..c8945a81a 100644 --- a/live-tests/clientless/Cargo.toml +++ b/live-tests/clientless/Cargo.toml @@ -36,7 +36,6 @@ zaino-fetch = { workspace = true } zaino-testutils = { workspace = true } # The lib's z_validate helper calls `ZcashIndexer::z_validate_address`. zaino-state = { workspace = true, features = ["test_dependencies"] } -tracing.workspace = true [dev-dependencies] anyhow = { workspace = true } @@ -46,6 +45,9 @@ anyhow = { workspace = true } zaino-state = { workspace = true, features = ["fast-test-seam"] } # Test utility +# Not imported by any test source; present so the +# transparent_address_history_experimental feature can forward to +# zainod/transparent_address_history_experimental in the spawned daemon. zainod = { workspace = true } zaino-testutils = { workspace = true } zaino-fetch = { workspace = true } @@ -55,13 +57,8 @@ wire_serialized_transaction_test_data = { workspace = true } zebra-chain = { workspace = true } zebra-state = { workspace = true } zebra-rpc = { workspace = true } -zip32 = { workspace = true } -corez = { workspace = true } -serde_json = { workspace = true } futures = { workspace = true } -tempfile = { workspace = true } -tower = { workspace = true } hex = { workspace = true } # Runtime diff --git a/live-tests/clientless/src/lib.rs b/live-tests/clientless/src/lib.rs index 9e7314757..12d93f9be 100644 --- a/live-tests/clientless/src/lib.rs +++ b/live-tests/clientless/src/lib.rs @@ -83,7 +83,17 @@ pub mod rpc { // assert_eq!(fs_sprout, expected_sprout); - // Sapling (differs by validator) + // Sapling + let expected_sapling = ValidZValidateAddress::sapling( + VALID_SAPLING_ADDRESS.to_string(), + Some(VALID_DIVERSIFIER.to_string()), + Some(VALID_DIVERSIFIED_TRANSMISSION_KEY.to_string()), + ); + assert_known_valid_eq( + rpc_call(VALID_SAPLING_ADDRESS.to_string()).await, + expected_sapling, + "Sapling", + ); // Unified (differs by validator) let expected_unified = @@ -101,70 +111,15 @@ pub mod rpc { assert_eq!(all_zeroes, ZValidateAddressResponse::invalid()); } - pub async fn run_z_validate_sapling(rpc_call: &F) - where - F: Fn(String) -> Fut, - Fut: Future, - { - let expected_sapling = ValidZValidateAddress::sapling( - VALID_SAPLING_ADDRESS.to_string(), - Some(VALID_DIVERSIFIER.to_string()), - Some(VALID_DIVERSIFIED_TRANSMISSION_KEY.to_string()), - ); - assert_known_valid_eq( - rpc_call(VALID_SAPLING_ADDRESS.to_string()).await, - expected_sapling, - "Sapling", - ); - } - - /// zebrad's JSON-RPC passthrough (via FetchService) omits `diversifier` - /// and `diversifiedtransmissionkey` from the Sapling response. This is - /// the safer behavior: address component extraction should happen - /// client-side, not by delegating to a remote actor. - /// - /// See [`DEPRECATION_NOTICE`](zaino_fetch::jsonrpsee::response::z_validate_address::DEPRECATION_NOTICE). - pub async fn run_z_validate_sapling_zebrad_passthrough_fetchservice(rpc_call: &F) - where - F: Fn(String) -> Fut, - Fut: Future, - { - let expected_sapling = ValidZValidateAddress::sapling( - VALID_SAPLING_ADDRESS.to_string(), - None::, - None::, - ); - assert_known_valid_eq( - rpc_call(VALID_SAPLING_ADDRESS.to_string()).await, - expected_sapling, - "Sapling (zebrad passthrough via FetchService — keys omitted)", - ); - } - - /// Which sapling suite to run after the shared suite. - pub enum SaplingSuite { - /// Full response — diversifier and diversifiedtransmissionkey present. - Standard, - /// zebrad's JSON-RPC passthrough (via FetchService) omits those keys. - ZebradPassthroughFetchService, - } - /// Build the `z_validate_address` rpc-call closure from `subscriber` and - /// run the shared validation suite plus the chosen sapling suite. Factors - /// the identical closure + suite-call preamble shared by the four - /// `z_validate_address` tests (fetch_service zcashd/zebrad, state_service, - /// json_server). + /// run the shared validation suite. Factors the identical closure + + /// suite-call preamble shared by the four `z_validate_address` tests + /// (fetch_service zcashd/zebrad, state_service, json_server). #[allow(deprecated)] - pub async fn run_z_validate_for(subscriber: &S, sapling: SaplingSuite) { + pub async fn run_z_validate_for(subscriber: &S) { let rpc_call = |addr: String| async move { subscriber.z_validate_address(addr).await.unwrap() }; run_z_validate_suite(&rpc_call).await; - match sapling { - SaplingSuite::Standard => run_z_validate_sapling(&rpc_call).await, - SaplingSuite::ZebradPassthroughFetchService => { - run_z_validate_sapling_zebrad_passthrough_fetchservice(&rpc_call).await - } - } } } } diff --git a/live-tests/clientless/tests/chain_cache.rs b/live-tests/clientless/tests/chain_cache.rs index a12cb7291..eb173f2e0 100644 --- a/live-tests/clientless/tests/chain_cache.rs +++ b/live-tests/clientless/tests/chain_cache.rs @@ -1,24 +1,20 @@ use zaino_common::network::ActivationHeights; use zaino_fetch::jsonrpsee::connector::JsonRpSeeConnector; -use zaino_state::{ZcashIndexer, ZcashService}; -use zaino_testutils::{TestManager, ValidatorExt, ValidatorKind}; -use zainodlib::error::IndexerError; +use zaino_testutils::{Direct, Rpc, TestManager, ValidatorExt, ValidatorKind}; #[allow(deprecated)] -async fn create_test_manager_and_connector( +async fn create_test_manager_and_connector( validator: &ValidatorKind, activation_heights: Option, chain_cache: Option, enable_zaino: bool, enable_clients: bool, -) -> (TestManager, JsonRpSeeConnector) +) -> (TestManager, JsonRpSeeConnector) where T: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let test_manager = TestManager::::launch( + let test_manager = TestManager::::launch( validator, None, activation_heights, @@ -46,8 +42,11 @@ mod chain_query_interface { source::ValidatorConnector, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, ShieldedPool, }, - test_dependencies::{chain_index::ChainIndex, ChainIndexConfig}, - FetchService, Height, StateService, StateServiceConfig, ZcashService, + test_dependencies::{ + chain_index::{ChainIndex, ChainIndexRpcExt}, + ChainIndexConfig, + }, + Height, NodeBackedIndexerService, NodeBackedIndexerServiceConfig, ZcashService, }; #[cfg(feature = "zcashd_support")] use zcash_local_net::validator::zcashd::Zcashd; @@ -63,26 +62,24 @@ mod chain_query_interface { use super::*; #[allow(deprecated)] - async fn create_test_manager_and_chain_index( + async fn create_test_manager_and_chain_index( validator: &ValidatorKind, chain_cache: Option, enable_zaino: bool, enable_clients: bool, ephemeral: bool, ) -> ( - TestManager, + TestManager, JsonRpSeeConnector, - Option, + Option, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, ) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (test_manager, json_service) = create_test_manager_and_connector::( + let (test_manager, json_service) = create_test_manager_and_connector::( validator, None, chain_cache.clone(), @@ -123,42 +120,43 @@ mod chain_query_interface { NetworkKind::Testnet => zebra_chain::parameters::Network::new_default_testnet(), NetworkKind::Mainnet => zebra_chain::parameters::Network::Mainnet, }; - // FIXME: when state service is integrated into chain index this initialization must change - let state_service = StateService::spawn(StateServiceConfig::new( - zebra_state::Config { - cache_dir: state_chain_cache_dir, - ephemeral: false, - delete_old_database: true, - debug_stop_at_height: None, - debug_validity_check_interval: None, - // todo: does this matter? - should_backup_non_finalized_state: true, - debug_skip_non_finalized_state_backup_task: false, - }, - test_manager.full_node_rpc_listen_address.to_string(), - test_manager.full_node_grpc_listen_address, - false, - None, - None, - None, - ServiceConfig::default(), - StorageConfig { - cache: CacheConfig::default(), - database: DatabaseConfig { - path: test_manager - .data_dir - .as_path() - .to_path_buf() - .join("state-service-zaino"), - ..Default::default() + // FIXME: when the direct connection is integrated into chain index this initialization must change + let state_service = + NodeBackedIndexerService::spawn(NodeBackedIndexerServiceConfig::new_direct( + zebra_state::Config { + cache_dir: state_chain_cache_dir, + ephemeral: false, + delete_old_database: true, + debug_stop_at_height: None, + debug_validity_check_interval: None, + // todo: does this matter? + should_backup_non_finalized_state: true, + debug_skip_non_finalized_state_backup_task: false, }, - }, - false, - network.into(), - None, - )) - .await - .unwrap(); + test_manager.full_node_rpc_listen_address.to_string(), + test_manager.full_node_grpc_listen_address, + false, + None, + None, + None, + ServiceConfig::default(), + StorageConfig { + cache: CacheConfig::default(), + database: DatabaseConfig { + path: test_manager + .data_dir + .as_path() + .to_path_buf() + .join("state-service-zaino"), + ..Default::default() + }, + }, + false, + network.into(), + None, + )) + .await + .unwrap(); let config = ChainIndexConfig { storage: StorageConfig { database: DatabaseConfig { @@ -173,11 +171,12 @@ mod chain_query_interface { }, ephemeral, db_version: 1, - network: zaino_common::Network::Regtest( - zaino_testutils::from_local_net_activation_heights( - &test_manager.local_net.get_activation_heights().await, - ), - ), + // This fixture derives its runtime network from the + // heights the harness launched the validator with. + network: zaino_testutils::from_local_net_activation_heights( + &test_manager.local_net.get_activation_heights().await, + ) + .to_regtest_network(), }; // **NOTE** The "fetch" backend is currently the backend used in the wild, and @@ -219,11 +218,12 @@ mod chain_query_interface { }, ephemeral, db_version: 1, - network: zaino_common::Network::Regtest( - zaino_testutils::from_local_net_activation_heights( - &test_manager.local_net.get_activation_heights().await, - ), - ), + // This fixture derives its runtime network from the + // heights the harness launched the validator with. + network: zaino_testutils::from_local_net_activation_heights( + &test_manager.local_net.get_activation_heights().await, + ) + .to_regtest_network(), }; let chain_index = NodeBackedChainIndex::new( ValidatorConnector::Fetch(json_service.clone()), @@ -242,25 +242,23 @@ mod chain_query_interface { // #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_block_range_zebrad() { - get_block_range::(&ValidatorKind::Zebrad).await + get_block_range::(&ValidatorKind::Zebrad).await } #[cfg(feature = "zcashd_support")] #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_block_range_zcashd() { - get_block_range::(&ValidatorKind::Zcashd).await + get_block_range::(&ValidatorKind::Zcashd).await } - async fn get_block_range(validator: &ValidatorKind) + async fn get_block_range(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (test_manager, _json_service, _option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, false) + create_test_manager_and_chain_index::(validator, None, false, false, false) .await; test_manager @@ -282,7 +280,7 @@ mod chain_query_interface { #[tokio::test(flavor = "multi_thread")] async fn ephemeral_serves_finalised_blocks_zebrad() { - ephemeral_serves_finalised_blocks::(&ValidatorKind::Zebrad).await + ephemeral_serves_finalised_blocks::(&ValidatorKind::Zebrad).await } /// Ephemeral mode on regtest: the chain index opens no persistent @@ -298,17 +296,15 @@ mod chain_query_interface { /// its hash, and asserts the two are identical; /// - streams compact blocks across the finalised / non-finalised boundary; /// - asserts nothing was persisted to disk. - async fn ephemeral_serves_finalised_blocks(validator: &ValidatorKind) + async fn ephemeral_serves_finalised_blocks(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { use zaino_proto::proto::utils::PoolTypeFilter; let (test_manager, json_service, _option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, true) + create_test_manager_and_chain_index::(validator, None, false, false, true) .await; // The finalised floor sits at `tip - seam`; the non-finalised cache retains a @@ -381,25 +377,23 @@ mod chain_query_interface { #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn sync_large_chain_zebrad() { - sync_large_chain::(&ValidatorKind::Zebrad).await + sync_large_chain::(&ValidatorKind::Zebrad).await } #[cfg(feature = "zcashd_support")] #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn sync_large_chain_zcashd() { - sync_large_chain::(&ValidatorKind::Zcashd).await + sync_large_chain::(&ValidatorKind::Zcashd).await } - async fn sync_large_chain(validator: &ValidatorKind) + async fn sync_large_chain(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (test_manager, json_service, option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, false) + create_test_manager_and_chain_index::(validator, None, false, false, false) .await; test_manager @@ -459,25 +453,23 @@ mod chain_query_interface { // #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_subtree_roots_zebrad() { - get_subtree_roots::(&ValidatorKind::Zebrad).await + get_subtree_roots::(&ValidatorKind::Zebrad).await } #[cfg(feature = "zcashd_support")] #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_subtree_roots_zcashd() { - get_subtree_roots::(&ValidatorKind::Zcashd).await + get_subtree_roots::(&ValidatorKind::Zcashd).await } - async fn get_subtree_roots(validator: &ValidatorKind) + async fn get_subtree_roots(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (test_manager, json_service, _option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, false) + create_test_manager_and_chain_index::(validator, None, false, false, false) .await; test_manager @@ -564,30 +556,26 @@ mod chain_query_interface { // #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_mempool_stream_fresh_snapshot_repeated_zebrad() { - get_mempool_stream_fresh_snapshot_repeated::(&ValidatorKind::Zebrad) - .await + get_mempool_stream_fresh_snapshot_repeated::(&ValidatorKind::Zebrad).await } #[cfg(feature = "zcashd_support")] #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn get_mempool_stream_fresh_snapshot_repeated_zcashd() { - get_mempool_stream_fresh_snapshot_repeated::(&ValidatorKind::Zcashd) - .await + get_mempool_stream_fresh_snapshot_repeated::(&ValidatorKind::Zcashd).await } - async fn get_mempool_stream_fresh_snapshot_repeated(validator: &ValidatorKind) + async fn get_mempool_stream_fresh_snapshot_repeated(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::StreamExt as _; use tokio::time::{timeout, Duration}; let (test_manager, _json_service, _option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, false) + create_test_manager_and_chain_index::(validator, None, false, false, false) .await; test_manager @@ -623,28 +611,26 @@ mod chain_query_interface { // #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn zallet_like_steady_state_loop_zebrad() { - zallet_like_steady_state_loop::(&ValidatorKind::Zebrad).await + zallet_like_steady_state_loop::(&ValidatorKind::Zebrad).await } #[cfg(feature = "zcashd_support")] #[ignore = "prone to timeouts and hangs, to be fixed in chain index integration"] #[tokio::test(flavor = "multi_thread")] async fn zallet_like_steady_state_loop_zcashd() { - zallet_like_steady_state_loop::(&ValidatorKind::Zcashd).await + zallet_like_steady_state_loop::(&ValidatorKind::Zcashd).await } - async fn zallet_like_steady_state_loop(validator: &ValidatorKind) + async fn zallet_like_steady_state_loop(validator: &ValidatorKind) where C: ValidatorExt, - Service: zaino_testutils::TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: zaino_testutils::PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::{StreamExt as _, TryStreamExt as _}; use tokio::time::{timeout, Duration}; let (test_manager, _json_service, _option_state_service, _chain_index, indexer) = - create_test_manager_and_chain_index::(validator, None, false, false, false) + create_test_manager_and_chain_index::(validator, None, false, false, false) .await; test_manager diff --git a/live-tests/clientless/tests/compact_block_consistency.rs b/live-tests/clientless/tests/compact_block_consistency.rs index 98d60daa8..18bf0116a 100644 --- a/live-tests/clientless/tests/compact_block_consistency.rs +++ b/live-tests/clientless/tests/compact_block_consistency.rs @@ -11,10 +11,9 @@ use zaino_common::network::ActivationHeights; use zaino_fetch::jsonrpsee::response::GetBlockResponse; #[allow(deprecated)] -use zaino_state::FetchService; use zaino_state::ZcashIndexer as _; use zaino_testutils::{ - MinerPool, TestManager, ValidatorKind, NU6_3_ACTIVE_ACTIVATION_HEIGHTS, + MinerPool, Rpc, TestManager, ValidatorKind, IRONWOOD_ONLY_ACTIVATION_HEIGHTS, NU6_3_TRANSITION_BOUNDARY, ORCHARD_ONLY_ACTIVATION_HEIGHTS, ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS, }; @@ -25,7 +24,7 @@ use zebra_chain::serialization::ZcashDeserialize as _; #[allow(deprecated)] #[tokio::test(flavor = "multi_thread")] async fn unfiltered_compact_blocks_match_chain_metadata_zebrad() { - let mut test_manager = TestManager::::launch_mining_to( + let mut test_manager = TestManager::::launch_mining_to( // Shielded mining: from NU6.3 an orchard-receiver coinbase is built as Ironwood // actions (the coinbase's Orchard component must be empty from NU6.3), so every // generated block carries ironwood data for the walk to check. A transparent @@ -33,7 +32,7 @@ async fn unfiltered_compact_blocks_match_chain_metadata_zebrad() { MinerPool::Orchard, &ValidatorKind::Zebrad, None, - Some(NU6_3_ACTIVE_ACTIVATION_HEIGHTS), + Some(IRONWOOD_ONLY_ACTIVATION_HEIGHTS), None, true, false, @@ -239,7 +238,7 @@ async fn assert_coinbase_routing( expected_era: impl Fn(u64) -> CoinbaseEra, ) { #[allow(deprecated)] - let mut test_manager = TestManager::::launch_mining_to( + let mut test_manager = TestManager::::launch_mining_to( MinerPool::Orchard, &ValidatorKind::Zebrad, None, @@ -326,7 +325,7 @@ async fn orchard_only_coinbase_routing_zebrad() { /// multi_thread required: the test manager spawns the validator and indexer services. #[tokio::test(flavor = "multi_thread")] async fn ironwood_only_coinbase_routing_zebrad() { - assert_coinbase_routing(NU6_3_ACTIVE_ACTIVATION_HEIGHTS, 6, |height| { + assert_coinbase_routing(IRONWOOD_ONLY_ACTIVATION_HEIGHTS, 6, |height| { if height >= 2 { CoinbaseEra::Ironwood } else { diff --git a/live-tests/clientless/tests/fetch_service.rs b/live-tests/clientless/tests/fetch_service.rs index 53ea44785..dfcc290ec 100644 --- a/live-tests/clientless/tests/fetch_service.rs +++ b/live-tests/clientless/tests/fetch_service.rs @@ -1,15 +1,14 @@ //! These tests compare the output of `FetchService` with the output of `JsonRpcConnector`. -use clientless::rpc::z_validate_address::{run_z_validate_for, SaplingSuite}; +use clientless::rpc::z_validate_address::run_z_validate_for; use futures::StreamExt as _; use zaino_fetch::jsonrpsee::connector::JsonRpSeeConnector; use zaino_proto::proto::compact_formats::CompactBlock; use zaino_proto::proto::service::{BlockId, BlockRange, GetSubtreeRootsArg}; -#[allow(deprecated)] use zaino_state::{ - FetchService, FetchServiceSubscriber, LightWalletIndexer, Status, StatusType, ZcashIndexer, + LightWalletIndexer, NodeBackedIndexerServiceSubscriber, Status, StatusType, ZcashIndexer, }; -use zaino_testutils::{TestManager, ValidatorExt, ValidatorKind}; +use zaino_testutils::{Rpc, TestManager, ValidatorExt, ValidatorKind}; use zebra_chain::parameters::subsidy::ParameterSubsidy as _; use zebra_rpc::client::ValidateAddressResponse; use zebra_rpc::methods::{GetBlock, GetBlockHash}; @@ -95,7 +94,7 @@ async fn fetch_service_get_latest_block(validator: &ValidatorKi #[allow(deprecated)] async fn assert_subscriber_matches_rpc( validator: &ValidatorKind, - fetch_query: impl FnOnce(FetchServiceSubscriber) -> FFut, + fetch_query: impl FnOnce(NodeBackedIndexerServiceSubscriber) -> FFut, rpc_query: impl FnOnce(JsonRpSeeConnector) -> RFut, ) where V: ValidatorExt, @@ -214,7 +213,6 @@ async fn fetch_service_get_block_subsidy(validator: &ValidatorK // first halving boundary plus a margin on both validators. let height_limit = fetch_service_subscriber .network() - .to_zebra_network() .height_for_first_halving() .0 + 10; @@ -321,7 +319,7 @@ async fn fetch_service_get_block_header(validator: &ValidatorKi #[allow(deprecated)] async fn launch_and_mine_five_blocks( validator: &ValidatorKind, -) -> (TestManager, FetchServiceSubscriber) { +) -> (TestManager, NodeBackedIndexerServiceSubscriber) { let (test_manager, fetch_service_subscriber) = zaino_testutils::launch_with_fetch_subscriber::(validator, None).await; @@ -417,11 +415,11 @@ async fn fetch_service_validate_address(validator: &ValidatorKi /// Launch a fetch-backend manager and run the shared `z_validate_address` /// suite against its subscriber. -async fn z_validate(validator: &ValidatorKind, suite: SaplingSuite) { +async fn z_validate(validator: &ValidatorKind) { let (mut test_manager, fetch_service_subscriber) = zaino_testutils::launch_with_fetch_subscriber::(validator, None).await; - run_z_validate_for(&fetch_service_subscriber, suite).await; + run_z_validate_for(&fetch_service_subscriber).await; test_manager.close().await; } @@ -649,7 +647,7 @@ mod zcashd { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] pub(crate) async fn z_validate_address() { - z_validate::(&ValidatorKind::Zcashd, SaplingSuite::Standard).await; + z_validate::(&ValidatorKind::Zcashd).await; } } @@ -720,11 +718,7 @@ mod zebrad { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] pub(crate) async fn z_validate_address() { - z_validate::( - &ValidatorKind::Zebrad, - SaplingSuite::ZebradPassthroughFetchService, - ) - .await; + z_validate::(&ValidatorKind::Zebrad).await; } } diff --git a/live-tests/clientless/tests/json_server.rs b/live-tests/clientless/tests/json_server.rs index 199e0709f..b8c17443c 100644 --- a/live-tests/clientless/tests/json_server.rs +++ b/live-tests/clientless/tests/json_server.rs @@ -5,9 +5,8 @@ //! docs/adr/0001-zcashd-support-feature-gate.md. #![cfg(feature = "zcashd_support")] -#[allow(deprecated)] -use zaino_state::{FetchService, FetchServiceSubscriber, ZcashIndexer}; -use zaino_testutils::TestManager; +use zaino_state::{NodeBackedIndexerServiceSubscriber, ZcashIndexer}; +use zaino_testutils::{Rpc, TestManager}; use zcash_local_net::validator::zcashd::Zcashd; /// Assert that `query` returns the same value from the zcashd-backed and @@ -16,11 +15,11 @@ use zcash_local_net::validator::zcashd::Zcashd; /// problem. The pure-compare building block of the json_server tests. #[allow(deprecated)] async fn assert_subscribers_agree( - zcashd_subscriber: &FetchServiceSubscriber, - zaino_subscriber: &FetchServiceSubscriber, + zcashd_subscriber: &NodeBackedIndexerServiceSubscriber, + zaino_subscriber: &NodeBackedIndexerServiceSubscriber, query: Q, ) where - Q: Fn(FetchServiceSubscriber) -> Fut, + Q: Fn(NodeBackedIndexerServiceSubscriber) -> Fut, Fut: std::future::Future, T: std::fmt::Debug + PartialEq, { @@ -34,13 +33,13 @@ async fn assert_subscribers_agree( /// tests that check an invariant holds across the chain. #[allow(deprecated)] async fn compare_over_blocks( - test_manager: &TestManager, - zcashd_subscriber: &FetchServiceSubscriber, - zaino_subscriber: &FetchServiceSubscriber, + test_manager: &TestManager, + zcashd_subscriber: &NodeBackedIndexerServiceSubscriber, + zaino_subscriber: &NodeBackedIndexerServiceSubscriber, blocks: u32, query: Q, ) where - Q: Fn(FetchServiceSubscriber) -> Fut, + Q: Fn(NodeBackedIndexerServiceSubscriber) -> Fut, Fut: std::future::Future, T: std::fmt::Debug + PartialEq, { @@ -430,11 +429,8 @@ mod zcashd { async fn z_validate_address() { let mut services = zaino_testutils::launch_zcashd_dual_fetch_services().await; - clientless::rpc::z_validate_address::run_z_validate_for( - &services.zcashd_subscriber, - clientless::rpc::z_validate_address::SaplingSuite::Standard, - ) - .await; + clientless::rpc::z_validate_address::run_z_validate_for(&services.zcashd_subscriber) + .await; services.test_manager.close().await; } diff --git a/live-tests/clientless/tests/state_service.rs b/live-tests/clientless/tests/state_service.rs index 5655dced3..59b607d40 100644 --- a/live-tests/clientless/tests/state_service.rs +++ b/live-tests/clientless/tests/state_service.rs @@ -1,11 +1,8 @@ use zaino_fetch::jsonrpsee::response::address_deltas::GetAddressDeltasParams; -#[allow(deprecated)] -use zaino_state::{ - FetchServiceSubscriber, LightWalletIndexer, StateServiceSubscriber, ZcashIndexer, -}; +use zaino_state::{LightWalletIndexer, NodeBackedIndexerServiceSubscriber, ZcashIndexer}; use zaino_testutils::{StateAndFetchServices, ValidatorExt}; -use zaino_testutils::{ValidatorKind, ZEBRAD_TESTNET_CACHE_DIR}; +use zaino_testutils::{ValidatorKind, ZEBRAD_THE_PUB_TESTNET_CACHE_DIR}; use zcash_local_net::validator::zebrad::Zebrad; use zebra_chain::parameters::NetworkKind; use zebra_rpc::methods::{GetAddressBalanceRequest, GetAddressTxIdsRequest}; @@ -26,7 +23,7 @@ async fn launch_regtest(enable_zaino: bool) -> StateAndFetchServices { async fn launch_testnet_cached() -> StateAndFetchServices { zaino_testutils::launch_state_and_fetch_services::( &ValidatorKind::Zebrad, - ZEBRAD_TESTNET_CACHE_DIR.clone(), + ZEBRAD_THE_PUB_TESTNET_CACHE_DIR.clone(), false, Some(NetworkKind::Testnet), ) @@ -42,8 +39,8 @@ async fn launch_testnet_cached() -> StateAndFetchServices { #[allow(deprecated)] async fn assert_subscribers_agree( services: &StateAndFetchServices, - fetch_query: impl FnOnce(FetchServiceSubscriber) -> FFut, - state_query: impl FnOnce(StateServiceSubscriber) -> SFut, + fetch_query: impl FnOnce(NodeBackedIndexerServiceSubscriber) -> FFut, + state_query: impl FnOnce(NodeBackedIndexerServiceSubscriber) -> SFut, ) -> T where T: std::fmt::Debug + PartialEq, @@ -386,7 +383,10 @@ mod zebra { #[tokio::test(flavor = "multi_thread")] async fn state_service_chaintip_update_subscriber() { let services = launch_regtest(true).await; - let mut chaintip_subscriber = services.state_subscriber.chaintip_update_subscriber(); + let mut chaintip_subscriber = services + .state_subscriber + .chaintip_update_subscriber() + .expect("the Direct connection exposes a local tip-change stream"); services .test_manager .generate_blocks_and_check_each( @@ -427,7 +427,7 @@ mod zebra { async fn testnet() { state_service_check_info::( &ValidatorKind::Zebrad, - ZEBRAD_TESTNET_CACHE_DIR.clone(), + ZEBRAD_THE_PUB_TESTNET_CACHE_DIR.clone(), NetworkKind::Testnet, ) .await; @@ -576,11 +576,8 @@ mod zebra { ) .await; - clientless::rpc::z_validate_address::run_z_validate_for( - &services.state_subscriber, - clientless::rpc::z_validate_address::SaplingSuite::Standard, - ) - .await; + clientless::rpc::z_validate_address::run_z_validate_for(&services.state_subscriber) + .await; services.test_manager.close().await; } @@ -615,7 +612,7 @@ mod zebra { async fn block_object_testnet() { state_service_get_block_object( &ValidatorKind::Zebrad, - ZEBRAD_TESTNET_CACHE_DIR.clone(), + ZEBRAD_THE_PUB_TESTNET_CACHE_DIR.clone(), NetworkKind::Testnet, ) .await; @@ -631,7 +628,7 @@ mod zebra { async fn block_raw_testnet() { state_service_get_block_raw( &ValidatorKind::Zebrad, - ZEBRAD_TESTNET_CACHE_DIR.clone(), + ZEBRAD_THE_PUB_TESTNET_CACHE_DIR.clone(), NetworkKind::Testnet, ) .await; diff --git a/live-tests/clientless/tests/the_pub_testnet_ironwood_boundary.rs b/live-tests/clientless/tests/the_pub_testnet_ironwood_boundary.rs new file mode 100644 index 000000000..41833480d --- /dev/null +++ b/live-tests/clientless/tests/the_pub_testnet_ironwood_boundary.rs @@ -0,0 +1,157 @@ +//! Observational walk of the real NU6.3 activation boundary on The Public +//! Testnet (glossary: The Public Testnet is the public test network and +//! nothing else). Historical blocks are permanent, so these invariants are +//! checkable forever, long after the epoch that produced them closed. +//! +//! The invariants, enforced one block below the boundary and at it: +//! +//! - Below the activation height the Ironwood pool holds no value. +//! - From the activation height the Orchard pool's value never increases: +//! the no-new-value rule admits only withdrawals and same-receiver change +//! (the cross-address restriction, +//! ). +//! +//! The activation height itself is read from the running validator's +//! `getblockchaininfo.upgrades` — the single source of truth for activation +//! heights — and cross-checked against the observed activation on The +//! Public Testnet at height 4,134,000 (~2026-07-04). +//! +//! Non-hermetic and therefore env-gated: the test needs a zebrad chain +//! cache at `~/.cache/zebra` (`ZEBRAD_THE_PUB_TESTNET_CACHE_DIR`) synced past the +//! activation height, and skips with a message when the cache is absent or +//! short. Run it manually, or from a job that maintains the cache. +//! +//! The validator is launched through `ZebradConfig` directly rather than +//! `TestManager`: the manager's launch path always writes regtest test +//! parameters into the config, while this test needs The Public Testnet +//! (`network_type: NetworkType::Testnet`, which makes zebrad ignore +//! `activation_heights` and `miner_address`). + +use zaino_fetch::jsonrpsee::connector::{test_node_and_return_url, JsonRpSeeConnector}; +use zaino_fetch::jsonrpsee::response::GetBlockResponse; +use zaino_testutils::ZEBRAD_THE_PUB_TESTNET_CACHE_DIR; +use zcash_local_net::process::Process as _; +use zcash_local_net::protocol::NetworkType; +use zcash_local_net::validator::zebrad::{Zebrad, ZebradConfig}; +use zcash_local_net::validator::Validator as _; +use zebra_chain::parameters::NetworkUpgrade; + +/// The height The Public Testnet activated NU6.3 at, recorded from the real +/// activation. A mismatch means either a reset of The Public Testnet or a +/// wrong validator pin — both worth failing loudly over. +const OBSERVED_NU6_3_ACTIVATION_ON_THE_PUB_TESTNET: u32 = 4_134_000; + +/// The chain value of `pool_id` as of `height`, from the validator's +/// verbosity-2 block object. +async fn pool_zats(connector: &JsonRpSeeConnector, height: u32, pool_id: &str) -> i64 { + let response = connector + .get_block(height.to_string(), Some(2)) + .await + .expect("getblock verbosity 2"); + let GetBlockResponse::Object(block) = response else { + panic!("verbosity-2 getblock must return a block object"); + }; + block + .value_pools() + .expect("verbosity-2 block object carries value pools") + .iter() + .map(|pool| pool.balance()) + .find(|pool| pool.id() == pool_id) + .unwrap_or_else(|| panic!("value pools must include {pool_id}")) + .chain_value_zat() + .zatoshis() +} + +/// multi_thread required: the test launches the validator process and polls +/// it over RPC. +#[tokio::test(flavor = "multi_thread")] +async fn value_pools_respect_the_boundary_on_the_pub_testnet() { + let Some(cache_dir) = ZEBRAD_THE_PUB_TESTNET_CACHE_DIR.clone() else { + eprintln!("skipping: no cache dir configured for The Public Testnet"); + return; + }; + if !cache_dir.exists() { + eprintln!( + "skipping: no zebrad chain cache of The Public Testnet at {}", + cache_dir.display() + ); + return; + } + + let config = ZebradConfig { + network_type: NetworkType::Testnet, + chain_cache: Some(cache_dir), + ..ZebradConfig::default() + }; + let mut zebrad = Zebrad::launch(config) + .await + .expect("launch a zebrad on The Public Testnet"); + + let rpc_address = format!("127.0.0.1:{}", zebrad.get_port()); + let connector = JsonRpSeeConnector::new_with_basic_auth( + test_node_and_return_url( + &rpc_address, + None, + Some("xxxxxx".to_string()), + Some("xxxxxx".to_string()), + ) + .await + .expect("validator RPC reachable"), + "xxxxxx".to_string(), + "xxxxxx".to_string(), + ) + .expect("connect to the validator RPC"); + + let blockchain_info = connector + .get_blockchain_info() + .await + .expect("getblockchaininfo"); + + // The validator's reported schedule is the source of truth for the + // boundary; the recorded constant pins the real history of The Public Testnet. + let boundary = blockchain_info + .upgrades + .values() + .find_map(|upgrade_info| { + let (upgrade, height, _status) = upgrade_info.into_parts(); + (upgrade == NetworkUpgrade::Nu6_3).then_some(height.0) + }) + .expect("the validator on The Public Testnet must report an NU6.3 activation height"); + assert_eq!( + boundary, OBSERVED_NU6_3_ACTIVATION_ON_THE_PUB_TESTNET, + "the validator's NU6.3 height must match the observed activation on The Public Testnet" + ); + + let tip = blockchain_info.blocks.0; + if tip <= boundary { + eprintln!( + "skipping: cache tip {tip} of The Public Testnet has not crossed the NU6.3 boundary {boundary}" + ); + zebrad.stop(); + return; + } + + // One block below the boundary and at it, plus a block of margin on + // each side: the Ironwood pool holds no value below the activation + // height, and the Orchard pool never grows from it. + for height in (boundary - 2)..boundary { + assert_eq!( + pool_zats(&connector, height, "ironwood").await, + 0, + "the ironwood pool must hold no value at height {height}, below the boundary" + ); + } + let walk_end = boundary + 1; + let mut previous_orchard = pool_zats(&connector, boundary - 1, "orchard").await; + for height in boundary..=walk_end { + let orchard = pool_zats(&connector, height, "orchard").await; + assert!( + orchard <= previous_orchard, + "the orchard pool must never grow from the boundary; \ + height {height} holds {orchard} zats after {previous_orchard}" + ); + previous_orchard = orchard; + } + + zebrad.stop(); +} diff --git a/live-tests/clientless/tests/validator_heights.rs b/live-tests/clientless/tests/validator_heights.rs new file mode 100644 index 000000000..62986f84d --- /dev/null +++ b/live-tests/clientless/tests/validator_heights.rs @@ -0,0 +1,153 @@ +//! Regression tests for the single source of truth for activation heights +//! (zaino#1076, `zainod-heights-from-validator-spec.md`). +//! +//! The invariant: the validator's configured activation heights are +//! authoritative. zainod's config carries only a network kind — its regtest +//! placeholder is the canonical `ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS` +//! regardless of the fixture — and both backends adopt the real schedule +//! from `getblockchaininfo.upgrades` at spawn. +//! +//! Two halves, matching the spec's acceptance criteria: +//! +//! 1. [`zainod_syncs_a_schedule_its_config_never_saw`] — boundary sync and +//! the no-recompile proof in one: the same zainod build and (kind-only) +//! configuration that every canonical-heights test runs is here pointed +//! at a validator on the NU6.3-at-6 transition schedule, and must sync +//! across the boundary and serve era-correct compact blocks. Before +//! adoption this exact misalignment killed the chain-index sync with +//! `InvalidData("Block commitment could not be computed")`. +//! 2. [`getblockchaininfo_reports_the_configured_schedule`] — the input +//! contract: what zebrad actually puts in the `upgrades` map for a +//! configured schedule, pinned against a live node rather than assumed. +//! The mapping from that shape to adopted heights is unit-tested next to +//! `activation_heights_from_upgrades` in zaino-state. + +use zaino_state::ZcashIndexer as _; +use zaino_testutils::{ + all_pools_i32, collect_block_range, MinerPool, Rpc, TestManager, ValidatorKind, + NU6_3_TRANSITION_BOUNDARY, ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS, + ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS, +}; +use zcash_local_net::validator::zebrad::Zebrad; + +/// Launches an orchard-receiver-mining zebrad on the transition schedule +/// with zainod enabled. The harness hands zainod only the canonical +/// placeholder (see `launch_mining_to`), so the launch itself is the +/// deliberate config/validator misalignment under test. +async fn launch_transition_validator() -> TestManager { + assert_ne!( + ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS, ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS, + "premise: zainod's config placeholder must differ from the fixture \ + schedule, or this proves nothing" + ); + + TestManager::::launch_mining_to( + MinerPool::Orchard, + &ValidatorKind::Zebrad, + None, + Some(ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS), + None, + true, + false, + false, + ) + .await + .expect("launch TestManager") +} + +/// Boundary sync + no-recompile proof: a kind-only-configured zainod adopts +/// the NU6.3-at-6 schedule from the validator, syncs across the boundary, +/// and serves era-correct compact blocks for both eras. +/// +/// multi_thread required: the test manager spawns the validator and indexer +/// services. +#[tokio::test(flavor = "multi_thread")] +async fn zainod_syncs_a_schedule_its_config_never_saw() { + let mut test_manager = launch_transition_validator().await; + let subscriber = test_manager.subscriber().clone(); + + // Two blocks past the boundary, so both eras carry more than one block. + // Reaching the tip at all is the core regression: pre-adoption, the + // chain-index sync died on the first block whose commitment scheme the + // misconfigured heights got wrong. + test_manager + .generate_blocks_and_wait_for_tip(NU6_3_TRANSITION_BOUNDARY + 1, &subscriber) + .await; + let tip = u64::from(subscriber.chain_height().await.expect("chain height").0); + assert!( + tip > u64::from(NU6_3_TRANSITION_BOUNDARY), + "sync must cross the boundary, tip is {tip}" + ); + + // Era composition of the served chain proves the adopted schedule is the + // validator's, not the placeholder: under the placeholder (NU6.3 at 2) + // the pre-boundary orchard coinbases would be misread as ironwood-era. + let blocks = collect_block_range(&subscriber, 2, tip, all_pools_i32()).await; + for block in &blocks { + let height = block.height; + let has_orchard = block.vtx.iter().any(|tx| !tx.actions.is_empty()); + let has_ironwood = block.vtx.iter().any(|tx| !tx.ironwood_actions.is_empty()); + if height >= u64::from(NU6_3_TRANSITION_BOUNDARY) { + assert!( + has_ironwood && !has_orchard, + "height {height} must be ironwood-era, got orchard={has_orchard} ironwood={has_ironwood}" + ); + } else { + assert!( + has_orchard && !has_ironwood, + "height {height} must be orchard-era, got orchard={has_orchard} ironwood={has_ironwood}" + ); + } + } + + test_manager.close().await; +} + +/// The input contract for adoption: the `upgrades` map a live zebrad reports +/// for the transition schedule, pinned exactly — upgrade set, order, and +/// heights. Establishes (from real output, not reasoning) that nothing +/// pre-Overwinter appears: the map is keyed by consensus branch ID, which +/// pre-Overwinter eras don't have. +/// +/// multi_thread required: the test manager spawns the validator and indexer +/// services. +#[tokio::test(flavor = "multi_thread")] +async fn getblockchaininfo_reports_the_configured_schedule() { + use zebra_chain::parameters::NetworkUpgrade; + + let mut test_manager = launch_transition_validator().await; + + let blockchain_info = test_manager + .full_node_jsonrpc_connector() + .await + .get_blockchain_info() + .await + .expect("getblockchaininfo"); + + let reported: Vec<(NetworkUpgrade, u32)> = blockchain_info + .upgrades + .values() + .map(|upgrade_info| { + let (upgrade, height, _status) = upgrade_info.into_parts(); + (upgrade, height.0) + }) + .collect(); + + assert_eq!( + reported, + vec![ + (NetworkUpgrade::Overwinter, 1), + (NetworkUpgrade::Sapling, 1), + (NetworkUpgrade::Blossom, 1), + (NetworkUpgrade::Heartwood, 1), + (NetworkUpgrade::Canopy, 1), + (NetworkUpgrade::Nu5, 2), + (NetworkUpgrade::Nu6, 2), + (NetworkUpgrade::Nu6_1, 2), + (NetworkUpgrade::Nu6_2, 2), + (NetworkUpgrade::Nu6_3, 6), + ], + ); + + test_manager.close().await; +} diff --git a/live-tests/e2e/src/devtool.rs b/live-tests/e2e/src/devtool.rs index 4fed529e6..13ed1679d 100644 --- a/live-tests/e2e/src/devtool.rs +++ b/live-tests/e2e/src/devtool.rs @@ -4,7 +4,7 @@ //! //! [`DevtoolClients`] mirrors [`crate::Clients`]' method names one-for-one so //! tests can swap backends mechanically. The clients are managed by -//! zcash_local_net's [`zcash_local_net::client`] module: each wallet +//! zcash_local_net's [`zcash_local_net::wallet`] module: each wallet //! operation is a run-to-completion `zcash-devtool` subprocess invocation //! (the binary must be built with `--features regtest_support` and be //! locatable via `TEST_BINARIES_DIR`/`PATH`). @@ -21,9 +21,9 @@ //! constants derived from zingolib note selection (e.g. 235_000 after //! shielding 250_000) must be re-verified on first devtool runs. -use zcash_local_net::client::{ +use zcash_local_net::wallet::{ zcash_devtool::{ZcashDevtool, ZcashDevtoolConfig}, - AddressReceiver, Client as _, GetInfo, WalletBalance, + AddressReceiver, GetInfo, Wallet as _, WalletBalance, }; use zcash_primitives::transaction::TxId; @@ -39,17 +39,27 @@ pub struct DevtoolClients { } /// Launch faucet + recipient devtool wallets against a running Zaino gRPC -/// listener. The devtool analogue of [`crate::build_clients`]; Zaino must -/// already be serving (wallet initialization fetches the chain tip and -/// birthday tree state from it). -pub async fn build_clients(zaino_grpc_listen_port: u16) -> DevtoolClients { - let mut faucet_config = ZcashDevtoolConfig::faucet(); +/// listener, deriving the wallet network from the running `validator`. That +/// derivation is the only construction the client API offers +/// (infrastructure ADR 0003: the Validator is the single source of truth +/// for activation heights), so wallet/validator height drift is +/// unrepresentable — the wallets follow whatever schedule the validator was +/// launched with, fixture or canonical. The devtool analogue of +/// [`crate::build_clients`]; Zaino must already be serving (wallet +/// initialization fetches the chain tip and birthday tree state from it). +pub async fn build_clients( + zaino_grpc_listen_port: u16, + validator: &V, +) -> DevtoolClients { + let network = zcash_local_net::wallet::WalletNetwork::from_validator(validator).await; + + let mut faucet_config = ZcashDevtoolConfig::faucet(network); faucet_config.indexer_port = zaino_grpc_listen_port; let faucet = ZcashDevtool::launch(faucet_config) .await .expect("launch devtool faucet wallet"); - let mut recipient_config = ZcashDevtoolConfig::recipient(); + let mut recipient_config = ZcashDevtoolConfig::recipient(network); recipient_config.indexer_port = zaino_grpc_listen_port; let recipient = ZcashDevtool::launch(recipient_config) .await diff --git a/live-tests/e2e/tests/compact_block_wire.rs b/live-tests/e2e/tests/compact_block_wire.rs index 495de0c2d..ab77775d6 100644 --- a/live-tests/e2e/tests/compact_block_wire.rs +++ b/live-tests/e2e/tests/compact_block_wire.rs @@ -15,10 +15,9 @@ use zaino_proto::proto::compact_formats::CompactBlock; use zaino_proto::proto::service::compact_tx_streamer_client::CompactTxStreamerClient; use zaino_proto::proto::service::{BlockId, BlockRange}; #[allow(deprecated)] -use zaino_state::FetchService; use zaino_state::ZcashIndexer as _; use zaino_testutils::{ - make_uri, MinerPool, TestManager, ValidatorKind, NU6_3_ACTIVE_ACTIVATION_HEIGHTS, + make_uri, MinerPool, Rpc, TestManager, ValidatorKind, IRONWOOD_ONLY_ACTIVATION_HEIGHTS, NU6_3_TRANSITION_BOUNDARY, ORCHARD_ONLY_ACTIVATION_HEIGHTS, ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS, }; @@ -49,7 +48,7 @@ async fn assert_wire_served_eras( expected_era: impl Fn(u64) -> CoinbaseEra, ) { #[allow(deprecated)] - let mut test_manager = TestManager::::launch_mining_to( + let mut test_manager = TestManager::::launch_mining_to( MinerPool::Orchard, &ValidatorKind::Zebrad, None, @@ -171,7 +170,7 @@ async fn orchard_only_wire_serving_zebrad() { /// multi_thread required: the test manager spawns the validator, indexer, and zainod. #[tokio::test(flavor = "multi_thread")] async fn ironwood_only_wire_serving_zebrad() { - assert_wire_served_eras(NU6_3_ACTIVE_ACTIVATION_HEIGHTS, 6, |height| { + assert_wire_served_eras(IRONWOOD_ONLY_ACTIVATION_HEIGHTS, 6, |height| { if height >= 2 { CoinbaseEra::Ironwood } else { diff --git a/live-tests/e2e/tests/devtool.rs b/live-tests/e2e/tests/devtool.rs index e334ca441..94012c554 100644 --- a/live-tests/e2e/tests/devtool.rs +++ b/live-tests/e2e/tests/devtool.rs @@ -43,9 +43,8 @@ use e2e::devtool::DevtoolClients; use zaino_proto::proto::service::{ AddressList, BlockId, BlockRange, GetAddressUtxosArg, TransparentAddressBlockFilter, TxFilter, }; -use zaino_state::{LightWalletIndexer, ZcashIndexer, ZcashService}; -use zaino_testutils::{PollableTip, TestManager, TestService, ValidatorKind}; -use zainodlib::error::IndexerError; +use zaino_state::{LightWalletIndexer, ZcashIndexer}; +use zaino_testutils::{PollableTip, Rpc, TestManager, ValidatorKind}; use zcash_local_net::validator::zebrad::Zebrad; use zebra_chain::subtree::NoteCommitmentSubtreeIndex; use zebra_rpc::methods::{GetAddressBalanceRequest, GetAddressTxIdsRequest}; @@ -58,15 +57,13 @@ use zebra_rpc::methods::{GetAddressBalanceRequest, GetAddressTxIdsRequest}; /// spendable orchard notes the faucet ends with — one per send a test makes /// before mining (devtool will not chain unconfirmed change). The shared /// launch+fund preamble of the ports below. -async fn launch_and_fund_faucet( +async fn launch_and_fund_faucet( coinbase_blocks: u32, -) -> (TestManager, DevtoolClients) +) -> (TestManager, DevtoolClients) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let test_manager = TestManager::::launch_mining_to( + let test_manager = TestManager::::launch_mining_to( zaino_testutils::SHIELDED_FUNDING_POOL, &ValidatorKind::Zebrad, None, @@ -84,6 +81,7 @@ where .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &test_manager.local_net, ) .await; @@ -98,13 +96,11 @@ where /// Launch an orchard-mining zebrad + Zaino on the `Service` backend and build /// devtool faucet/recipient wallets against it, without mining or syncing — the /// minimal preamble for tests that only exercise wallet↔server connectivity. -async fn launch_and_build_clients() -> (TestManager, DevtoolClients) +async fn launch_and_build_clients() -> (TestManager, DevtoolClients) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let test_manager = TestManager::::launch_mining_to( + let test_manager = TestManager::::launch_mining_to( zaino_testutils::SHIELDED_FUNDING_POOL, &ValidatorKind::Zebrad, None, @@ -122,6 +118,7 @@ where .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &test_manager.local_net, ) .await; @@ -132,13 +129,11 @@ where /// and recipient wallets can report node/server info without erroring. Smoke /// test — the original discards the result. (`chain_name` is "test" for regtest /// and `server_uri` has no trailing slash, so this asserts neither.) -async fn connect_to_node_get_info() +async fn connect_to_node_get_info() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, clients) = launch_and_build_clients::().await; + let (mut test_manager, clients) = launch_and_build_clients::().await; clients.get_info_faucet().await; clients.get_info_recipient().await; @@ -148,13 +143,11 @@ where /// Port of `wallet_to_validator::check_received_mining_reward` (zebrad): the /// faucet's synced wallet sees the orchard coinbase note. -async fn receives_mining_reward() +async fn receives_mining_reward() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, clients) = launch_and_fund_faucet::(1).await; + let (mut test_manager, clients) = launch_and_fund_faucet::(1).await; let faucet_balance = dbg!(clients.faucet_balance().await); // Under the canonical NU6.3-at-2 heights the orchard-receiver mining reward @@ -176,13 +169,11 @@ where /// `send_to_transparent` additionally mines across the finalization boundary /// under transparent mining; that variant is deferred, as it exercises the /// transparent-coinbase detection path devtool has not yet been run against.) -async fn send_to_pool(pool: e2e::Pool) +async fn send_to_pool(pool: e2e::Pool) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient = clients.get_recipient_address(pool.address_kind()).await; let txid = clients.send_from_faucet(&recipient, 250_000).await; @@ -211,14 +202,12 @@ where /// balance assertions — the sends confirm in one block, and received funds are /// regular (non-coinbase) outputs with no maturity rule — and 100 orchard /// blocks would cost 100 halo2 proofs against the net-speedup criterion. -async fn send_to_all() +async fn send_to_all() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { // Three orchard notes — one per send (devtool will not chain unconfirmed change). - let (mut test_manager, mut clients) = launch_and_fund_faucet::(3).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(3).await; let recipient_ua = clients.get_recipient_address("unified").await; let recipient_zaddr = clients.get_recipient_address("sapling").await; @@ -250,13 +239,11 @@ where /// confirms the shielded balance net of the ZIP-317 fee (250_000 − 15_000 = /// 235_000 — the first devtool exercise of `shield`, and the constant flagged /// as needing re-verification against devtool's note selection). -async fn shield_for_validator() +async fn shield_for_validator() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient_taddr = clients.get_recipient_address("transparent").await; clients.send_from_faucet(&recipient_taddr, 250_000).await; @@ -288,14 +275,12 @@ where /// mining) one transparent and one unified send into the mempool. Returns the /// manager and the two broadcast txids (transparent, then unified). The /// devtool analogue of `fund_and_fill_mempool`. -async fn fund_and_fill_mempool() -> (TestManager, String, String) +async fn fund_and_fill_mempool() -> (TestManager, String, String) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { // Two orchard notes — one per unmined send. - let (test_manager, mut clients) = launch_and_fund_faucet::(2).await; + let (test_manager, mut clients) = launch_and_fund_faucet::(2).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let recipient_ua = clients.get_recipient_address("unified").await; @@ -313,15 +298,13 @@ where /// txid hex. The txid is in display (reversed) order — devtool prints it that /// way, which matches the txid strings zaino's address queries return, so no /// conversion is needed for those comparisons. -async fn fund_and_send_to( +async fn fund_and_send_to( pool: e2e::Pool, -) -> (TestManager, DevtoolClients, String) +) -> (TestManager, DevtoolClients, String) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient = clients.get_recipient_address(pool.address_kind()).await; let txid_hex = clients.send_from_faucet(&recipient, 250_000).await; @@ -336,14 +319,12 @@ where /// Port of `fetch_service_get_address_tx_ids` (zebrad): after a transparent /// send to the recipient, `get_address_tx_ids` over the recipient's taddr /// returns the send's txid. -async fn get_address_tx_ids() +async fn get_address_tx_ids() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, clients, txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; // A range start a couple of blocks below the tip, covering the send block. @@ -367,14 +348,12 @@ where /// Port of `fetch_service_get_address_utxos` (zebrad): after a transparent /// send to the recipient, `z_get_address_utxos` over the recipient's taddr /// returns a utxo whose txid is the send's. -async fn get_address_utxos() +async fn get_address_utxos() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, clients, txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let utxos = test_manager @@ -392,14 +371,12 @@ where /// Port of `fetch_service_z_get_treestate` (zebrad, smoke): `z_get_treestate` /// at the tip height succeeds. -async fn z_get_treestate() +async fn z_get_treestate() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, _clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Orchard).await; + fund_and_send_to::(e2e::Pool::Orchard).await; let tip = test_manager.subscriber().tip_height().await; dbg!(test_manager @@ -413,14 +390,12 @@ where /// Port of `fetch_service_z_get_subtrees_by_index` (zebrad, smoke): /// `z_get_subtrees_by_index` for orchard succeeds. -async fn z_get_subtrees_by_index() +async fn z_get_subtrees_by_index() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, _clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Orchard).await; + fund_and_send_to::(e2e::Pool::Orchard).await; dbg!(test_manager .subscriber() @@ -433,14 +408,11 @@ where /// Port of `fetch_service_get_raw_transaction` (zebrad, smoke): /// `get_raw_transaction` for the orchard send's txid succeeds. -async fn get_raw_transaction() +async fn get_raw_transaction() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, _clients, txid_hex) = - fund_and_send_to::(e2e::Pool::Orchard).await; + let (mut test_manager, _clients, txid_hex) = fund_and_send_to::(e2e::Pool::Orchard).await; dbg!(test_manager .subscriber() @@ -454,16 +426,14 @@ where /// Port of `fetch_service_get_taddress_txids` (zebrad, smoke): /// `get_taddress_txids` over the recipient's taddr and a height range around /// the send succeeds. -async fn get_taddress_txids() +async fn get_taddress_txids() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::StreamExt as _; let (mut test_manager, clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let tip = test_manager.subscriber().tip_height().await; @@ -497,14 +467,12 @@ where /// Port of `fetch_service_get_taddress_utxos` (zebrad, smoke): /// `get_address_utxos` over the recipient's taddr succeeds. -async fn get_taddress_utxos() +async fn get_taddress_utxos() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let utxos_arg = GetAddressUtxosArg { @@ -523,16 +491,14 @@ where /// Port of `fetch_service_get_taddress_utxos_stream` (zebrad, smoke): /// `get_address_utxos_stream` over the recipient's taddr succeeds. -async fn get_taddress_utxos_stream() +async fn get_taddress_utxos_stream() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::StreamExt as _; let (mut test_manager, clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let utxos_arg = GetAddressUtxosArg { @@ -556,14 +522,12 @@ where /// Port of `fetch_service_get_raw_mempool` (zebrad): with two transactions /// broadcast but unmined, the indexer's `get_raw_mempool` matches the /// validator's own JSON-RPC `getrawmempool`. -async fn get_raw_mempool() +async fn get_raw_mempool() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, _transparent_txid, _unified_txid) = - fund_and_fill_mempool::().await; + fund_and_fill_mempool::().await; let json_service = test_manager.full_node_jsonrpc_connector().await; @@ -582,17 +546,14 @@ where /// Port of `fetch_service_get_mempool_tx` (zebrad): the `get_mempool_tx` /// stream returns the two unmined transactions keyed by txid (internal byte /// order), and the exclude-by-txid-suffix filter drops the named one. -async fn get_mempool_tx() +async fn get_mempool_tx() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::StreamExt as _; use zaino_proto::proto::service::GetMempoolTxRequest; - let (mut test_manager, transparent_txid, unified_txid) = - fund_and_fill_mempool::().await; + let (mut test_manager, transparent_txid, unified_txid) = fund_and_fill_mempool::().await; let to_bytes = |hex: &str| -> [u8; 32] { e2e::devtool::txid_internal_bytes(hex) @@ -643,16 +604,14 @@ where /// Port of `fetch_service_get_mempool_stream` (zebrad): a `get_mempool_stream` /// subscription opened before two sends observes those unmined transactions. /// Smoke test — the original only `dbg!`s the collected stream. -async fn get_mempool_stream() +async fn get_mempool_stream() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { use futures::StreamExt as _; // Two orchard notes — one per unmined send. - let (mut test_manager, mut clients) = launch_and_fund_faucet::(2).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(2).await; // Subscribe before the sends so the stream observes them entering the mempool. let subscriber = test_manager.subscriber().clone(); @@ -692,13 +651,11 @@ where /// order. The devtool analogue of `fund_and_send(Pool::Orchard)`; the wallet /// clients are dropped once the transaction is mined (the queries below hit /// zaino, not the wallet). -async fn fund_and_send_orchard() -> (TestManager, Vec) +async fn fund_and_send_orchard() -> (TestManager, Vec) where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient_ua = clients.get_recipient_address("unified").await; let txid_hex = clients.send_from_faucet(&recipient_ua, 250_000).await; @@ -713,13 +670,11 @@ where /// Port of `fetch_service_get_transaction_mined` (zebrad): the indexer serves /// `get_transaction` for the mined orchard send, keyed by its txid. Also /// confirms `txid_internal_bytes` yields the order the indexer matches on. -async fn get_transaction_mined() +async fn get_transaction_mined() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, txid_bytes) = fund_and_send_orchard::().await; + let (mut test_manager, txid_bytes) = fund_and_send_orchard::().await; let tx_filter = TxFilter { block: None, @@ -739,13 +694,11 @@ where /// Port of `fetch_service_get_transaction_mempool` (zebrad): the indexer /// serves `get_transaction` for an orchard send that is broadcast but not /// mined, keyed by its txid — i.e. from the mempool. -async fn get_transaction_mempool() +async fn get_transaction_mempool() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient_ua = clients.get_recipient_address("unified").await; let txid_hex = clients.send_from_faucet(&recipient_ua, 250_000).await; @@ -789,6 +742,7 @@ async fn block_range_returns_default_pools() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -880,6 +834,7 @@ async fn block_range_returns_all_pools() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -948,14 +903,12 @@ async fn block_range_returns_all_pools() { /// Port of `fetch_service_get_address_balance` (zebrad): after a transparent /// send of 250_000 to the recipient, `z_get_address_balance` over that taddr /// reports exactly 250_000. -async fn get_address_balance() +async fn get_address_balance() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let balance = test_manager @@ -974,14 +927,12 @@ where /// Port of `fetch_service_get_taddress_balance` (zebrad): after a transparent /// send of 250_000 to the recipient, `get_taddress_balance` over that taddr /// reports value_zat 250_000. -async fn get_taddress_balance() +async fn get_taddress_balance() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip + LightWalletIndexer, + Conn: zaino_testutils::ValidatorConnectionMarker, { let (mut test_manager, clients, _txid_hex) = - fund_and_send_to::(e2e::Pool::Transparent).await; + fund_and_send_to::(e2e::Pool::Transparent).await; let recipient_taddr = clients.get_recipient_address("transparent").await; let address_list = AddressList { @@ -1021,6 +972,7 @@ async fn fund_and_send_dual( .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -1172,6 +1124,7 @@ async fn fund_and_fill_mempool_dual() -> zaino_testutils::StateAndFetchServices< .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -1346,6 +1299,7 @@ async fn launch_transparent_and_faucet_taddr( .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -1592,13 +1546,11 @@ async fn get_block_range_out_of_range_lower_bound() { /// per-call override, so the advance can't be cheap transparent filler until /// per-call filler mining lands; un-ignore and mine the advance to /// a transparent throwaway then. -async fn send_to_transparent_finalization() +async fn send_to_transparent_finalization() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(1).await; let recipient_taddr = clients.get_recipient_address("transparent").await; clients.send_from_faucet(&recipient_taddr, 250_000).await; @@ -1679,6 +1631,7 @@ async fn address_deltas() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -1838,6 +1791,7 @@ async fn get_block_deltas_resolves_transparent_spend() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -2093,6 +2047,7 @@ async fn get_outpoint_spenders_fetch_vs_state() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &svc.test_manager.local_net, ) .await; @@ -2156,13 +2111,11 @@ async fn get_outpoint_spenders_fetch_vs_state() { /// sync is block-based so it never scans the mempool). Restore the assertions /// (using `Pool::spendable_balance` for the confirmed ones) and un-ignore when /// devtool surfaces unconfirmed balances. -async fn monitor_unverified_mempool() +async fn monitor_unverified_mempool() where - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: zaino_testutils::ValidatorConnectionMarker, { - let (mut test_manager, mut clients) = launch_and_fund_faucet::(2).await; + let (mut test_manager, mut clients) = launch_and_fund_faucet::(2).await; let recipient_ua = clients.get_recipient_address("unified").await; let txid_1 = clients.send_from_faucet(&recipient_ua, 250_000).await; @@ -2229,13 +2182,11 @@ where /// Port of `test_get_mempool_info` (fetch_service, zebrad): `get_mempool_info` /// matches values recomputed from the fetch subscriber's mempool internals. /// FetchService-only — the recompute reads `FetchServiceSubscriber.indexer`. -#[allow(deprecated)] // FetchService is a deprecated re-export. async fn get_mempool_info_fetch() { use hex::ToHex as _; use zaino_state::ChainIndex as _; - let (mut test_manager, _transparent_txid, _unified_txid) = - fund_and_fill_mempool::().await; + let (mut test_manager, _transparent_txid, _unified_txid) = fund_and_fill_mempool::().await; let subscriber = test_manager.subscriber(); let info = subscriber.get_mempool_info().await.unwrap(); @@ -2271,7 +2222,7 @@ async fn get_mempool_info_state() { let mut svc = fund_and_fill_mempool_dual().await; let info = svc.state_subscriber.get_mempool_info().await.unwrap(); - let entries = svc.state_subscriber.mempool.get_mempool().await; + let entries = svc.state_subscriber.mempool().get_mempool().await; assert_eq!(entries.len() as u64, info.size); assert!(info.size >= 1); @@ -2292,45 +2243,42 @@ async fn get_mempool_info_state() { } mod zebrad { - // FetchService is a deprecated re-export; the deprecation fires at the - // turbofish use sites below, so the allow covers the whole module. - #[allow(deprecated)] mod fetch_service { - use zaino_state::FetchService; + use zaino_testutils::Rpc; #[tokio::test(flavor = "multi_thread")] async fn receives_mining_reward() { - crate::receives_mining_reward::().await; + crate::receives_mining_reward::().await; } #[tokio::test(flavor = "multi_thread")] async fn connect_to_node_get_info() { - crate::connect_to_node_get_info::().await; + crate::connect_to_node_get_info::().await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_ironwood() { - crate::send_to_pool::(e2e::Pool::Ironwood).await; + crate::send_to_pool::(e2e::Pool::Ironwood).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_sapling() { - crate::send_to_pool::(e2e::Pool::Sapling).await; + crate::send_to_pool::(e2e::Pool::Sapling).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_transparent() { - crate::send_to_pool::(e2e::Pool::Transparent).await; + crate::send_to_pool::(e2e::Pool::Transparent).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_all() { - crate::send_to_all::().await; + crate::send_to_all::().await; } #[tokio::test(flavor = "multi_thread")] async fn shield_for_validator() { - crate::shield_for_validator::().await; + crate::shield_for_validator::().await; } #[tokio::test(flavor = "multi_thread")] @@ -2339,27 +2287,27 @@ mod zebrad { ignore = "heavy: seam-deep orchard advance (~100 halo2 proofs); un-ignore + transparent filler when cheap filler mining lands" )] async fn send_to_transparent_finalization() { - crate::send_to_transparent_finalization::().await; + crate::send_to_transparent_finalization::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_transaction_mined() { - crate::get_transaction_mined::().await; + crate::get_transaction_mined::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_raw_mempool() { - crate::get_raw_mempool::().await; + crate::get_raw_mempool::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_mempool_tx() { - crate::get_mempool_tx::().await; + crate::get_mempool_tx::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_mempool_stream() { - crate::get_mempool_stream::().await; + crate::get_mempool_stream::().await; } #[tokio::test(flavor = "multi_thread")] @@ -2373,62 +2321,62 @@ mod zebrad { ignore = "devtool WalletBalance has no unconfirmed_*/confirmed_* fields; balance asserts are commented out — restore + un-ignore when devtool surfaces unconfirmed balances" )] async fn monitor_unverified_mempool() { - crate::monitor_unverified_mempool::().await; + crate::monitor_unverified_mempool::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_address_tx_ids() { - crate::get_address_tx_ids::().await; + crate::get_address_tx_ids::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_address_utxos() { - crate::get_address_utxos::().await; + crate::get_address_utxos::().await; } #[tokio::test(flavor = "multi_thread")] async fn z_get_treestate() { - crate::z_get_treestate::().await; + crate::z_get_treestate::().await; } #[tokio::test(flavor = "multi_thread")] async fn z_get_subtrees_by_index() { - crate::z_get_subtrees_by_index::().await; + crate::z_get_subtrees_by_index::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_raw_transaction() { - crate::get_raw_transaction::().await; + crate::get_raw_transaction::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_taddress_txids() { - crate::get_taddress_txids::().await; + crate::get_taddress_txids::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_taddress_utxos() { - crate::get_taddress_utxos::().await; + crate::get_taddress_utxos::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_taddress_utxos_stream() { - crate::get_taddress_utxos_stream::().await; + crate::get_taddress_utxos_stream::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_transaction_mempool() { - crate::get_transaction_mempool::().await; + crate::get_transaction_mempool::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_address_balance() { - crate::get_address_balance::().await; + crate::get_address_balance::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_taddress_balance() { - crate::get_taddress_balance::().await; + crate::get_taddress_balance::().await; } } @@ -2548,42 +2496,41 @@ mod zebrad { } mod state_service { - #[allow(deprecated)] - use zaino_state::StateService; + use zaino_testutils::Direct; #[tokio::test(flavor = "multi_thread")] async fn receives_mining_reward() { - crate::receives_mining_reward::().await; + crate::receives_mining_reward::().await; } #[tokio::test(flavor = "multi_thread")] async fn connect_to_node_get_info() { - crate::connect_to_node_get_info::().await; + crate::connect_to_node_get_info::().await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_ironwood() { - crate::send_to_pool::(e2e::Pool::Ironwood).await; + crate::send_to_pool::(e2e::Pool::Ironwood).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_sapling() { - crate::send_to_pool::(e2e::Pool::Sapling).await; + crate::send_to_pool::(e2e::Pool::Sapling).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_transparent() { - crate::send_to_pool::(e2e::Pool::Transparent).await; + crate::send_to_pool::(e2e::Pool::Transparent).await; } #[tokio::test(flavor = "multi_thread")] async fn send_to_all() { - crate::send_to_all::().await; + crate::send_to_all::().await; } #[tokio::test(flavor = "multi_thread")] async fn shield_for_validator() { - crate::shield_for_validator::().await; + crate::shield_for_validator::().await; } #[tokio::test(flavor = "multi_thread")] @@ -2592,17 +2539,17 @@ mod zebrad { ignore = "heavy: seam-deep orchard advance (~100 halo2 proofs); un-ignore + transparent filler when cheap filler mining lands" )] async fn send_to_transparent_finalization() { - crate::send_to_transparent_finalization::().await; + crate::send_to_transparent_finalization::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_transaction_mined() { - crate::get_transaction_mined::().await; + crate::get_transaction_mined::().await; } #[tokio::test(flavor = "multi_thread")] async fn get_raw_mempool() { - crate::get_raw_mempool::().await; + crate::get_raw_mempool::().await; } #[tokio::test(flavor = "multi_thread")] @@ -2616,7 +2563,7 @@ mod zebrad { ignore = "devtool WalletBalance has no unconfirmed_*/confirmed_* fields; balance asserts are commented out — restore + un-ignore when devtool surfaces unconfirmed balances" )] async fn monitor_unverified_mempool() { - crate::monitor_unverified_mempool::().await; + crate::monitor_unverified_mempool::().await; } // No get_mempool_tx here: the state backend returns mempool-tx txids diff --git a/live-tests/e2e/tests/devtool_zcashd.rs b/live-tests/e2e/tests/devtool_zcashd.rs index 4405ff344..5950d0806 100644 --- a/live-tests/e2e/tests/devtool_zcashd.rs +++ b/live-tests/e2e/tests/devtool_zcashd.rs @@ -22,11 +22,9 @@ // launchers, all gated behind `zcashd_support`. Gate the whole binary so it // compiles out under `--no-default-features` (mirrors the clientless partition's json_server.rs). #![cfg(feature = "zcashd_support")] -#![allow(deprecated)] // FetchService is a deprecated re-export. - use e2e::devtool::DevtoolClients; -use zaino_state::{ChainIndex, FetchService, ZcashIndexer}; -use zaino_testutils::{TestManager, ValidatorKind, ZcashdDualFetchServices}; +use zaino_state::{ChainIndex, ZcashIndexer}; +use zaino_testutils::{Rpc, TestManager, ValidatorKind, ZcashdDualFetchServices}; use zcash_local_net::validator::zcashd::Zcashd; use zebra_chain::subtree::NoteCommitmentSubtreeIndex; use zebra_rpc::client::GetAddressBalanceRequest; @@ -38,13 +36,13 @@ use zebra_rpc::methods::GetAddressTxIdsRequest; /// faucet/recipient wallets against the resulting Zaino, without mining or /// syncing. The zcashd analogue of devtool.rs's `launch_and_build_clients`, /// concrete on zcashd (which has no StateService backend). -async fn launch_zcashd_and_build_clients() -> (TestManager, DevtoolClients) { - let test_manager = TestManager::::launch_mining_to( +async fn launch_zcashd_and_build_clients() -> (TestManager, DevtoolClients) { + let test_manager = TestManager::::launch_mining_to( zaino_testutils::SHIELDED_FUNDING_POOL, // ORCHARD &ValidatorKind::Zcashd, None, // network -> Regtest // The heights the devtool wallet accepts (same as the zebrad path). - Some(zaino_common::network::ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS), + Some(zaino_testutils::ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS), None, // no chain cache: build fresh at these heights true, // enable zaino false, // no json-rpc server @@ -58,6 +56,7 @@ async fn launch_zcashd_and_build_clients() -> (TestManager .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &test_manager.local_net, ) .await; @@ -70,7 +69,7 @@ async fn launch_zcashd_and_build_clients() -> (TestManager /// height 2). The send/shield analogue of devtool.rs's `launch_and_fund_faucet`. async fn launch_and_fund_zcashd_faucet( orchard_notes: u32, -) -> (TestManager, DevtoolClients) { +) -> (TestManager, DevtoolClients) { let (test_manager, mut clients) = launch_zcashd_and_build_clients().await; test_manager .generate_blocks_and_wait_for_tip(orchard_notes + 1, test_manager.subscriber()) @@ -110,7 +109,7 @@ async fn faucet_receives_zcashd_orchard_reward() { /// of json_server's `create_zcashd_test_manager_and_fetch_services`. async fn create_zcashd_devtool_services() -> (ZcashdDualFetchServices, DevtoolClients) { let services = zaino_testutils::launch_zcashd_dual_fetch_services_at( - zaino_common::network::ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS, + zaino_testutils::ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS, ) .await; let clients = e2e::devtool::build_clients( diff --git a/live-tests/e2e/tests/ironwood_activation.rs b/live-tests/e2e/tests/ironwood_activation.rs new file mode 100644 index 000000000..a5db94ea1 --- /dev/null +++ b/live-tests/e2e/tests/ironwood_activation.rs @@ -0,0 +1,482 @@ +//! Wallet-tier predicates across the Orchard→Ironwood activation boundary. +//! +//! Every test here runs a devtool wallet on +//! [`ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS`] — the hermetic replay of what +//! The Public Testnet did once at height 4,134,000: heights 2 through 5 are +//! Orchard era, [`NU6_3_TRANSITION_BOUNDARY`] (6) onward is Ironwood era. +//! The wallets derive their activation schedule from the running validator +//! (`WalletNetwork::from_validator`, infrastructure ADR 0003), so the +//! fixture heights are typed in exactly one place: the zebrad launch +//! config. Height drift between wallet, indexer, and validator is +//! unrepresentable — zainod adopts the same schedule over +//! `getblockchaininfo` (zaino#1076). +//! +//! # The predicates, and where each era's cell is covered +//! +//! | predicate (wallet-observable) | Orchard era | Ironwood era | +//! |----------------------------------------------|-------------|--------------| +//! | unified-address receipt lands in Orchard | here | false — `devtool.rs` `send_to_ironwood` asserts the Orchard pool stays empty | +//! | unified-address receipt lands in Ironwood | false (pool inactive) | `devtool.rs` `send_to_ironwood` | +//! | shielded-receiver coinbase pays the era pool | here (wallet view); wire tier in `compact_block_wire.rs` | `devtool.rs` `receives_mining_reward`; wire tier in `compact_block_wire.rs` | +//! | an Orchard note spends into an Ironwood receipt (ZIP 318 migration) | n/a (nothing to exit) | here | +//! | `shield` deposits into the era pool | here | `devtool.rs` `shield_for_validator` | +//! | receipt pool flips between the last Orchard block and the activation block | here (boundary − 1) | here (exactly at the boundary) | +//! | Orchard pool value grows only below the boundary; Ironwood holds value only from it | here (per-height value pools) | here | +//! +//! Era composition of the *served* chain (coinbase routing, compact-block +//! action fields) is covered clientless in +//! `clientless/tests/compact_block_consistency.rs` and over the real gRPC +//! wire in `compact_block_wire.rs`; this file owns the cells that need a +//! wallet on both sides of the boundary. +//! +//! The Public Testnet cannot host the migration cell for us: its pre-NU6.3 +//! epoch closed at height 4,134,000, no new value may enter Orchard from +//! there (post-activation Orchard actions permit only same-receiver change +//! or withdrawal — the cross-address restriction, +//! ), +//! and we hold no pre-activation Orchard TAZ — so this hermetic fixture is +//! the only controlled venue for it. +//! +//! Deferred cells the cross-address restriction implies (chain-walk tier, +//! not wallet tier): the Orchard pool value is non-increasing from the +//! boundary, and post-activation Orchard commitments exist only as +//! same-receiver change — note the Orchard note-commitment tree therefore +//! still grows after activation; do not encode a frozen-finalRoot predicate. +//! +//! Requires a `zcash-devtool` binary built with `--features regtest_support` +//! in `TEST_BINARIES_DIR`/`PATH`, alongside the usual validator binaries. + +use e2e::devtool::DevtoolClients; +use zaino_state::ZcashIndexer; +use zaino_testutils::{ + all_pools_i32, collect_block_range, PollableTip, TestManager, ValidatorKind, + NU6_3_TRANSITION_BOUNDARY, ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS, +}; +use zainodlib::error::IndexerError; +use zcash_local_net::validator::zebrad::Zebrad; + +/// Launch an orchard-receiver-mining zebrad + Zaino on the transition +/// heights, build devtool faucet/recipient wallets (their schedule derived +/// from the launched validator), mine one block, and sync the faucet. The +/// launch itself already leaves the tip at 2 (`TestManager` mines an +/// NU-activation block after block 1), so the helper returns at tip 3 with +/// the faucet holding the Orchard coinbase notes of heights 2 and 3. Tests +/// that need exact boundary positioning mine to absolute heights from the +/// observed tip rather than counting from here. The transition-fixture +/// analogue of `devtool.rs::launch_and_fund_faucet`. +async fn launch_transition_chain_and_fund_faucet( +) -> (TestManager, DevtoolClients) +where + Conn: zaino_testutils::ValidatorConnectionMarker, +{ + let test_manager = TestManager::::launch_mining_to( + zaino_testutils::SHIELDED_FUNDING_POOL, + &ValidatorKind::Zebrad, + None, + Some(ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS), + None, + true, + false, + false, + ) + .await + .expect("launch TestManager"); + + let mut clients = e2e::devtool::build_clients( + test_manager + .zaino_grpc_listen_address + .expect("zaino enabled") + .port(), + &test_manager.local_net, + ) + .await; + + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_faucet().await; + + (test_manager, clients) +} + +/// The chain value (in zatoshis) of the pool named `pool_id` as of +/// `height`, read from the served verbosity-1 block object — the same +/// per-height `valuePools` a zcashd `getblock` reports. Verbosity 1, not 2: +/// the fetch backend cannot deserialize a verbosity-2 block object (its +/// `tx` entries are maps where zaino-fetch expects txid strings), and the +/// value pools ride along at verbosity 1. +async fn pool_zats_at_height(subscriber: &S, height: u32, pool_id: &str) -> i64 +where + S: ZcashIndexer, + IndexerError: From, +{ + let response = subscriber + .z_get_block(height.to_string(), Some(1)) + .await + .map_err(IndexerError::from) + .expect("z_get_block verbosity 1"); + let zebra_rpc::methods::GetBlock::Object(block) = response else { + panic!("verbosity-1 getblock must return a block object"); + }; + let pools = block + .value_pools() + .as_ref() + .expect("verbosity-1 block object carries value pools"); + pools + .iter() + .find(|pool| pool.id() == pool_id) + .unwrap_or_else(|| panic!("value pools must include {pool_id}")) + .chain_value_zat() + .zatoshis() +} + +/// Orchard-era receipt: with the tip still below the boundary, the faucet's +/// coinbase note is an Orchard note (not Ironwood), and a unified-address +/// send received before the boundary lands in the recipient's Orchard pool +/// with the Ironwood pool exactly empty — the era-mirror of +/// `devtool.rs::send_to_pool(Ironwood)`. +async fn unified_receipt_lands_in_orchard_before_boundary() +where + Conn: zaino_testutils::ValidatorConnectionMarker, +{ + let (mut test_manager, mut clients) = launch_transition_chain_and_fund_faucet::().await; + + // Tip is 3: inside the Orchard era, with room to confirm the send at + // height 4 while staying below the boundary at 6. + let faucet_balance = clients.faucet_balance().await; + assert!( + faucet_balance.orchard_spendable > 0, + "pre-boundary coinbase should be an orchard note, got {faucet_balance:?}" + ); + assert_eq!( + faucet_balance.ironwood_spendable, 0, + "no ironwood note can exist below the boundary, got {faucet_balance:?}" + ); + + let recipient = clients.get_recipient_address("unified").await; + let txid = clients.send_from_faucet(&recipient, 250_000).await; + dbg!(txid); + + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_recipient().await; + + let balance = clients.recipient_balance().await; + assert_eq!(e2e::Pool::Orchard.spendable_balance(&balance), 250_000); + assert_eq!(e2e::Pool::Ironwood.spendable_balance(&balance), 0); + + test_manager.close().await; +} + +/// The ZIP 318 migration shape: an Orchard note minted before the boundary +/// is spent after it, to a unified address generated after activation, and +/// the receipt lands in the Ironwood pool with the recipient's Orchard pool +/// exactly empty. The faucet's Orchard balance must shrink: from the +/// boundary, the cross-address restriction limits each Orchard action to +/// same-receiver change or withdrawal +/// (), +/// so a genuine Orchard spend nets sent-amount-plus-fee out of the pool even +/// when change returns to the spent note's address. +async fn orchard_note_spends_to_ironwood_across_boundary() +where + Conn: zaino_testutils::ValidatorConnectionMarker, +{ + let (mut test_manager, mut clients) = launch_transition_chain_and_fund_faucet::().await; + + let pre_boundary_balance = clients.faucet_balance().await; + assert!( + pre_boundary_balance.orchard_spendable > 0, + "pre-boundary coinbase should be an orchard note, got {pre_boundary_balance:?}" + ); + + // Mine to the boundary itself, from the observed tip rather than a + // hand-count. The blocks below the boundary add more Orchard coinbase + // notes; the boundary block is the first Ironwood-era block, and its + // coinbase is the faucet's first Ironwood note. + let tip = u32::try_from(test_manager.subscriber().tip_height().await) + .expect("regtest tips fit in u32"); + test_manager + .generate_blocks_and_wait_for_tip( + NU6_3_TRANSITION_BOUNDARY + .checked_sub(tip) + .expect("the launch preamble must leave room below the boundary"), + test_manager.subscriber(), + ) + .await; + clients.sync_faucet().await; + let crossed_balance = clients.faucet_balance().await; + let orchard_before_send = crossed_balance.orchard_spendable; + assert!( + crossed_balance.ironwood_spendable > 0, + "the boundary coinbase should be an ironwood note, got {crossed_balance:?}" + ); + + // Generated only now — after activation — per the migration shape under + // test: old-pool note, new-era address. + let recipient = clients.get_recipient_address("unified").await; + let txid = clients.send_from_faucet(&recipient, 250_000).await; + dbg!(txid); + + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_faucet().await; + clients.sync_recipient().await; + + let balance = clients.recipient_balance().await; + assert_eq!(e2e::Pool::Ironwood.spendable_balance(&balance), 250_000); + assert_eq!(e2e::Pool::Orchard.spendable_balance(&balance), 0); + + // Pins that the send actually exited the Orchard pool rather than + // spending the boundary-height Ironwood coinbase — the note-selection + // question the first live runs of this suite exist to answer. + assert!( + clients.faucet_balance().await.orchard_spendable < orchard_before_send, + "the migration send must spend an orchard note" + ); + + // Chain-tier consequences, read from the served per-height value pools. + // The Ironwood pool holds no value below the activation height; the + // Orchard pool grows only below it (its coinbases), holds exactly + // steady across the boundary edge (the activation block's coinbase pays + // Ironwood, and no new value may enter Orchard), and shrinks at the + // migration block by the withdrawn amount plus fee. + let boundary = NU6_3_TRANSITION_BOUNDARY; + let migration_height = boundary + 1; + let subscriber = test_manager.subscriber(); + let mut orchard_at = Vec::new(); + for height in 1..=migration_height { + let ironwood = pool_zats_at_height(subscriber, height, "ironwood").await; + let orchard = pool_zats_at_height(subscriber, height, "orchard").await; + if height < boundary { + assert_eq!( + ironwood, 0, + "the ironwood pool must hold no value at height {height}, below the boundary" + ); + } + orchard_at.push(orchard); + } + for (index, orchard) in orchard_at.iter().enumerate() { + let height = index + 1; + let ironwood = pool_zats_at_height(subscriber, height as u32, "ironwood").await; + eprintln!("pool values at height {height}: orchard={orchard} ironwood={ironwood}"); + } + let orchard = |height: u32| orchard_at[height as usize - 1]; + for height in 2..boundary { + assert!( + orchard(height) > orchard(height - 1), + "each pre-boundary coinbase must grow the orchard pool (height {height}: {} after {})", + orchard(height), + orchard(height - 1) + ); + } + assert_eq!( + orchard(boundary), + orchard(boundary - 1), + "the orchard pool must hold exactly steady across the boundary edge" + ); + assert!( + pool_zats_at_height(subscriber, boundary, "ironwood").await > 0, + "the activation block's coinbase must give the ironwood pool its first value" + ); + assert!( + orchard(migration_height) < orchard(boundary), + "the migration block must shrink the orchard pool ({} at the boundary, {} at the migration block)", + orchard(boundary), + orchard(migration_height) + ); + + test_manager.close().await; +} + +/// The receipt pool flips between adjacent blocks: a send confirmed in the +/// last Orchard-era block (boundary − 1) lands in Orchard, and a send built +/// one block below the boundary but confirmed in the activation block +/// itself lands in Ironwood. The second send also pins the wallet's era +/// anticipation at the edge: it is constructed while the tip is still +/// Orchard-era, targeting the first Ironwood-era height, and it spends an +/// Orchard note (the faucet holds no Ironwood note until the activation +/// block is mined) — a migration transaction in the activation block. +async fn receipts_flip_pools_exactly_at_the_boundary() +where + Conn: zaino_testutils::ValidatorConnectionMarker, +{ + let (mut test_manager, mut clients) = launch_transition_chain_and_fund_faucet::().await; + + // Position the tip at exactly boundary − 2, from the observed tip + // rather than a hand-count, so the two sends below confirm at exactly + // boundary − 1 and the boundary. + let tip = u32::try_from(test_manager.subscriber().tip_height().await) + .expect("regtest tips fit in u32"); + test_manager + .generate_blocks_and_wait_for_tip( + NU6_3_TRANSITION_BOUNDARY + .checked_sub(2 + tip) + .expect("the launch preamble must leave room below the boundary"), + test_manager.subscriber(), + ) + .await; + clients.sync_faucet().await; + + let recipient = clients.get_recipient_address("unified").await; + + // Confirmed in block 5: the last Orchard-era block. + let last_orchard_txid = clients.send_from_faucet(&recipient, 250_000).await; + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_faucet().await; + clients.sync_recipient().await; + let balance = clients.recipient_balance().await; + assert_eq!( + e2e::Pool::Orchard.spendable_balance(&balance), + 250_000, + "receipt confirmed at boundary - 1 must be orchard, got {balance:?}" + ); + assert_eq!( + e2e::Pool::Ironwood.spendable_balance(&balance), + 0, + "no ironwood receipt below the boundary, got {balance:?}" + ); + + // Built at tip boundary − 1, confirmed in block 6: the activation block. + let first_ironwood_txid = clients.send_from_faucet(&recipient, 250_000).await; + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_recipient().await; + let balance = clients.recipient_balance().await; + assert_eq!(e2e::Pool::Ironwood.spendable_balance(&balance), 250_000); + assert_eq!( + e2e::Pool::Orchard.spendable_balance(&balance), + 250_000, + "the pre-boundary receipt must survive the flip unchanged" + ); + + // Era composition of the two served edge blocks. + let subscriber = test_manager.subscriber(); + let boundary = u64::from(NU6_3_TRANSITION_BOUNDARY); + let blocks = collect_block_range(subscriber, boundary - 1, boundary, all_pools_i32()).await; + let [last_orchard_block, activation_block] = blocks.as_slice() else { + panic!("expected exactly the two edge blocks, got {}", blocks.len()); + }; + assert_eq!(last_orchard_block.height, boundary - 1); + assert_eq!(activation_block.height, boundary); + let last_orchard_txid = e2e::devtool::txid_from_devtool(&last_orchard_txid); + e2e::assert_pool_present(last_orchard_block, &last_orchard_txid, e2e::Pool::Orchard); + e2e::assert_pool_absent(last_orchard_block, &last_orchard_txid, e2e::Pool::Ironwood); + let first_ironwood_txid = e2e::devtool::txid_from_devtool(&first_ironwood_txid); + e2e::assert_pool_present(activation_block, &first_ironwood_txid, e2e::Pool::Ironwood); + // The second send is itself migration-shaped: built one block below the + // boundary, it spends an Orchard note (the faucet's only spendable kind + // at that tip), so its compact form carries the Orchard spend's data + // alongside the Ironwood receipt. The receipt's pool routing is what + // flips at the boundary, and that is asserted at the wallet tier above. + e2e::assert_pool_present(activation_block, &first_ironwood_txid, e2e::Pool::Orchard); + + test_manager.close().await; +} + +/// Below the boundary, `shield` deposits into the Orchard pool: the faucet +/// funds the recipient's transparent address, the recipient shields, and +/// the shielded balance (net of the ZIP-317 fee, mirroring +/// `devtool.rs::shield_for_validator`) lands in Orchard with the Ironwood +/// pool exactly empty — the era-mirror of the Ironwood-era shield cell. +async fn shield_deposits_to_orchard_before_boundary() +where + Conn: zaino_testutils::ValidatorConnectionMarker, +{ + let (mut test_manager, mut clients) = launch_transition_chain_and_fund_faucet::().await; + + // Tip is 3; the transparent receipt confirms at 4 and the shield at 5, + // all below the boundary at 6. + let recipient_t = clients.get_recipient_address("transparent").await; + clients.send_from_faucet(&recipient_t, 250_000).await; + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_recipient().await; + assert_eq!( + e2e::Pool::Transparent.spendable_balance(&clients.recipient_balance().await), + 250_000 + ); + + clients.shield_recipient().await; + test_manager + .generate_blocks_and_wait_for_tip(1, test_manager.subscriber()) + .await; + clients.sync_recipient().await; + + let balance = clients.recipient_balance().await; + assert_eq!(e2e::Pool::Orchard.spendable_balance(&balance), 235_000); + assert_eq!(e2e::Pool::Ironwood.spendable_balance(&balance), 0); + + test_manager.close().await; +} + +mod zebrad { + mod fetch_service { + use zaino_testutils::Rpc; + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn unified_receipt_lands_in_orchard_before_boundary() { + crate::unified_receipt_lands_in_orchard_before_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn orchard_note_spends_to_ironwood_across_boundary() { + crate::orchard_note_spends_to_ironwood_across_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn receipts_flip_pools_exactly_at_the_boundary() { + crate::receipts_flip_pools_exactly_at_the_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn shield_deposits_to_orchard_before_boundary() { + crate::shield_deposits_to_orchard_before_boundary::().await; + } + } + + mod state_service { + use zaino_testutils::Direct; + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn unified_receipt_lands_in_orchard_before_boundary() { + crate::unified_receipt_lands_in_orchard_before_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn orchard_note_spends_to_ironwood_across_boundary() { + crate::orchard_note_spends_to_ironwood_across_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn receipts_flip_pools_exactly_at_the_boundary() { + crate::receipts_flip_pools_exactly_at_the_boundary::().await; + } + + /// multi_thread required: the test manager spawns the validator and + /// indexer services. + #[tokio::test(flavor = "multi_thread")] + async fn shield_deposits_to_orchard_before_boundary() { + crate::shield_deposits_to_orchard_before_boundary::().await; + } + } +} diff --git a/live-tests/e2e/tests/test_vectors.rs b/live-tests/e2e/tests/test_vectors.rs index aeee240f4..ec18a2ad3 100644 --- a/live-tests/e2e/tests/test_vectors.rs +++ b/live-tests/e2e/tests/test_vectors.rs @@ -14,11 +14,9 @@ use zaino_state::read_u64_le; use zaino_state::write_u32_le; use zaino_state::write_u64_le; use zaino_state::CompactSize; -#[allow(deprecated)] -use zaino_state::StateService; use zaino_state::ZcashIndexer; use zaino_state::{ChainWork, IndexedBlock}; -use zaino_testutils::{TestManager, ValidatorKind}; +use zaino_testutils::{Direct, TestManager, ValidatorKind}; use zcash_local_net::validator::zebrad::Zebrad; use zebra_chain::serialization::{ZcashDeserialize, ZcashSerialize}; use zebra_rpc::methods::GetAddressUtxos; @@ -47,7 +45,7 @@ async fn create_200_block_regtest_chain_vectors() { // The committed unit-test vectors encode a mixed-pool chain built by // repeatedly shielding transparent coinbase — mining must stay transparent // so regenerated vectors keep that shape. - let mut test_manager = TestManager::::launch_mining_to( + let mut test_manager = TestManager::::launch_mining_to( zaino_testutils::MinerPool::Transparent, &ValidatorKind::Zebrad, None, @@ -67,6 +65,7 @@ async fn create_200_block_regtest_chain_vectors() { .zaino_grpc_listen_address .expect("zaino enabled") .port(), + &test_manager.local_net, ) .await; @@ -243,7 +242,7 @@ async fn create_200_block_regtest_chain_vectors() { .await .and_then(|response| match response { zebra_rpc::methods::GetBlock::Raw(_) => { - Err(zaino_state::StateServiceError::Custom( + Err(zaino_state::NodeBackedIndexerServiceError::Custom( "Found transaction of `Raw` type, expected only `Object` types." .to_string(), )) @@ -255,7 +254,7 @@ async fn create_200_block_regtest_chain_vectors() { match item { GetBlockTransaction::Hash(h) => Ok(h.0.to_vec()), GetBlockTransaction::Object(_) => Err( - zaino_state::StateServiceError::Custom( + zaino_state::NodeBackedIndexerServiceError::Custom( "Found transaction of `Object` type, expected only `Hash` types." .to_string(), ), @@ -274,7 +273,7 @@ async fn create_200_block_regtest_chain_vectors() { .await .and_then(|response| match response { zebra_rpc::methods::GetBlock::Object { .. } => { - Err(zaino_state::StateServiceError::Custom( + Err(zaino_state::NodeBackedIndexerServiceError::Custom( "Found transaction of `Object` type, expected only `Raw` types." .to_string(), )) @@ -283,7 +282,7 @@ async fn create_200_block_regtest_chain_vectors() { }) .unwrap(); - let mut state = state_service_subscriber.read_state_service.clone(); + let mut state = state_service_subscriber.read_state_service(); let (sapling_root, orchard_root) = { let (sapling_tree_response, orchard_tree_response) = futures::future::join( state.clone().call(zebra_state::ReadRequest::SaplingTree( @@ -319,7 +318,7 @@ async fn create_200_block_regtest_chain_vectors() { }; let sapling_treestate = match zebra_chain::parameters::NetworkUpgrade::Sapling - .activation_height(&state_service_subscriber.network().to_zebra_network()) + .activation_height(&state_service_subscriber.network()) { Some(activation_height) if height >= activation_height.0 => Some( state @@ -342,7 +341,7 @@ async fn create_200_block_regtest_chain_vectors() { }) .unwrap(); let orchard_treestate = match zebra_chain::parameters::NetworkUpgrade::Nu5 - .activation_height(&state_service_subscriber.network().to_zebra_network()) + .activation_height(&state_service_subscriber.network()) { Some(activation_height) if height >= activation_height.0 => Some( state diff --git a/live-tests/test_environment/Containerfile b/live-tests/test_environment/Containerfile index dabb4dea2..a6a19c7fd 100644 --- a/live-tests/test_environment/Containerfile +++ b/live-tests/test_environment/Containerfile @@ -46,7 +46,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ clang=1:19.0-63 \ cmake=3.31.6-2 \ pkg-config=1.8.1-4 \ - protobuf-compiler=3.21.12-11 \ + protobuf-compiler=3.21.12-11+deb13u1 \ libstdc++6=14.2.0-19 WORKDIR /zebra # Single RUN (DL3059): clone, checkout, build, and stage the binary. @@ -71,7 +71,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ autoconf=2.72-3.1 \ libtool=2.5.4-4 \ bsdmainutils=12.1.8 \ - curl=8.14.1-2+deb13u3 \ + curl=8.14.1-2+deb13u4 \ ca-certificates=20250419 \ netbase=6.5 @@ -99,7 +99,7 @@ ARG DEVTOOL_REV RUN apt-get update && apt-get install -y --no-install-recommends \ git=1:2.47.3-0+deb13u1 \ pkg-config=1.8.1-4 \ - protobuf-compiler=3.21.12-11 \ + protobuf-compiler=3.21.12-11+deb13u1 \ libsqlite3-dev=3.46.1-7+deb13u1 WORKDIR /devtool # Single RUN (DL3059): clone, checkout, build, and stage the binary. The @@ -140,12 +140,12 @@ LABEL org.zaino.test-artifacts.versions.rust="${RUST_VERSION}" # Versions pinned (DL3008) to the candidates in rust:1.95.0-trixie; bump # together with the base image (query with `apt-cache policy `). RUN apt-get update && apt-get install -y --no-install-recommends \ - curl=8.14.1-2+deb13u3 \ + curl=8.14.1-2+deb13u4 \ libclang-dev=1:19.0-63 \ build-essential=12.12 \ cmake=3.31.6-2 \ librocksdb-dev=9.10.0-1+b1 \ - protobuf-compiler=3.21.12-11 \ + protobuf-compiler=3.21.12-11+deb13u1 \ && rm -rf /var/lib/apt/lists/* # Create container_user. The GID may already belong to a base-image group diff --git a/live-tests/zaino-testutils/Cargo.toml b/live-tests/zaino-testutils/Cargo.toml index 38f2ef883..09b132e9f 100644 --- a/live-tests/zaino-testutils/Cargo.toml +++ b/live-tests/zaino-testutils/Cargo.toml @@ -65,5 +65,7 @@ futures = { workspace = true } http = { workspace = true } once_cell = { workspace = true } tokio = { workspace = true } -tonic.workspace = true tracing.workspace = true + +[dev-dependencies] +tonic.workspace = true diff --git a/live-tests/zaino-testutils/src/lib.rs b/live-tests/zaino-testutils/src/lib.rs index 9fcd5649b..78ecf274c 100644 --- a/live-tests/zaino-testutils/src/lib.rs +++ b/live-tests/zaino-testutils/src/lib.rs @@ -7,6 +7,7 @@ use futures::StreamExt as _; use once_cell::sync::Lazy; use std::{ future::Future, + marker::PhantomData, net::{IpAddr, Ipv4Addr, SocketAddr}, path::PathBuf, }; @@ -14,7 +15,7 @@ use std::{ use tonic::transport::Channel; use tracing::{debug, info, instrument}; use zaino_common::{ - network::{ActivationHeights, ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS}, + network::ActivationHeights, probing::{Liveness, Readiness}, status::Status, validator::ValidatorConfig, @@ -24,15 +25,12 @@ use zaino_fetch::jsonrpsee::connector::{test_node_and_return_url, JsonRpSeeConne use zaino_proto::proto::compact_formats::CompactBlock; use zaino_proto::proto::service::{BlockId, BlockRange}; use zaino_serve::server::config::{GrpcServerConfig, JsonRpcServerConfig}; -#[cfg(feature = "zcashd_support")] -use zaino_state::BackendType; use zaino_state::{ - BlockchainSource, ChainIndex, FetchServiceSubscriber, LightWalletIndexer, LightWalletService, - NodeBackedChainIndexSubscriber, StateServiceSubscriber, ZcashIndexer, ZcashService, + BlockchainSource, ChainIndex, LightWalletIndexer, NodeBackedChainIndexSubscriber, + NodeBackedIndexerService, NodeBackedIndexerServiceConfig, NodeBackedIndexerServiceSubscriber, + ZcashService, }; -#[allow(deprecated)] -use zaino_state::{FetchService, FetchServiceConfig, StateService, StateServiceConfig}; -use zainodlib::{config::ZainodConfig, error::IndexerError, indexer::Indexer}; +use zainodlib::{config::BackendType, error::IndexerError, indexer::Indexer}; pub use zcash_local_net as services; use zcash_local_net::error::LaunchError; #[cfg(feature = "zcashd_support")] @@ -67,6 +65,29 @@ macro_rules! validator_tests { }; } +/// The canonical regtest activation heights the harness *launches* validators +/// with when a test names no others (everything through NU6.3 active from +/// height 2). This is validator-launch vocabulary — the configuring side of +/// the truth source — and is never zainod's own schedule: zainod's config +/// carries only a network kind, and both backends adopt the runtime schedule +/// from the running validator's `getblockchaininfo.upgrades` (zaino#1076). +/// It matches zcash_local_net's `supported_regtest_activation_heights`, which +/// the devtool wallet client currently requires (zaino#1368). +pub const ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS: ActivationHeights = ActivationHeights { + before_overwinter: Some(1), + overwinter: Some(1), + sapling: Some(1), + blossom: Some(1), + heartwood: Some(1), + canopy: Some(1), + nu5: Some(2), + nu6: Some(2), + nu6_1: Some(2), + nu6_2: Some(2), + nu6_3: Some(2), + nu7: None, +}; + /// Orchard-era-only fixture: every upgrade through NU6.2 active from height 2 and /// NU6.3 never activating, so coinbases stay Orchard for the whole chain. For /// client-free launches only — the zcash-devtool wallet hardcodes the canonical @@ -89,7 +110,7 @@ pub const ORCHARD_ONLY_ACTIVATION_HEIGHTS: ActivationHeights = ActivationHeights /// Zebrad regtest heights with every upgrade through NU6.3 active from height 2, so /// generated blocks carry V6 coinbases from the first post-genesis era. -pub const NU6_3_ACTIVE_ACTIVATION_HEIGHTS: ActivationHeights = ActivationHeights { +pub const IRONWOOD_ONLY_ACTIVATION_HEIGHTS: ActivationHeights = ActivationHeights { before_overwinter: Some(1), overwinter: Some(1), sapling: Some(1), @@ -285,20 +306,11 @@ pub trait PollableTip: Status + Sync { fn tip_height(&self) -> impl std::future::Future; } -impl PollableTip for FetchServiceSubscriber { - async fn tip_height(&self) -> u64 { - self.get_latest_block() - .await - .expect("PollableTip: FetchServiceSubscriber::get_latest_block failed") - .height - } -} - -impl PollableTip for StateServiceSubscriber { +impl PollableTip for NodeBackedIndexerServiceSubscriber { async fn tip_height(&self) -> u64 { self.get_latest_block() .await - .expect("PollableTip: StateServiceSubscriber::get_latest_block failed") + .expect("PollableTip: NodeBackedIndexerServiceSubscriber::get_latest_block failed") .height } } @@ -318,39 +330,32 @@ impl PollableTip for NodeBackedChainIndexSubscriber` -/// - `Service::Subscriber: PollableTip` -/// -/// **Not** bundled: the reverse bound -/// `IndexerError: From`. Rust does not -/// propagate non-`Self` bounds declared in a trait's `where`-clause -/// through a `T: TestService` constraint, so call sites that touch -/// `TestManager::launch` (or anything else exercising -/// `Indexer::launch_inner`'s `?` propagation) must still restate that -/// one bound explicitly. Everything else collapses to `TestService`. -pub trait TestService: - LightWalletService, Subscriber: PollableTip> - + Send - + Sync - + 'static -{ +/// Replaces the former `FetchService` / `StateService` `Service` type parameters: +/// there is now one service, parameterized by connection. Implementors are the +/// zero-sized markers [`Rpc`] and [`Direct`]. +pub trait ValidatorConnectionMarker: Send + Sync + 'static { + /// The on-disk backend selector this connection maps to. + const BACKEND: BackendType; } -impl TestService for T where - T: LightWalletService< - Config: TryFrom, - Subscriber: PollableTip, - > + Send - + Sync - + 'static -{ +/// JSON-RPC validator connection (formerly `FetchService`). +#[derive(Debug, Clone, Copy)] +pub struct Rpc; + +impl ValidatorConnectionMarker for Rpc { + const BACKEND: BackendType = BackendType::Rpc; +} + +/// Direct Zebra `ReadStateService` validator connection (formerly `StateService`). +#[derive(Debug, Clone, Copy)] +pub struct Direct; + +impl ValidatorConnectionMarker for Direct { + const BACKEND: BackendType = BackendType::Direct; } // temporary until activation heights are unified to zebra-chain type. @@ -468,8 +473,8 @@ pub static ZEBRAD_CHAIN_CACHE_DIR: Lazy> = Lazy::new(|| { Some(workspace_root_path.join("live-tests/chain_cache/client_rpc_tests_large")) }); -/// Path for the Zebra chain cache in the user's home directory. -pub static ZEBRAD_TESTNET_CACHE_DIR: Lazy> = Lazy::new(|| { +/// Path for the Zebra chain cache of The Public Testnet in the user's home directory. +pub static ZEBRAD_THE_PUB_TESTNET_CACHE_DIR: Lazy> = Lazy::new(|| { let home_path = PathBuf::from(std::env::var("HOME").unwrap()); Some(home_path.join(".cache/zebra")) }); @@ -509,7 +514,11 @@ pub enum ValidatorTestConfig { } /// Configuration data for Zaino Tests. -pub struct TestManager { +/// +/// Generic over the validator control plane `C` and the [`ValidatorConnectionMarker`] +/// `Conn` — the validator connection ([`Rpc`] or [`Direct`]) the single +/// [`NodeBackedIndexerService`] uses. +pub struct TestManager { /// Control plane for a validator pub local_net: C, /// Data directory for the validator. @@ -527,9 +536,11 @@ pub struct TestManager, /// Service subscriber. - pub service_subscriber: Option, + pub service_subscriber: Option, /// JsonRPC server cookie dir. pub json_server_cookie_dir: Option, + /// Selected validator connection marker. + _connection: PhantomData, } /// Needed validator functionality that is not implemented in infrastructure @@ -613,19 +624,17 @@ pub fn default_mining_pool(validator: &ValidatorKind) -> MinerPool { } } -impl TestManager +impl TestManager where C: ValidatorExt, - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: ValidatorConnectionMarker, { /// Returns the service subscriber, panicking if zaino wasn't enabled at launch. /// /// Convenience for tests that always pass `enable_zaino: true` and want to /// hand the subscriber to [`Self::generate_blocks_and_wait_for_tip`] without /// repeating `.service_subscriber.as_ref().unwrap()` at every call site. - pub fn subscriber(&self) -> &Service::Subscriber { + pub fn subscriber(&self) -> &NodeBackedIndexerServiceSubscriber { self.service_subscriber .as_ref() .expect("TestManager::subscriber called but service_subscriber is None (zaino disabled at launch?)") @@ -709,9 +718,9 @@ where enable_clients: bool, ) -> Result { #[cfg(feature = "zcashd_support")] - if (validator == &ValidatorKind::Zcashd) && (Service::BACKEND_TYPE == BackendType::State) { + if (validator == &ValidatorKind::Zcashd) && (Conn::BACKEND == BackendType::Direct) { return Err(std::io::Error::other( - "Cannot use state backend with zcashd.", + "Cannot use the direct (ReadStateService) connection with zcashd.", )); } zaino_common::logging::try_init(); @@ -719,8 +728,17 @@ where let activation_heights = activation_heights.unwrap_or_else(|| validator.default_activation_heights()); let network_kind = network.unwrap_or(NetworkKind::Regtest); - let zaino_network_kind = - Network::from_network_kind_and_activation_heights(&network_kind, &activation_heights); + // The validator is the single source of truth for activation heights + // (zaino#1076). zainod's config is a payload-free network kind: the + // fixture heights configure the validator alone, and zainod adopts + // the real schedule from the validator at spawn. Every + // custom-heights launch is therefore a standing regression test of + // that adoption. + let zaino_network_kind = match network_kind { + NetworkKind::Mainnet => Network::Mainnet, + NetworkKind::Testnet => Network::PubTestnet, + NetworkKind::Regtest => Network::Regtest, + }; if enable_clients && !enable_zaino { return Err(std::io::Error::other( @@ -791,8 +809,7 @@ where "[TEST] Launching Zaino indexer" ); let indexer_config = zainodlib::config::ZainodConfig { - // TODO: Make configurable. - backend: Service::BACKEND_TYPE, + backend: Conn::BACKEND, json_server_settings: if enable_zaino_jsonrpc_server { Some(JsonRpcServerConfig { json_rpc_listen_address: zaino_json_listen_address, @@ -821,15 +838,16 @@ where metrics_endpoint: None, }; - let (handle, service_subscriber) = Indexer::::launch_inner_with_listeners( - Service::Config::try_from(indexer_config.clone()) - .expect("Failed to convert ZainodConfig to service config"), - indexer_config, - zaino_grpc_listener, - zaino_json_listener, - ) - .await - .unwrap(); + let (handle, service_subscriber) = + Indexer::::launch_inner_with_listeners( + NodeBackedIndexerServiceConfig::try_from(indexer_config.clone()) + .expect("Failed to convert ZainodConfig to service config"), + indexer_config, + zaino_grpc_listener, + zaino_json_listener, + ) + .await + .unwrap(); ( Some(handle), @@ -866,6 +884,7 @@ where zaino_grpc_listen_address, service_subscriber: zaino_service_subscriber, json_server_cookie_dir: zaino_json_server_cookie_dir, + _connection: PhantomData, }; test_manager.activate_nu5_nu6(enable_zaino).await; @@ -1052,25 +1071,27 @@ where } } -/// Handles from [`launch_state_and_fetch_services`]: a state-backend -/// [`TestManager`] plus standalone fetch and state services pointed at the -/// same validator. The owned services exist only to keep their subscribers -/// alive — tests interact through the manager and the subscribers. -#[allow(deprecated)] +/// Handles from [`launch_state_and_fetch_services`]: a direct-connection +/// [`TestManager`] plus standalone `Rpc`- and `Direct`-connection +/// [`NodeBackedIndexerService`]s pointed at the same validator. The owned services +/// exist only to keep their subscribers alive — tests interact through the manager and +/// the subscribers. +/// +/// The `fetch_*` fields carry the `Rpc` (JSON-RPC) connection; the `state_*` fields carry +/// the `Direct` (`ReadStateService`) connection. pub struct StateAndFetchServices { - /// The launched validator + Zaino test manager. - pub test_manager: TestManager, - /// Owned fetch service; keeps `fetch_subscriber` alive. - pub fetch_service: FetchService, - /// Subscriber to the standalone fetch service's chain index. - pub fetch_subscriber: FetchServiceSubscriber, - /// Owned state service; keeps `state_subscriber` alive. - pub state_service: StateService, - /// Subscriber to the standalone state service's chain index. - pub state_subscriber: StateServiceSubscriber, + /// The launched validator + Zaino test manager (direct connection). + pub test_manager: TestManager, + /// Owned `Rpc`-connection service; keeps `fetch_subscriber` alive. + pub fetch_service: NodeBackedIndexerService, + /// Subscriber to the standalone `Rpc`-connection service's chain index. + pub fetch_subscriber: NodeBackedIndexerServiceSubscriber, + /// Owned `Direct`-connection service; keeps `state_subscriber` alive. + pub state_service: NodeBackedIndexerService, + /// Subscriber to the standalone `Direct`-connection service's chain index. + pub state_subscriber: NodeBackedIndexerServiceSubscriber, } -#[allow(deprecated)] impl StateAndFetchServices { /// Mine `n` blocks and wait for both the fetch and state subscribers to /// observe the new tip. @@ -1089,7 +1110,6 @@ impl StateAndFetchServices { /// harness used by both the clientless and e2e live-test partitions. /// Wallet callers wrap this and additionally build lightclients from the /// returned manager's gRPC address. -#[allow(deprecated)] pub async fn launch_state_and_fetch_services( validator: &ValidatorKind, chain_cache: Option, @@ -1110,7 +1130,6 @@ pub async fn launch_state_and_fetch_services( /// caller instead of [`default_mining_pool`]: [`SHIELDED_FUNDING_POOL`] for /// sessions funding wallets from coinbase, or a pinned pool for tests whose /// subject is the miner's coinbase footprint. -#[allow(deprecated)] pub async fn launch_state_and_fetch_services_mining_to( mine_to_pool: MinerPool, validator: &ValidatorKind, @@ -1118,7 +1137,7 @@ pub async fn launch_state_and_fetch_services_mining_to( enable_zaino: bool, network: Option, ) -> StateAndFetchServices { - let test_manager = TestManager::::launch_mining_to( + let test_manager = TestManager::::launch_mining_to( mine_to_pool, validator, network, @@ -1140,25 +1159,11 @@ pub async fn launch_state_and_fetch_services_mining_to( Some(NetworkKind::Testnet) => { println!("Waiting for validator to spawn.."); tokio::time::sleep(std::time::Duration::from_millis(5000)).await; - Network::Testnet + Network::PubTestnet } - _ => Network::Regtest({ - let activation_heights = test_manager.local_net.get_activation_heights().await; - ActivationHeights { - before_overwinter: activation_heights.overwinter(), - overwinter: activation_heights.overwinter(), - sapling: activation_heights.sapling(), - blossom: activation_heights.blossom(), - heartwood: activation_heights.heartwood(), - canopy: activation_heights.canopy(), - nu5: activation_heights.nu5(), - nu6: activation_heights.nu6(), - nu6_1: activation_heights.nu6_1(), - nu6_2: activation_heights.nu6_2(), - nu6_3: activation_heights.nu6_3(), - nu7: activation_heights.nu7(), - } - }), + // The kind suffices: zainod adopts the schedule from the validator + // at spawn (zaino#1076). + _ => Network::Regtest, }; test_manager.local_net.print_stdout(); @@ -1182,41 +1187,42 @@ pub async fn launch_state_and_fetch_services_mining_to( None => test_manager.data_dir.clone(), }; - let state_service = StateService::spawn(StateServiceConfig::new( - zebra_state::Config { - cache_dir: state_chain_cache_dir, - ephemeral: false, - delete_old_database: true, - debug_stop_at_height: None, - debug_validity_check_interval: None, - should_backup_non_finalized_state: false, - debug_skip_non_finalized_state_backup_task: false, - }, - test_manager.full_node_rpc_listen_address.to_string(), - test_manager.full_node_grpc_listen_address, - false, - None, - None, - None, - ServiceConfig::default(), - StorageConfig { - database: DatabaseConfig { - path: test_manager - .local_net - .data_dir() - .path() - .to_path_buf() - .join("state-srvice-zaino"), + let state_service = + NodeBackedIndexerService::spawn(NodeBackedIndexerServiceConfig::new_direct( + zebra_state::Config { + cache_dir: state_chain_cache_dir, + ephemeral: false, + delete_old_database: true, + debug_stop_at_height: None, + debug_validity_check_interval: None, + should_backup_non_finalized_state: false, + debug_skip_non_finalized_state_backup_task: false, + }, + test_manager.full_node_rpc_listen_address.to_string(), + test_manager.full_node_grpc_listen_address, + false, + None, + None, + None, + ServiceConfig::default(), + StorageConfig { + database: DatabaseConfig { + path: test_manager + .local_net + .data_dir() + .path() + .to_path_buf() + .join("state-srvice-zaino"), + ..Default::default() + }, ..Default::default() }, - ..Default::default() - }, - false, // ephemeral_finalised_state: tests use a persistent finalised DB - network_type, - None, - )) - .await - .unwrap(); + false, // ephemeral_finalised_state: tests use a persistent finalised DB + network_type, + None, + )) + .await + .unwrap(); let state_subscriber = state_service.get_subscriber().inner(); @@ -1231,16 +1237,15 @@ pub async fn launch_state_and_fetch_services_mining_to( } } -/// Spawn a standalone [`FetchService`] pointed at `rpc_url`, storing its index -/// under `db_path`. Shared by the launch helpers below. -#[allow(deprecated)] +/// Spawn a standalone `Rpc`-connection [`NodeBackedIndexerService`] pointed at `rpc_url`, +/// storing its index under `db_path`. Shared by the launch helpers below. async fn spawn_fetch_service( rpc_url: String, cookie_dir: Option, db_path: PathBuf, network: Network, -) -> FetchService { - FetchService::spawn(FetchServiceConfig::new( +) -> NodeBackedIndexerService { + NodeBackedIndexerService::spawn(NodeBackedIndexerServiceConfig::new_rpc( rpc_url, cookie_dir, None, @@ -1267,18 +1272,17 @@ async fn spawn_fetch_service( /// server — for tests that compare the two. The owned services exist only to /// keep their subscribers alive. #[cfg(feature = "zcashd_support")] -#[allow(deprecated)] pub struct ZcashdDualFetchServices { /// The launched zcashd + Zaino test manager. - pub test_manager: TestManager, - /// Owned zcashd-direct fetch service; keeps `zcashd_subscriber` alive. - pub zcashd_fetch_service: FetchService, + pub test_manager: TestManager, + /// Owned zcashd-direct `Rpc` service; keeps `zcashd_subscriber` alive. + pub zcashd_fetch_service: NodeBackedIndexerService, /// Subscriber whose answers come from zcashd directly. - pub zcashd_subscriber: FetchServiceSubscriber, - /// Owned Zaino-pointed fetch service; keeps `zaino_subscriber` alive. - pub zaino_fetch_service: FetchService, + pub zcashd_subscriber: NodeBackedIndexerServiceSubscriber, + /// Owned Zaino-pointed `Rpc` service; keeps `zaino_subscriber` alive. + pub zaino_fetch_service: NodeBackedIndexerService, /// Subscriber whose answers come through Zaino's JSON-RPC server. - pub zaino_subscriber: FetchServiceSubscriber, + pub zaino_subscriber: NodeBackedIndexerServiceSubscriber, } #[cfg(feature = "zcashd_support")] @@ -1317,7 +1321,7 @@ pub async fn launch_zcashd_dual_fetch_services() -> ZcashdDualFetchServices { pub async fn launch_zcashd_dual_fetch_services_at( activation_heights: ActivationHeights, ) -> ZcashdDualFetchServices { - let test_manager = TestManager::::launch( + let test_manager = TestManager::::launch( &ValidatorKind::Zcashd, None, Some(activation_heights), @@ -1339,7 +1343,7 @@ pub async fn launch_zcashd_dual_fetch_services_at( .data_dir() .path() .join("zcashd-fetch-service-zaino"), - Network::Regtest(activation_heights), + Network::Regtest, ) .await; let zcashd_subscriber = zcashd_fetch_service.get_subscriber().inner(); @@ -1357,7 +1361,7 @@ pub async fn launch_zcashd_dual_fetch_services_at( .data_dir() .path() .join("zaino-fetch-service-zaino"), - Network::Regtest(activation_heights), + Network::Regtest, ) .await; let zaino_subscriber = zaino_fetch_service.get_subscriber().inner(); @@ -1414,11 +1418,10 @@ pub fn get_info_with_zeroed_timestamp(info: GetInfo) -> GetInfo { /// harness in both live-test partitions (`clientless` and `e2e`). The `e2e` /// partition wraps this and builds lightclients from the returned manager's /// gRPC address. -#[allow(deprecated)] pub async fn launch_with_fetch_subscriber( validator: &ValidatorKind, chain_cache: Option, -) -> (TestManager, FetchServiceSubscriber) { +) -> (TestManager, NodeBackedIndexerServiceSubscriber) { launch_with_fetch_subscriber_mining_to::( default_mining_pool(validator), validator, @@ -1431,13 +1434,12 @@ pub async fn launch_with_fetch_subscriber( /// caller instead of [`default_mining_pool`]: [`SHIELDED_FUNDING_POOL`] for /// sessions funding wallets from coinbase, or a pinned pool for tests whose /// subject is the miner's coinbase footprint. -#[allow(deprecated)] pub async fn launch_with_fetch_subscriber_mining_to( mine_to_pool: MinerPool, validator: &ValidatorKind, chain_cache: Option, -) -> (TestManager, FetchServiceSubscriber) { - let mut test_manager = TestManager::::launch_mining_to( +) -> (TestManager, NodeBackedIndexerServiceSubscriber) { + let mut test_manager = TestManager::::launch_mining_to( mine_to_pool, validator, None, @@ -1453,9 +1455,7 @@ pub async fn launch_with_fetch_subscriber_mining_to( (test_manager, fetch_service_subscriber) } -impl Drop - for TestManager -{ +impl Drop for TestManager { fn drop(&mut self) { debug!("[TEST] Shutting down test environment"); if let Some(handle) = &self.zaino_handle { @@ -1477,24 +1477,20 @@ async fn build_client( #[cfg(test)] mod launch_testmanager { use super::*; - #[allow(deprecated)] - use zaino_state::FetchService; /// Launch with no network/heights overrides and the optional servers off — /// the smoke tests' shared launch shape; only the chain cache and the /// zaino toggle vary. - async fn launch_minimal( + async fn launch_minimal( validator: &ValidatorKind, chain_cache: Option, enable_zaino: bool, - ) -> TestManager + ) -> TestManager where C: ValidatorExt, - Service: TestService, - IndexerError: From<<::Subscriber as ZcashIndexer>::Error>, - ::Subscriber: PollableTip, + Conn: ValidatorConnectionMarker, { - TestManager::::launch( + TestManager::::launch( validator, None, None, @@ -1517,7 +1513,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn basic() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zcashd, None, false).await; + launch_minimal::(&ValidatorKind::Zcashd, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.close().await; } @@ -1526,7 +1522,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn generate_blocks() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zcashd, None, false).await; + launch_minimal::(&ValidatorKind::Zcashd, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.local_net.generate_blocks(1).await.unwrap(); assert_eq!(3, (test_manager.local_net.get_chain_height().await)); @@ -1537,7 +1533,7 @@ mod launch_testmanager { #[tokio::test(flavor = "multi_thread")] #[allow(deprecated)] pub(crate) async fn with_chain() { - let mut test_manager = launch_minimal::( + let mut test_manager = launch_minimal::( &ValidatorKind::Zcashd, ZCASHD_CHAIN_CACHE_DIR.clone(), false, @@ -1551,7 +1547,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn zaino() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zcashd, None, true).await; + launch_minimal::(&ValidatorKind::Zcashd, None, true).await; let _grpc_client = build_client(test_manager.grpc_socket_to_uri()) .await @@ -1573,8 +1569,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn basic() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, false) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.close().await; } @@ -1583,8 +1578,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn generate_blocks() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, false) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.local_net.generate_blocks(1).await.unwrap(); assert_eq!(3, (test_manager.local_net.get_chain_height().await)); @@ -1595,7 +1589,7 @@ mod launch_testmanager { #[tokio::test(flavor = "multi_thread")] #[allow(deprecated)] pub(crate) async fn with_chain() { - let mut test_manager = launch_minimal::( + let mut test_manager = launch_minimal::( &ValidatorKind::Zebrad, ZEBRAD_CHAIN_CACHE_DIR.clone(), false, @@ -1609,8 +1603,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn zaino() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, true) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, true).await; let _grpc_client = build_client(test_manager.grpc_socket_to_uri()) .await .unwrap(); @@ -1620,15 +1613,12 @@ mod launch_testmanager { mod state_service { use super::*; - #[allow(deprecated)] - use zaino_state::StateService; #[tokio::test(flavor = "multi_thread")] #[allow(deprecated)] pub(crate) async fn basic() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, false) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.close().await; } @@ -1637,8 +1627,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn generate_blocks() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, false) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, false).await; assert_eq!(2, (test_manager.local_net.get_chain_height().await)); test_manager.local_net.generate_blocks(1).await.unwrap(); assert_eq!(3, (test_manager.local_net.get_chain_height().await)); @@ -1649,7 +1638,7 @@ mod launch_testmanager { #[tokio::test(flavor = "multi_thread")] #[allow(deprecated)] pub(crate) async fn with_chain() { - let mut test_manager = launch_minimal::( + let mut test_manager = launch_minimal::( &ValidatorKind::Zebrad, ZEBRAD_CHAIN_CACHE_DIR.clone(), false, @@ -1663,8 +1652,7 @@ mod launch_testmanager { #[allow(deprecated)] pub(crate) async fn zaino() { let mut test_manager = - launch_minimal::(&ValidatorKind::Zebrad, None, true) - .await; + launch_minimal::(&ValidatorKind::Zebrad, None, true).await; let _grpc_client = build_client(test_manager.grpc_socket_to_uri()) .await .unwrap(); diff --git a/packages/zaino-common/CHANGELOG.md b/packages/zaino-common/CHANGELOG.md index e2356e551..c126067a0 100644 --- a/packages/zaino-common/CHANGELOG.md +++ b/packages/zaino-common/CHANGELOG.md @@ -21,6 +21,10 @@ and this library adheres to Rust's notion of `sync_write_batch_size` so the two operations cannot inflate each other's peak memory. ### Changed +- `crypto::ensure_default_crypto_provider` now installs rustls's + **aws-lc-rs** provider (was ring) as the process-level default, and the + crate's rustls features become `aws_lc_rs` + `prefer-post-quantum` + (ADR-0006). First-install-wins semantics are unchanged. - **Breaking** — `StorageConfig::database.sync_write_batch_bytes` (raw bytes) is renamed to `sync_write_batch_size` and now expressed in **GiB** (new `SyncWriteBatchSize` newtype, mirroring `DatabaseSize`); the default is 8 GiB. @@ -31,8 +35,27 @@ and this library adheres to Rust's notion of unrecognized key under `[storage.database]` (e.g. a stale `sync_write_batch_bytes`) is a hard parse error instead of being silently ignored and falling back to the default budget — the silent fallback previously OOM-killed nodes. +- The `ActivationHeights` ↔ `ConfiguredActivationHeights` conversions and the + zebra-network → `ActivationHeights` extraction are now generated from a single + variant/field pair list (internal refactor; conversion behavior unchanged). +- `logging::init` / `logging::try_init` build their subscriber through one shared + installer (internal refactor). The `ZAINOLOG_FORMAT` / `ZAINOLOG_COLOR` / + `RUST_LOG` runtime interface and output formats are unchanged. ### Deprecated ### Removed +- Unused dependencies `thiserror`, `nu-ansi-term`, and `hex` (`hex`'s last + consumers were the display wrappers removed below). Verified empirically: + each dependency was deleted in turn and the crate re-checked with + `cargo check --all-targets`. +- **Breaking** — `Network::zaino_regtest_heights` (unused; regtest heights come + from `ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS` or an explicit `ActivationHeights`). +- **Breaking** — `logging::DisplayHash` and `logging::DisplayHexStr` (unused). +- **Breaking** — the unused programmatic logging-configuration surface: + `logging::init_with_config` / `logging::try_init_with_config` and the `LogConfig` + / `LogFormat` types (including the `LogConfig` builder methods and + `show_span_events`, which no caller could reach). Logging is configured via the + `ZAINOLOG_FORMAT` / `ZAINOLOG_COLOR` / `RUST_LOG` environment variables, whose + behavior is unchanged. ### Fixed ## [0.2.0] - 2026-06-17 diff --git a/packages/zaino-common/Cargo.toml b/packages/zaino-common/Cargo.toml index 5eed93aba..b3f12bad7 100644 --- a/packages/zaino-common/Cargo.toml +++ b/packages/zaino-common/Cargo.toml @@ -15,13 +15,12 @@ zebra-chain = { workspace = true } # Serialization serde = { workspace = true, features = ["derive"] } -# Error handling -thiserror = { workspace = true } - # Logging tracing = { workspace = true } tracing-subscriber = { workspace = true } tracing-tree = { workspace = true } -nu-ansi-term = { workspace = true } -hex = { workspace = true } time = { workspace = true } +rustls = { version = "0.23", default-features = false, features = [ + "aws_lc_rs", + "prefer-post-quantum", +] } diff --git a/packages/zaino-common/src/config/network.rs b/packages/zaino-common/src/config/network.rs index d518d895b..61a3dd294 100644 --- a/packages/zaino-common/src/config/network.rs +++ b/packages/zaino-common/src/config/network.rs @@ -5,80 +5,38 @@ use std::fmt; use serde::{Deserialize, Serialize}; use zebra_chain::parameters::testnet::ConfiguredActivationHeights; -/// Must equal zcash_local_net's `supported_regtest_activation_heights`: the -/// zcash-devtool wallet client hardcodes that canonical set (NU6.3 at 2), and a -/// zebrad configured differently rejects wallet-built transactions with -/// "incorrect consensus branch id". See -/// . -pub const ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS: ActivationHeights = ActivationHeights { - overwinter: Some(1), - before_overwinter: Some(1), - sapling: Some(1), - blossom: Some(1), - heartwood: Some(1), - canopy: Some(1), - nu5: Some(2), - nu6: Some(2), - nu6_1: Some(2), - nu6_2: Some(2), - nu6_3: Some(2), - nu7: None, -}; - -/// Network type for Zaino configuration. +/// The network *kind* zaino is configured for. Deliberately payload-free: +/// activation heights are chain facts the validator owns, so a config value +/// cannot carry them — the backends adopt the runtime schedule from the +/// validator's `getblockchaininfo.upgrades` at spawn and hold it as a +/// `zebra_chain::parameters::Network` +/// (). A pre-adoption +/// height read is unrepresentable: this type has no heights to read. #[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Deserialize, serde::Serialize)] -#[serde(from = "NetworkSerde", into = "NetworkSerde")] pub enum Network { /// Mainnet network Mainnet, - /// Testnet network - Testnet, + /// The Public Testnet: the shared-consensus test network, whose + /// activation heights come only from checked-in zebra parameters — + /// never from configuration. Accepts the legacy config spelling + /// `"Testnet"`. + #[serde(alias = "Testnet")] + PubTestnet, /// Regtest network (for local testing) - Regtest(ActivationHeights), + Regtest, } impl fmt::Display for Network { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { Network::Mainnet => write!(f, "Mainnet"), - Network::Testnet => write!(f, "Testnet"), - Network::Regtest(_) => write!(f, "Regtest"), - } - } -} - -/// Helper type for Network serialization/deserialization. -/// -/// This allows Network to serialize as simple strings ("Mainnet", "Testnet", "Regtest") -/// while the actual Network::Regtest variant carries activation heights internally. -#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Deserialize, serde::Serialize)] -enum NetworkSerde { - Mainnet, - Testnet, - Regtest, -} - -impl From for Network { - fn from(value: NetworkSerde) -> Self { - match value { - NetworkSerde::Mainnet => Network::Mainnet, - NetworkSerde::Testnet => Network::Testnet, - NetworkSerde::Regtest => Network::Regtest(ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS), - } - } -} - -impl From for NetworkSerde { - fn from(value: Network) -> Self { - match value { - Network::Mainnet => NetworkSerde::Mainnet, - Network::Testnet => NetworkSerde::Testnet, - Network::Regtest(_) => NetworkSerde::Regtest, + Network::PubTestnet => write!(f, "PubTestnet"), + Network::Regtest => write!(f, "Regtest"), } } } -/// Configurable activation heights for Regtest and configured Testnets. +/// Configurable activation heights for a Regtest validator launch. /// /// We use our own type instead of the zebra type /// as the zebra type is missing a number of useful @@ -141,116 +99,90 @@ impl Default for ActivationHeights { } } -impl From for ActivationHeights { - fn from( - ConfiguredActivationHeights { - before_overwinter, - overwinter, - sapling, - blossom, - heartwood, - canopy, - nu5, - nu6, - nu6_1, - nu6_2, - nu6_3, - nu7, - }: ConfiguredActivationHeights, - ) -> Self { - Self { - before_overwinter, - overwinter, - sapling, - blossom, - heartwood, - canopy, - nu5, - nu6, - nu6_1, - nu6_2, - nu6_3, - nu7, - } - } -} -impl From for ConfiguredActivationHeights { - fn from( - ActivationHeights { - before_overwinter, - overwinter, - sapling, - blossom, - heartwood, - canopy, - nu5, - nu6, - nu6_1, - nu6_2, - nu6_3, - nu7, - }: ActivationHeights, - ) -> Self { - Self { - before_overwinter, - overwinter, - sapling, - blossom, - heartwood, - canopy, - nu5, - nu6, - nu6_1, - nu6_2, - nu6_3, - nu7, +/// Records the `NetworkUpgrade`-variant ↔ `ActivationHeights`-field correspondence +/// exactly once, generating everything derived from it: the two field-by-field +/// `From` conversions between [`ActivationHeights`] and zebra's +/// [`ConfiguredActivationHeights`] (the structs share field names), the +/// all-`None` [`ActivationHeights::NEVER_ACTIVATED`] schedule, and the +/// per-upgrade [`ActivationHeights::slot_mut`] accessor. +/// +/// A declarative macro rather than functions because plain `fn`s cannot abstract +/// over struct fields, and the variant/field spellings (`Nu5`/`nu5`) differ only +/// by casing, which `macro_rules!` cannot derive — hence explicit pairs. +/// +/// Zebra's side of these conversions is structurally stable; the recurring edit +/// here is a new network upgrade, which lands as a single `(Variant, field)` +/// entry in the invocation below (after adding the struct field). The exhaustive +/// destructures and match keep full compile-time drift detection: a new zebra +/// field or variant fails the build until its pair is added. +macro_rules! activation_heights_mirror { + ($(($variant:ident, $field:ident)),* $(,)?) => { + impl From for ActivationHeights { + fn from( + ConfiguredActivationHeights { $($field),* }: ConfiguredActivationHeights, + ) -> Self { + Self { $($field),* } + } } - } -} -impl Network { - /// Convert to Zebra's network type for internal use (alias for to_zebra_default). - pub fn to_zebra_network(&self) -> zebra_chain::parameters::Network { - self.into() - } + impl From for ConfiguredActivationHeights { + fn from(ActivationHeights { $($field),* }: ActivationHeights) -> Self { + Self { $($field),* } + } + } - /// Get the standard regtest activation heights used by Zaino. - pub fn zaino_regtest_heights() -> ConfiguredActivationHeights { - ConfiguredActivationHeights { - before_overwinter: Some(1), - overwinter: Some(1), - sapling: Some(1), - blossom: Some(1), - heartwood: Some(1), - canopy: Some(1), - nu5: Some(1), - nu6: Some(1), - nu6_1: None, - nu6_2: None, - nu6_3: None, - nu7: None, + impl ActivationHeights { + /// The all-`None` schedule: every upgrade never-activated. The + /// starting point for building heights from an external report, + /// where an upgrade absent from the report must stay unset. + pub const NEVER_ACTIVATED: Self = Self { $($field: None),* }; + + /// Mutable slot for `upgrade`'s activation height, or `None` for + /// `Genesis` (height 0 by definition; it has no slot). + pub fn slot_mut( + &mut self, + upgrade: zebra_chain::parameters::NetworkUpgrade, + ) -> Option<&mut Option> { + match upgrade { + zebra_chain::parameters::NetworkUpgrade::Genesis => None, + $( + zebra_chain::parameters::NetworkUpgrade::$variant => { + Some(&mut self.$field) + } + )* + } + } } - } + }; +} +activation_heights_mirror!( + (BeforeOverwinter, before_overwinter), + (Overwinter, overwinter), + (Sapling, sapling), + (Blossom, blossom), + (Heartwood, heartwood), + (Canopy, canopy), + (Nu5, nu5), + (Nu6, nu6), + (Nu6_1, nu6_1), + (Nu6_2, nu6_2), + (Nu6_3, nu6_3), + (Nu7, nu7), +); + +impl Network { /// Determines if we should wait for the server to fully sync. Used for testing /// - /// - Mainnet/Testnet: Skip sync (false) because we don't want to sync real chains in tests + /// - Mainnet / The Public Testnet: Skip sync (false) because we don't want + /// to sync real chains in tests /// - Regtest: Enable sync (true) because regtest is local and fast to sync pub fn wait_on_server_sync(&self) -> bool { match self { - Network::Mainnet | Network::Testnet => false, // Real networks - don't try to sync the whole chain - Network::Regtest(_) => true, // Local network - safe and fast to sync - } - } - - pub fn from_network_kind_and_activation_heights( - network: &zebra_chain::parameters::NetworkKind, - activation_heights: &ActivationHeights, - ) -> Self { - match network { - zebra_chain::parameters::NetworkKind::Mainnet => Network::Mainnet, - zebra_chain::parameters::NetworkKind::Testnet => Network::Testnet, - zebra_chain::parameters::NetworkKind::Regtest => Network::Regtest(*activation_heights), + // Real networks - don't try to sync the whole chain + Network::Mainnet | Network::PubTestnet => false, + // Local network - safe and fast to sync + Network::Regtest => true, } } } @@ -261,85 +193,26 @@ impl From for Network { zebra_chain::parameters::Network::Mainnet => Network::Mainnet, zebra_chain::parameters::Network::Testnet(parameters) => { if parameters.is_regtest() { - let mut activation_heights = ActivationHeights { - before_overwinter: None, - overwinter: None, - sapling: None, - blossom: None, - heartwood: None, - canopy: None, - nu5: None, - nu6: None, - nu6_1: None, - nu6_2: None, - nu6_3: None, - nu7: None, - }; - for (height, upgrade) in parameters.activation_heights().iter() { - match upgrade { - zebra_chain::parameters::NetworkUpgrade::Genesis => (), - zebra_chain::parameters::NetworkUpgrade::BeforeOverwinter => { - activation_heights.before_overwinter = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Overwinter => { - activation_heights.overwinter = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Sapling => { - activation_heights.sapling = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Blossom => { - activation_heights.blossom = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Heartwood => { - activation_heights.heartwood = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Canopy => { - activation_heights.canopy = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu5 => { - activation_heights.nu5 = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu6 => { - activation_heights.nu6 = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu6_1 => { - activation_heights.nu6_1 = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu6_2 => { - activation_heights.nu6_2 = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu6_3 => { - activation_heights.nu6_3 = Some(height.0) - } - zebra_chain::parameters::NetworkUpgrade::Nu7 => { - activation_heights.nu7 = Some(height.0) - } - } - } - Network::Regtest(activation_heights) + Network::Regtest } else { - Network::Testnet + Network::PubTestnet } } } } } -impl From for zebra_chain::parameters::Network { - fn from(val: Network) -> Self { - match val { - Network::Regtest(activation_heights) => zebra_chain::parameters::Network::new_regtest( - Into::::into(activation_heights).into(), - ), - Network::Testnet => zebra_chain::parameters::Network::new_default_testnet(), - Network::Mainnet => zebra_chain::parameters::Network::Mainnet, - } - } -} - -impl From<&Network> for zebra_chain::parameters::Network { - fn from(val: &Network) -> Self { - (*val).into() +impl ActivationHeights { + /// Builds the runtime regtest network for a chain whose schedule is + /// known first-hand: a validator being *launched* with these heights, or + /// a test fixture that is its own chain (mockchain sources, proptest + /// block generators). Production indexer code never calls this with + /// configured values — it adopts the runtime network from the + /// validator's reported schedule instead (zaino#1076). + pub fn to_regtest_network(&self) -> zebra_chain::parameters::Network { + zebra_chain::parameters::Network::new_regtest( + Into::::into(*self).into(), + ) } } diff --git a/packages/zaino-common/src/crypto.rs b/packages/zaino-common/src/crypto.rs new file mode 100644 index 000000000..c24dc59d9 --- /dev/null +++ b/packages/zaino-common/src/crypto.rs @@ -0,0 +1,36 @@ +//! Process-level rustls `CryptoProvider` management. + +/// Installs rustls's `aws-lc-rs` provider as the process-level default +/// if no provider is installed yet. +/// +/// Call this before constructing anything that builds a rustls config — +/// a `reqwest` client (the workspace enables reqwest's +/// `rustls-no-provider` feature, which never auto-selects a provider) or +/// a TLS-enabled tonic server. rustls's own auto-selection is also +/// unreliable here: it works only when exactly one of rustls's `ring` / +/// `aws-lc-rs` features is enabled, and dependency feature-unification +/// can silently enable both, which panicked at runtime on TLS-enabled +/// deployments (zingolabs/zaino#1360). Installing explicitly removes +/// that dependence on the feature graph. +/// +/// aws-lc-rs is the preferred provider (ADR-0006): it is the only rustls +/// provider with post-quantum key exchange, and the workspace enables +/// rustls's `prefer-post-quantum` feature so the X25519MLKEM768 hybrid +/// group leads this provider's defaults. That ordering governs zaino's +/// outbound (client-role) handshakes; rustls servers follow the client's +/// group preference, so inbound hybrid uptake is decided by the +/// connecting wallet's TLS stack. Classical key-exchange groups remain +/// offered and accepted, but are deprecated (see the ADR). +/// +/// The process default is first-install-wins and is not constrained by +/// our crates' rustls features, so an embedder (e.g. zallet) that has +/// already installed a provider keeps it: zaino then handshakes through +/// that provider instead of aws-lc-rs, which is fine for any provider +/// implementing the standard TLS suites (ring and aws-lc-rs both do). +pub fn ensure_default_crypto_provider() { + if rustls::crypto::CryptoProvider::get_default().is_none() { + // A racing concurrent install is the only error case; either + // provider serves. + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + } +} diff --git a/packages/zaino-common/src/lib.rs b/packages/zaino-common/src/lib.rs index e3f62182c..818f23935 100644 --- a/packages/zaino-common/src/lib.rs +++ b/packages/zaino-common/src/lib.rs @@ -5,6 +5,7 @@ pub mod config; pub mod consensus; +pub mod crypto; pub mod logging; pub mod net; pub mod probing; @@ -16,7 +17,7 @@ pub use net::{resolve_socket_addr, try_resolve_address, AddressResolution}; // Re-export commonly used config types at crate root for backward compatibility. // This allows existing code using `use zaino_common::Network` to continue working. -pub use config::network::{ActivationHeights, Network, ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS}; +pub use config::network::{ActivationHeights, Network}; pub use config::service::ServiceConfig; pub use config::storage::{ AccumulatorRebuildMemorySize, CacheConfig, DatabaseConfig, DatabaseSize, StorageConfig, diff --git a/packages/zaino-common/src/logging.rs b/packages/zaino-common/src/logging.rs index e97703bd3..7bf812671 100644 --- a/packages/zaino-common/src/logging.rs +++ b/packages/zaino-common/src/logging.rs @@ -18,23 +18,19 @@ //! ```no_run //! use zaino_common::logging; //! -//! // Initialize with defaults (tree format, info level) +//! // Initialize logging, configured via the environment variables above. //! logging::init(); -//! -//! // Or with custom configuration -//! logging::init_with_config(logging::LogConfig::default().format(logging::LogFormat::Json)); //! ``` use std::env; -use std::fmt; use std::io::IsTerminal; use time::macros::format_description; use tracing::Level; use tracing_subscriber::{ - fmt::{format::FmtSpan, time::UtcTime}, + fmt::time::UtcTime, layer::SubscriberExt, - util::SubscriberInitExt, + util::{SubscriberInitExt, TryInitError}, EnvFilter, }; use tracing_tree::HierarchicalLayer; @@ -45,7 +41,7 @@ const TIME_FORMAT: &[time::format_description::FormatItem<'static>] = /// Log output format. #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] -pub enum LogFormat { +enum LogFormat { /// Hierarchical tree view showing span nesting. Tree, /// Flat chronological stream (default). @@ -57,7 +53,7 @@ pub enum LogFormat { impl LogFormat { /// Parse from string (case-insensitive). - pub fn parse_str(s: &str) -> Option { + fn parse_str(s: &str) -> Option { match s.to_lowercase().as_str() { "tree" => Some(LogFormat::Tree), "stream" => Some(LogFormat::Stream), @@ -67,7 +63,7 @@ impl LogFormat { } /// Get from ZAINOLOG_FORMAT environment variable. - pub fn from_env() -> Self { + fn from_env() -> Self { env::var("ZAINOLOG_FORMAT") .ok() .and_then(|s| Self::parse_str(&s)) @@ -75,27 +71,15 @@ impl LogFormat { } } -impl fmt::Display for LogFormat { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - LogFormat::Tree => write!(f, "tree"), - LogFormat::Stream => write!(f, "stream"), - LogFormat::Json => write!(f, "json"), - } - } -} - -/// Logging configuration. +/// Logging configuration, read from the environment. #[derive(Debug, Clone)] -pub struct LogConfig { +struct LogConfig { /// Output format (tree, stream, or json). - pub format: LogFormat, + format: LogFormat, /// Enable ANSI colors. - pub color: bool, + color: bool, /// Default log level. - pub level: Level, - /// Show span events (enter/exit). - pub show_span_events: bool, + level: Level, } impl Default for LogConfig { @@ -119,228 +103,72 @@ impl Default for LogConfig { format: LogFormat::from_env(), color, level: Level::INFO, - show_span_events: false, } } } -impl LogConfig { - /// Create a new config with defaults. - pub fn new() -> Self { - Self::default() - } - - /// Set the log format. - pub fn format(mut self, format: LogFormat) -> Self { - self.format = format; - self - } - - /// Enable or disable colors. - pub fn color(mut self, color: bool) -> Self { - self.color = color; - self - } - - /// Set the default log level. - pub fn level(mut self, level: Level) -> Self { - self.level = level; - self - } - - /// Show span enter/exit events. - pub fn show_span_events(mut self, show: bool) -> Self { - self.show_span_events = show; - self - } -} - -/// Initialize logging with default configuration. +/// Initialize logging, configured from the environment (see module docs). +/// +/// # Panics /// -/// Reads `ZAINOLOG_FORMAT` environment variable to determine format: -/// - "stream" (default): Flat chronological output with timestamps -/// - "tree": Hierarchical span-based output -/// - "json": Machine-parseable JSON +/// Panics if a global tracing subscriber has already been set. pub fn init() { - init_with_config(LogConfig::default()); -} - -/// Initialize logging with custom configuration. -pub fn init_with_config(config: LogConfig) { - // If RUST_LOG is set, use it directly. Otherwise, default to zaino crates only. - // Users can set RUST_LOG=info to see all crates including zebra. - let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| { - EnvFilter::new(format!( - "zaino={level},zainod={level},zainodlib={level}", - level = config.level.as_str() - )) - }); - - match config.format { - LogFormat::Tree => init_tree(env_filter, config), - LogFormat::Stream => init_stream(env_filter, config), - LogFormat::Json => init_json(env_filter), - } + try_install(LogConfig::default()).expect("global tracing subscriber already set"); } /// Try to initialize logging (won't fail if already initialized). /// /// Useful for tests where multiple test functions may try to initialize logging. pub fn try_init() { - try_init_with_config(LogConfig::default()); + let _ = try_install(LogConfig::default()); } -/// Try to initialize logging with custom configuration. -pub fn try_init_with_config(config: LogConfig) { +/// Build the subscriber described by `config` and install it as the global +/// default, erroring if one is already set. +fn try_install(config: LogConfig) -> Result<(), TryInitError> { + // If RUST_LOG is set, use it directly. Otherwise, default to zaino crates only. + // Users can set RUST_LOG=info to see all crates including zebra. let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| { EnvFilter::new(format!( "zaino={level},zainod={level},zainodlib={level}", level = config.level.as_str() )) }); + let registry = tracing_subscriber::registry().with(env_filter); match config.format { - LogFormat::Tree => { - let _ = try_init_tree(env_filter, config); - } - LogFormat::Stream => { - let _ = try_init_stream(env_filter, config); - } - LogFormat::Json => { - let _ = try_init_json(env_filter); - } - } -} - -fn init_tree(env_filter: EnvFilter, config: LogConfig) { - let layer = HierarchicalLayer::new(2) - .with_ansi(config.color) - .with_targets(true) - .with_bracketed_fields(true) - .with_indent_lines(true) - .with_thread_ids(false) - .with_thread_names(false) - .with_deferred_spans(true) // Only show spans when they have events - .with_verbose_entry(false) // Don't repeat span info on entry - .with_verbose_exit(false); // Don't repeat span info on exit - - tracing_subscriber::registry() - .with(env_filter) - .with(layer) - .init(); -} - -fn try_init_tree( - env_filter: EnvFilter, - config: LogConfig, -) -> Result<(), tracing_subscriber::util::TryInitError> { - let layer = HierarchicalLayer::new(2) - .with_ansi(config.color) - .with_targets(true) - .with_bracketed_fields(true) - .with_indent_lines(true) - .with_thread_ids(false) - .with_thread_names(false) - .with_deferred_spans(true) // Only show spans when they have events - .with_verbose_entry(false) // Don't repeat span info on entry - .with_verbose_exit(false); // Don't repeat span info on exit - - tracing_subscriber::registry() - .with(env_filter) - .with(layer) - .try_init() -} - -fn init_stream(env_filter: EnvFilter, config: LogConfig) { - let span_events = if config.show_span_events { - FmtSpan::ENTER | FmtSpan::EXIT - } else { - FmtSpan::NONE - }; - - tracing_subscriber::fmt() - .with_env_filter(env_filter) - .with_timer(UtcTime::new(TIME_FORMAT)) - .with_target(true) - .with_ansi(config.color) - .with_span_events(span_events) - .pretty() - .init(); -} - -fn try_init_stream( - env_filter: EnvFilter, - config: LogConfig, -) -> Result<(), tracing_subscriber::util::TryInitError> { - let span_events = if config.show_span_events { - FmtSpan::ENTER | FmtSpan::EXIT - } else { - FmtSpan::NONE - }; - - let fmt_layer = tracing_subscriber::fmt::layer() - .with_timer(UtcTime::new(TIME_FORMAT)) - .with_target(true) - .with_ansi(config.color) - .with_span_events(span_events) - .pretty(); - - tracing_subscriber::registry() - .with(env_filter) - .with(fmt_layer) - .try_init() -} - -fn init_json(env_filter: EnvFilter) { - // JSON format keeps full RFC3339 timestamps for machine parsing - tracing_subscriber::fmt() - .with_env_filter(env_filter) - .json() - .with_timer(UtcTime::rfc_3339()) - .with_target(true) - .init(); -} - -fn try_init_json(env_filter: EnvFilter) -> Result<(), tracing_subscriber::util::TryInitError> { - // JSON format keeps full RFC3339 timestamps for machine parsing - let fmt_layer = tracing_subscriber::fmt::layer() - .json() - .with_timer(UtcTime::rfc_3339()) - .with_target(true); - - tracing_subscriber::registry() - .with(env_filter) - .with(fmt_layer) - .try_init() -} - -/// Wrapper for displaying hashes in a compact format. -/// -/// Shows the first 4 bytes (8 hex chars) followed by "..". -pub struct DisplayHash<'a>(pub &'a [u8]); - -impl fmt::Display for DisplayHash<'_> { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - if self.0.len() >= 4 { - write!(f, "{}..", hex::encode(&self.0[..4])) - } else { - write!(f, "{}", hex::encode(self.0)) - } - } -} - -/// Wrapper for displaying hex strings in a compact format. -/// -/// Shows the first 8 chars followed by "..". -pub struct DisplayHexStr<'a>(pub &'a str); - -impl fmt::Display for DisplayHexStr<'_> { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - if self.0.len() > 8 { - write!(f, "{}..", &self.0[..8]) - } else { - write!(f, "{}", self.0) - } + LogFormat::Tree => registry + .with( + HierarchicalLayer::new(2) + .with_ansi(config.color) + .with_targets(true) + .with_bracketed_fields(true) + .with_indent_lines(true) + .with_thread_ids(false) + .with_thread_names(false) + .with_deferred_spans(true) // Only show spans when they have events + .with_verbose_entry(false) // Don't repeat span info on entry + .with_verbose_exit(false), // Don't repeat span info on exit + ) + .try_init(), + LogFormat::Stream => registry + .with( + tracing_subscriber::fmt::layer() + .with_timer(UtcTime::new(TIME_FORMAT)) + .with_target(true) + .with_ansi(config.color) + .pretty(), + ) + .try_init(), + LogFormat::Json => registry + .with( + // JSON format keeps full RFC3339 timestamps for machine parsing + tracing_subscriber::fmt::layer() + .json() + .with_timer(UtcTime::rfc_3339()) + .with_target(true), + ) + .try_init(), } } @@ -356,31 +184,4 @@ mod tests { assert_eq!(LogFormat::parse_str("json"), Some(LogFormat::Json)); assert_eq!(LogFormat::parse_str("unknown"), None); } - - #[test] - fn test_display_hash() { - let hash = [0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0]; - assert_eq!(format!("{}", DisplayHash(&hash)), "12345678.."); - } - - #[test] - fn test_display_hex_str() { - let hex_str = "1234567890abcdef"; - assert_eq!(format!("{}", DisplayHexStr(hex_str)), "12345678.."); - - let short_hex = "1234"; - assert_eq!(format!("{}", DisplayHexStr(short_hex)), "1234"); - } - - #[test] - fn test_config_builder() { - let config = LogConfig::new() - .format(LogFormat::Json) - .color(false) - .level(Level::DEBUG); - - assert_eq!(config.format, LogFormat::Json); - assert!(!config.color); - assert_eq!(config.level, Level::DEBUG); - } } diff --git a/packages/zaino-fetch/src/jsonrpsee/connector.rs b/packages/zaino-fetch/src/jsonrpsee/connector.rs index cc8090150..056821afa 100644 --- a/packages/zaino-fetch/src/jsonrpsee/connector.rs +++ b/packages/zaino-fetch/src/jsonrpsee/connector.rs @@ -210,6 +210,29 @@ fn record_outbound_rpc_metrics(method: &'static str, start: std::time::Instant, } } +/// Whether the outbound JSON-RPC client keeps a cookie store (required +/// for cookie-authenticated validators). +enum CookieSupport { + Enabled, + Disabled, +} + +/// Builds the outbound JSON-RPC HTTP client shared by every connector +/// entry point: 2s connect timeout, 5s request timeout, no redirects. +fn build_rpc_client(cookies: CookieSupport) -> Result { + // reqwest is built with `rustls-no-provider`, which never + // auto-selects a rustls CryptoProvider (zingolabs/zaino#1360). + zaino_common::crypto::ensure_default_crypto_provider(); + let mut builder = ClientBuilder::new() + .connect_timeout(Duration::from_secs(2)) + .timeout(Duration::from_secs(5)) + .redirect(reqwest::redirect::Policy::none()); + if matches!(cookies, CookieSupport::Enabled) { + builder = builder.cookie_store(true); + } + builder.build() +} + impl JsonRpSeeConnector { /// Creates a new JsonRpSeeConnector with Basic Authentication. pub fn new_with_basic_auth( @@ -217,12 +240,8 @@ impl JsonRpSeeConnector { username: String, password: String, ) -> Result { - let client = ClientBuilder::new() - .connect_timeout(Duration::from_secs(2)) - .timeout(Duration::from_secs(5)) - .redirect(reqwest::redirect::Policy::none()) - .build() - .map_err(TransportError::ReqwestError)?; + let client = + build_rpc_client(CookieSupport::Disabled).map_err(TransportError::ReqwestError)?; Ok(Self { url, @@ -236,13 +255,8 @@ impl JsonRpSeeConnector { pub fn new_with_cookie_auth(url: Url, cookie_path: &Path) -> Result { let cookie_password = read_and_parse_cookie_token(cookie_path)?; - let client = ClientBuilder::new() - .connect_timeout(Duration::from_secs(2)) - .timeout(Duration::from_secs(5)) - .redirect(reqwest::redirect::Policy::none()) - .cookie_store(true) - .build() - .map_err(TransportError::ReqwestError)?; + let client = + build_rpc_client(CookieSupport::Enabled).map_err(TransportError::ReqwestError)?; Ok(Self { url, @@ -979,12 +993,8 @@ async fn test_node_connection( url: Url, auth_method: AuthMethod, ) -> Result<(), TestNodeConnectionError> { - let client = Client::builder() - .connect_timeout(std::time::Duration::from_secs(2)) - .timeout(std::time::Duration::from_secs(5)) - .redirect(reqwest::redirect::Policy::none()) - .build() - .map_err(TestNodeConnectionError::ClientBuild)?; + let client = + build_rpc_client(CookieSupport::Disabled).map_err(TestNodeConnectionError::ClientBuild)?; let request_body = r#"{"jsonrpc":"2.0","method":"getinfo","params":[],"id":1}"#; let mut request_builder = client diff --git a/packages/zaino-fetch/src/jsonrpsee/response.rs b/packages/zaino-fetch/src/jsonrpsee/response.rs index b63fe58f5..f29e2c051 100644 --- a/packages/zaino-fetch/src/jsonrpsee/response.rs +++ b/packages/zaino-fetch/src/jsonrpsee/response.rs @@ -1213,8 +1213,10 @@ pub struct BlockObject { #[serde(default, skip_serializing_if = "Option::is_none")] pub difficulty: Option, - /// List of transaction IDs in block order, hex-encoded. - pub tx: Vec, + /// The block's transactions in block order: hex-encoded transaction IDs + /// at verbosity 1, full transaction objects at verbosity 2. Both shapes + /// share zebra-rpc's untagged enum. + pub tx: Vec, /// Chain supply balance #[serde(default)] @@ -1245,6 +1247,21 @@ pub struct BlockObject { pub next_block_hash: Option, } +impl BlockObject { + /// Per-pool chain value balances as of this block, when the validator + /// reports them (`getblock` verbosity 2). + pub fn value_pools(&self) -> Option<&[ChainBalance]> { + self.value_pools.as_deref() + } +} + +impl ChainBalance { + /// The underlying per-pool balance entry. + pub fn balance(&self) -> &GetBlockchainInfoBalance { + &self.0 + } +} + impl TryFrom for zebra_rpc::methods::GetBlock { type Error = zebra_chain::serialization::SerializationError; @@ -1254,15 +1271,7 @@ impl TryFrom for zebra_rpc::methods::GetBlock { Ok(zebra_rpc::methods::GetBlock::Raw(serialized_block.0)) } GetBlockResponse::Object(block) => { - let tx: Result, _> = block - .tx - .into_iter() - .map(|txid| { - txid.parse::() - .map(zebra_rpc::methods::GetBlockTransaction::Hash) - }) - .collect(); - let tx = tx?; + let tx = block.tx; Ok(zebra_rpc::methods::GetBlock::Object(Box::new( zebra_rpc::client::BlockObject::new( @@ -2028,3 +2037,103 @@ mod get_transaction_response { ); } } + +#[cfg(test)] +mod get_block_response { + use super::GetBlockResponse; + + /// Builds the `tx` entry of a verbosity-2 `getblock` response exactly as + /// zebrad serializes it, by round-tripping a real test-vector + /// transaction through zebra-rpc's own response types. Constructing the + /// fixture from zebra's serializer keeps it faithful to the wire across + /// zebra upgrades, with no hand-maintained JSON to drift. + fn verbosity_2_transaction_json() -> serde_json::Value { + use zebra_chain::serialization::ZcashDeserializeInto as _; + + let vector = wire_serialized_transaction_test_data::transactions::get_test_vectors() + .first() + .cloned() + .expect("the test-vector crate provides at least one transaction"); + let transaction: zebra_chain::transaction::Transaction = vector + .tx + .as_slice() + .zcash_deserialize_into() + .expect("test-vector transactions deserialize"); + let txid = transaction.hash(); + let tx_object = zebra_rpc::client::TransactionObject::from_transaction( + std::sync::Arc::new(transaction), + Some(zebra_chain::block::Height(2)), + Some(1), + &zebra_chain::parameters::Network::Mainnet, + None, + None, + None, + txid, + ); + serde_json::to_value(zebra_rpc::methods::GetBlockTransaction::Object(Box::new( + tx_object, + ))) + .expect("zebra's response types serialize") + } + + fn block_json_with_tx(tx: serde_json::Value) -> serde_json::Value { + serde_json::json!({ + "hash": "0000000000000000000000000000000000000000000000000000000000000002", + "confirmations": 1, + "tx": [tx], + "trees": {}, + }) + } + + /// A verbosity-1 `getblock` response carries txid strings in `tx`. This + /// pins the shape the type already handles, so the verbosity-2 fix below + /// cannot regress it. + #[test] + fn parses_verbosity_1_txid_strings() { + let block_json = block_json_with_tx(serde_json::json!( + "e85e5729b34c34e8e62aa639302cb3434bb961666e13191da6dc39fcf775b320" + )); + + let parsed: GetBlockResponse = + serde_json::from_value(block_json).expect("verbosity-1 block objects deserialize"); + let GetBlockResponse::Object(block) = parsed else { + panic!("a JSON block object must parse as the object variant"); + }; + assert_eq!(block.tx.len(), 1); + } + + /// A verbosity-2 `getblock` response carries full transaction objects in + /// `tx`; the response type must deserialize that shape as well, and the + /// conversion into zebra-rpc's `GetBlock` must preserve the objects + /// (issue #1380). + #[test] + fn parses_verbosity_2_transaction_objects() { + let block_json = block_json_with_tx(verbosity_2_transaction_json()); + + let parsed: GetBlockResponse = + serde_json::from_value(block_json).expect("verbosity-2 block objects deserialize"); + let GetBlockResponse::Object(ref block) = parsed else { + panic!("a JSON block object must parse as the object variant"); + }; + assert!( + matches!( + block.tx.as_slice(), + [zebra_rpc::methods::GetBlockTransaction::Object(_)] + ), + "the verbosity-2 entry must parse as a transaction object" + ); + + let converted: zebra_rpc::methods::GetBlock = + parsed.try_into().expect("the object form converts"); + let zebra_rpc::methods::GetBlock::Object(converted) = converted else { + panic!("the conversion must keep the object variant"); + }; + assert!( + matches!( + converted.tx().as_slice(), + [zebra_rpc::methods::GetBlockTransaction::Object(_)] + ), + "the conversion must pass the transaction object through" + ); + } +} diff --git a/packages/zaino-fetch/src/jsonrpsee/response/address_deltas.rs b/packages/zaino-fetch/src/jsonrpsee/response/address_deltas.rs index ef422bb39..f509ec857 100644 --- a/packages/zaino-fetch/src/jsonrpsee/response/address_deltas.rs +++ b/packages/zaino-fetch/src/jsonrpsee/response/address_deltas.rs @@ -278,7 +278,7 @@ mod tests { fn sample_delta_with_block_index(i: u32, bi: Option) -> AddressDelta { AddressDelta { - satoshis: if i % 2 == 0 { 1_000 } else { -500 }, + satoshis: if i.is_multiple_of(2) { 1_000 } else { -500 }, txid: format!("deadbeef{:02x}", i), index: i, height: 123_456 + i, diff --git a/packages/zaino-serve/CHANGELOG.md b/packages/zaino-serve/CHANGELOG.md index 0df7dc8c7..8a6296e20 100644 --- a/packages/zaino-serve/CHANGELOG.md +++ b/packages/zaino-serve/CHANGELOG.md @@ -9,6 +9,8 @@ and this library adheres to Rust's notion of ### Added ### Changed +- tonic's TLS provider feature switches from `tls-ring` to `tls-aws-lc`, + following the workspace's aws-lc-rs preferred CryptoProvider (ADR-0006). ### Deprecated ### Removed ### Fixed diff --git a/packages/zaino-serve/Cargo.toml b/packages/zaino-serve/Cargo.toml index b5cee4792..c421c8a4e 100644 --- a/packages/zaino-serve/Cargo.toml +++ b/packages/zaino-serve/Cargo.toml @@ -53,9 +53,10 @@ metrics = { workspace = true, optional = true } # Miscellaneous Workspace tokio = { workspace = true, features = ["full"] } -tonic = { workspace = true, features = ["tls-native-roots"] } +tonic = { workspace = true, features = ["tls-native-roots", "tls-aws-lc"] } thiserror = { workspace = true } futures = { workspace = true } jsonrpsee = { workspace = true, features = ["server", "macros"] } serde = { workspace = true, features = ["derive"] } tower = { workspace = true } +zaino-common.workspace = true diff --git a/packages/zaino-serve/src/server/grpc.rs b/packages/zaino-serve/src/server/grpc.rs index 7d77b9ad5..e0854aea1 100644 --- a/packages/zaino-serve/src/server/grpc.rs +++ b/packages/zaino-serve/src/server/grpc.rs @@ -74,6 +74,9 @@ impl TonicServer { let mut server_builder = Server::builder(); if let Some(tls_config) = server_config.get_valid_tls().await? { + // Building the TLS acceptor requires a process-level rustls + // CryptoProvider (zingolabs/zaino#1360). + zaino_common::crypto::ensure_default_crypto_provider(); server_builder = server_builder.tls_config(tls_config).map_err(|e| { ServerError::ServerConfigError(format!("TLS configuration error: {e}")) })?; diff --git a/packages/zaino-serve/src/server/grpc/tests.rs b/packages/zaino-serve/src/server/grpc/tests.rs index 05eb1db71..946cbaa9a 100644 --- a/packages/zaino-serve/src/server/grpc/tests.rs +++ b/packages/zaino-serve/src/server/grpc/tests.rs @@ -1,3 +1,4 @@ //! Zaino-Serve gRPC server unit tests. mod spawn; +mod tls; diff --git a/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_ca_cert.pem b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_ca_cert.pem new file mode 100644 index 000000000..8c6998f49 --- /dev/null +++ b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_ca_cert.pem @@ -0,0 +1,12 @@ +-----BEGIN CERTIFICATE----- +MIIBzzCCAXWgAwIBAgIUKx27V49MTw1PXcKunTwIjHzXWuswCgYIKoZIzj0EAwIw +PDE6MDgGA1UEAwwxemFpbm8tc2VydmUgVExTIHJlZ3Jlc3Npb24tdGVzdCBDQSAo +Tk9UIEEgU0VDUkVUKTAgFw0yNjA3MDQwMTQ0MTNaGA8yMTI2MDYxMDAxNDQxM1ow +PDE6MDgGA1UEAwwxemFpbm8tc2VydmUgVExTIHJlZ3Jlc3Npb24tdGVzdCBDQSAo +Tk9UIEEgU0VDUkVUKTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABGT74i0fh1Vj +gYb0rPW36IYlyvvO/SdT0Mf8dJiFnKmoyqcXtORcw8antcjqe0myghHayx6xEOte +6blNT72knn+jUzBRMB0GA1UdDgQWBBSJ4axoCJxNd1mPygmrBgTI8IosCDAfBgNV +HSMEGDAWgBSJ4axoCJxNd1mPygmrBgTI8IosCDAPBgNVHRMBAf8EBTADAQH/MAoG +CCqGSM49BAMCA0gAMEUCIGhNgBQ5G/pars7fOccRedyOegikzDlMbdM8g4NIDErg +AiEAjlmMXJTBDW5Y/E1rLZgcC/dEdLodnTet3vatf6Vq1lM= +-----END CERTIFICATE----- diff --git a/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_cert.pem b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_cert.pem new file mode 100644 index 000000000..f78071a86 --- /dev/null +++ b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_cert.pem @@ -0,0 +1,12 @@ +-----BEGIN CERTIFICATE----- +MIIB1TCCAXqgAwIBAgIBATAKBggqhkjOPQQDAjA8MTowOAYDVQQDDDF6YWluby1z +ZXJ2ZSBUTFMgcmVncmVzc2lvbi10ZXN0IENBIChOT1QgQSBTRUNSRVQpMCAXDTI2 +MDcwNDAxNDQxM1oYDzIxMjYwNjEwMDE0NDEzWjAUMRIwEAYDVQQDDAlsb2NhbGhv +c3QwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASvy0yCNcEp5I28O0EOd7bJOZ2f +tTYE3E1AHSaMFmiiN/Foxw5KZzF7tRDFZxl0CANx+o+igLjQaerNk/bhoZepo4GS +MIGPMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAMBgNVHRMBAf8EAjAAMA4G +A1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4EFgQUtHTz +QTMs8vfbh3m4IVJIah8jAt8wHwYDVR0jBBgwFoAUieGsaAicTXdZj8oJqwYEyPCK +LAgwCgYIKoZIzj0EAwIDSQAwRgIhAIIkUaCFwsFMyAsZOasP/fVQSiU8C5jU8gmt +1xmx/2XvAiEApJiQJ1Kxdd/vjpJ5H1JYktPwoqUpZDmDKJMF1O7zMDM= +-----END CERTIFICATE----- diff --git a/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_key.pem b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_key.pem new file mode 100644 index 000000000..290fd34eb --- /dev/null +++ b/packages/zaino-serve/src/server/grpc/tests/fixtures/tls_test_key.pem @@ -0,0 +1,5 @@ +-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg6+O2+tTz29iMo5yr +SS8d+lb3nG8G7x3s4bHfvC52G3KhRANCAASvy0yCNcEp5I28O0EOd7bJOZ2ftTYE +3E1AHSaMFmiiN/Foxw5KZzF7tRDFZxl0CANx+o+igLjQaerNk/bhoZep +-----END PRIVATE KEY----- diff --git a/packages/zaino-serve/src/server/grpc/tests/tls.rs b/packages/zaino-serve/src/server/grpc/tests/tls.rs new file mode 100644 index 000000000..9b60a7567 --- /dev/null +++ b/packages/zaino-serve/src/server/grpc/tests/tls.rs @@ -0,0 +1,83 @@ +//! Regression test for zingolabs/zaino#1360. +//! +//! Spawning the gRPC server with TLS enabled must succeed and serve a +//! working TLS handshake. In 0.5.0, dependency feature-unification +//! enabled both of rustls's `ring` and `aws-lc-rs` provider features, +//! which defeats rustls's automatic provider selection: building the +//! TLS acceptor panicked at runtime ("Could not automatically determine +//! the process-level CryptoProvider") — but only on TLS-enabled +//! deployments, a path no test exercised because every existing test +//! ran plaintext. The fix pins the feature graph to a single provider — +//! `aws-lc-rs`, the preferred provider per ADR-0006 — and installs it +//! explicitly (`ensure_default_crypto_provider`); this test covers +//! the full TLS startup + handshake path so a future provider-selection +//! or certificate-wiring regression fails in CI instead of production. +//! +//! The certificate fixtures were minted once with openssl (EC P-256, +//! ~100-year expiry): a self-signed test CA plus a `localhost` leaf +//! (SAN `DNS:localhost` + `IP:127.0.0.1`) signed by it — rustls-webpki +//! rejects a single self-signed CA cert presented as the server's +//! end-entity cert (`CaUsedAsEndEntity`). The committed leaf key is not +//! a secret; the CA key was discarded, so regeneration remints all +//! three files. The server reads the leaf pair from disk (the +//! production `GrpcTls` path); the client trusts the CA as its root. + +use std::net::{Ipv4Addr, SocketAddr}; +use std::path::PathBuf; + +use tonic::service::Routes; +use tonic::transport::{server::TcpIncoming, Certificate, ClientTlsConfig, Endpoint}; + +use crate::server::config::{GrpcServerConfig, GrpcTls}; +use crate::server::grpc::TonicServer; + +fn fixture_path(name: &str) -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("src/server/grpc/tests/fixtures") + .join(name) +} + +#[tokio::test] +async fn tls_spawn_serves_a_handshake() { + // Pre-bind so the OS-assigned port is known before spawning; hand the + // open socket to `spawn_inner` directly (this module is a child of + // `grpc`, so the private constructor is reachable without the + // `test_dependencies`-gated `spawn_from_listener` wrapper). + let listener = tokio::net::TcpListener::bind(SocketAddr::from((Ipv4Addr::LOCALHOST, 0))) + .await + .expect("bind on ephemeral port must succeed"); + let local_addr = listener + .local_addr() + .expect("local_addr on a bound listener is infallible"); + + let mut server = TonicServer::spawn_inner( + Routes::default(), + GrpcServerConfig { + listen_address: local_addr, + tls: Some(GrpcTls { + cert_path: fixture_path("tls_test_cert.pem"), + key_path: fixture_path("tls_test_key.pem"), + }), + }, + TcpIncoming::from(listener), + ) + .await + .expect("TLS-enabled spawn must succeed (zingolabs/zaino#1360)"); + + let channel = Endpoint::try_from(format!("https://127.0.0.1:{}", local_addr.port())) + .expect("loopback endpoint URI is valid") + .tls_config( + ClientTlsConfig::new() + .ca_certificate(Certificate::from_pem(include_str!( + "fixtures/tls_test_ca_cert.pem" + ))) + .domain_name("localhost"), + ) + .expect("client TLS config from the fixture cert must be valid") + .connect() + .await + .expect("TLS handshake against the spawned server must succeed"); + + drop(channel); + server.close().await; +} diff --git a/packages/zaino-state/Cargo.toml b/packages/zaino-state/Cargo.toml index 4f96dbe9f..0f60eab31 100644 --- a/packages/zaino-state/Cargo.toml +++ b/packages/zaino-state/Cargo.toml @@ -83,22 +83,17 @@ lmdb = { workspace = true } lmdb-sys = { workspace = true } serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true, features = ["preserve_order"] } -prost = { workspace = true } -primitive-types = { workspace = true } blake2 = { workspace = true } sha2 = { workspace = true } corez = { workspace = true } -bs58 = { workspace = true } -nonempty = { workspace = true } arc-swap = { workspace = true } reqwest.workspace = true bitflags = { workspace = true } -derive_more = { workspace = true, features = ["from"] } [dev-dependencies] tempfile = { workspace = true } tracing-subscriber = { workspace = true } -once_cell = { workspace = true } +primitive-types = { workspace = true } zebra-chain = { workspace = true, features = ["proptest-impl"] } proptest.workspace = true incrementalmerkletree = "0.8.2" diff --git a/packages/zaino-state/src/backends.rs b/packages/zaino-state/src/backends.rs deleted file mode 100644 index 7af6467ef..000000000 --- a/packages/zaino-state/src/backends.rs +++ /dev/null @@ -1,156 +0,0 @@ -//! Zaino's chain fetch and tx submission backend services. - -pub mod fetch; - -pub mod state; - -/// Builds the gRPC [`TreeState`] shared by the Fetch and State backends from a -/// `z_gettreestate` response: hex-encoded per-pool final states (the ironwood field is -/// the empty string below NU6.3 activation, matching lightwalletd behaviour). -/// -/// [`TreeState`]: zaino_proto::proto::service::TreeState -fn tree_state_from_treestate_response( - network: String, - treestate_response: zebra_rpc::client::GetTreestateResponse, -) -> zaino_proto::proto::service::TreeState { - let sapling_tree = hex::encode( - treestate_response - .sapling() - .commitments() - .final_state() - .clone() - .unwrap_or_default(), - ); - let orchard_tree = hex::encode( - treestate_response - .orchard() - .commitments() - .final_state() - .clone() - .unwrap_or_default(), - ); - let ironwood_tree = treestate_response - .ironwood() - .clone() - .and_then(|treestate| treestate.commitments().final_state().clone()) - .map(hex::encode) - .unwrap_or_default(); - - zaino_proto::proto::service::TreeState { - network, - height: treestate_response.height().0 as u64, - hash: treestate_response.hash().to_string(), - time: treestate_response.time(), - sapling_tree, - orchard_tree, - ironwood_tree, - } -} - -/// Builds the `z_gettreestate` response shared by the Fetch and State backends from the -/// per-pool treestates the chain index reported. -/// -/// `Commitments::new(final_root, final_state)`: the note-commitment tree is the -/// `final_state`. The ironwood treestate is `Some` only from NU6.3 activation, so -/// pre-NU6.3 responses omit the field exactly as zebrad does. -fn build_treestate_response( - hash: zebra_chain::block::Hash, - height: zebra_chain::block::Height, - time: u32, - (sapling, orchard, ironwood): ( - Option, - Option, - Option, - ), -) -> zebra_rpc::client::GetTreestateResponse { - fn treestate( - pool: Option, - ) -> zebra_rpc::client::Treestate { - let (final_root, final_state) = match pool { - Some(pool) => (pool.final_root, Some(pool.final_state)), - None => (None, None), - }; - zebra_rpc::client::Treestate::new(zebra_rpc::client::Commitments::new( - final_root, - final_state, - )) - } - - let sprout_treestate = None; - let ironwood_treestate = ironwood.map(|pool| treestate(Some(pool))); - zebra_rpc::client::GetTreestateResponse::new( - hash, - height, - time, - sprout_treestate, - treestate(sapling), - treestate(orchard), - ironwood_treestate, - ) -} - -fn latest_network_upgrade( - upgrades: &indexmap::IndexMap< - zebra_rpc::methods::ConsensusBranchIdHex, - zebra_rpc::methods::NetworkUpgradeInfo, - >, -) -> Result<&zebra_rpc::methods::NetworkUpgradeInfo, tonic::Status> { - upgrades.last().map(|(_, upgrade)| upgrade).ok_or_else(|| { - tonic::Status::failed_precondition("validator returned no network upgrade metadata") - }) -} - -/// Maximum number of addresses a single `get_address_utxos` / `get_address_utxos_stream` -/// request may carry. -/// -/// Both backends resolve the full backend UTXO set before applying `max_entries` / -/// `start_height` (issue #974). A complete pushdown fix needs upstream interface changes -/// the caller-supplied entry cap cannot reach today, so until then this bounds the one -/// input the service controls locally: the address fan-out. It stops an unauthenticated -/// caller forcing an unbounded number of backend address lookups in a single request, and -/// is set well above realistic wallet usage. -/// -/// TODO: make this deployment-configurable rather than a fixed constant. -const UTXO_MAX_ADDRESSES: usize = 1000; - -/// Reject a `get_address_utxos` request whose address list exceeds [`UTXO_MAX_ADDRESSES`]. -/// -/// `max_entries` bounds the response size, not the backend work; this guard bounds the -/// address fan-out, the part the service can cap without upstream changes. -fn validate_utxo_address_count(count: usize) -> Result<(), tonic::Status> { - if count > UTXO_MAX_ADDRESSES { - return Err(tonic::Status::invalid_argument(format!( - "Error: too many addresses in request: {count} exceeds the maximum of {UTXO_MAX_ADDRESSES}." - ))); - } - Ok(()) -} - -#[cfg(test)] -mod tests { - #[test] - fn latest_network_upgrade_rejects_empty_metadata() { - let upgrades = indexmap::IndexMap::new(); - let err = super::latest_network_upgrade(&upgrades).expect_err("empty upgrades must fail"); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert_eq!( - err.message(), - "validator returned no network upgrade metadata" - ); - } - - #[test] - fn utxo_address_count_within_limit_is_accepted() { - assert!(super::validate_utxo_address_count(0).is_ok()); - assert!(super::validate_utxo_address_count(super::UTXO_MAX_ADDRESSES).is_ok()); - } - - #[test] - fn utxo_address_count_over_limit_is_rejected() { - let err = super::validate_utxo_address_count(super::UTXO_MAX_ADDRESSES + 1) - .expect_err("over-limit address count must fail"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - } -} diff --git a/packages/zaino-state/src/backends/state.rs b/packages/zaino-state/src/backends/state.rs deleted file mode 100644 index 33c19c87c..000000000 --- a/packages/zaino-state/src/backends/state.rs +++ /dev/null @@ -1,3002 +0,0 @@ -//! Zcash chain fetch and tx submission service backed by Zebras [`ReadStateService`]. - -#[allow(deprecated)] -use crate::{ - chain_index::{ - mempool::{Mempool, MempoolSubscriber}, - source::ValidatorConnector, - types as chain_types, ChainIndex, - }, - config::{DonationAddress, StateServiceConfig}, - error::{BlockCacheError, StateServiceError}, - indexer::{ - handle_raw_transaction, IndexerSubscriber, LightWalletIndexer, ZcashIndexer, ZcashService, - }, - status::{NamedAtomicStatus, Status, StatusType}, - stream::{ - AddressStream, CompactBlockStream, CompactTransactionStream, RawTransactionStream, - UtxoReplyStream, - }, - utils::{get_build_info, ServiceMetadata}, - BackendType, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, State, -}; -use crate::{ - chain_index::{types::BestChainLocation, NonFinalizedSnapshot}, - TransactionHash, -}; -use tokio_stream::StreamExt as _; -use zaino_fetch::{ - chain::{transaction::FullTransaction, utils::ParseFromSlice}, - jsonrpsee::{ - connector::{JsonRpSeeConnector, RpcError}, - raw_transaction::validate_raw_transaction_hex, - response::{ - address_deltas::{GetAddressDeltasParams, GetAddressDeltasResponse}, - block_deltas::{BlockDelta, BlockDeltas, InputDelta, OutputDelta}, - block_header::GetBlockHeader, - block_subsidy::GetBlockSubsidy, - chain_tips::GetChainTipsResponse, - mining_info::GetMiningInfoWire, - peer_info::GetPeerInfo, - z_validate_address::{ - InvalidZValidateAddress, KnownZValidateAddress, ZValidateAddressResponse, - DEPRECATION_NOTICE as Z_VALIDATE_DEPRECATION, - }, - GetMempoolInfoResponse, GetNetworkSolPsResponse, GetSpentInfoRequest, - GetSpentInfoResponse, GetSubtreesResponse, GetTxOutResponse, GetTxOutSetInfoResponse, - }, - }, -}; -use zaino_proto::proto::utils::{ - blockid_to_hashorheight, compact_block_to_nullifiers, GetBlockRangeError, PoolTypeError, - PoolTypeFilter, ValidatedBlockRangeRequest, -}; -use zaino_proto::proto::{ - compact_formats::CompactBlock, - service::{ - AddressList, Balance, BlockId, BlockRange, GetAddressUtxosArg, GetAddressUtxosReply, - GetAddressUtxosReplyList, GetMempoolTxRequest, LightdInfo, PingResponse, RawTransaction, - SendResponse, TransparentAddressBlockFilter, TreeState, TxFilter, - }, -}; -use zcash_keys::{address::Address, encoding::AddressCodec}; - -use zcash_protocol::consensus::NetworkType; -use zcash_transparent::address::TransparentAddress; -use zebra_chain::{ - amount::{Amount, NonNegative}, - block::{Header, Height, SerializedBlock}, - chain_tip::NetworkChainTipHeightEstimator, - parameters::{ConsensusBranchId, Network, NetworkKind, NetworkUpgrade}, - serialization::{BytesInDisplayOrder as _, ZcashDeserialize as _, ZcashSerialize}, - subtree::NoteCommitmentSubtreeIndex, -}; -use zebra_rpc::{ - client::{ - GetAddressBalanceRequest, GetBlockchainInfoBalance, GetSubtreesByIndexResponse, - GetTreestateResponse, HexData, Input, SubtreeRpcData, TransactionObject, - ValidateAddressResponse, - }, - methods::{ - chain_tip_difficulty, AddressBalance, ConsensusBranchIdHex, GetAddressTxIdsRequest, - GetAddressUtxos, GetBlock, GetBlockHash, GetBlockHeader as GetBlockHeaderZebra, - GetBlockHeaderObject, GetBlockTransaction, GetBlockTrees, GetBlockchainInfoResponse, - GetInfo, GetRawTransaction, NetworkUpgradeInfo, NetworkUpgradeStatus, SentTransactionHash, - TipConsensusBranch, - }, - server::error::LegacyCode, - sync::init_read_state_with_syncer, -}; -use zebra_state::{HashOrHeight, ReadRequest, ReadResponse, ReadStateService}; - -use chrono::{DateTime, Utc}; -use futures::TryFutureExt as _; -use hex::{FromHex as _, ToHex}; -use indexmap::IndexMap; -use std::{collections::HashMap, error::Error, fmt, str::FromStr, sync::Arc}; -use tokio::{ - sync::mpsc, - time::{self, timeout}, -}; -use tower::{Service, ServiceExt}; -use tracing::{info, instrument, warn}; - -macro_rules! expected_read_response { - ($response:ident, $expected_variant:ident) => { - match $response { - ReadResponse::$expected_variant(inner) => inner, - unexpected => { - unreachable!("Unexpected response from state service: {unexpected:?}") - } - } - }; -} - -/// Chain fetch service backed by Zebra's `ReadStateService` and `TrustedChainSync`. -/// -/// NOTE: We currently dop not implement clone for chain fetch services -/// as this service is responsible for maintaining and closing its child processes. -/// ServiceSubscribers are used to create separate chain fetch processes -/// while allowing central state processes to be managed in a single place. -/// If we want the ability to clone Service all JoinHandle's should be -/// converted to Arc\. -#[derive(Debug)] -// #[deprecated = "Will be eventually replaced by `BlockchainSource"] -pub struct StateService { - /// `ReadeStateService` from Zebra-State. - read_state_service: ReadStateService, - - /// Internal mempool. - mempool: Mempool, - - /// StateService config data. - #[allow(deprecated)] - config: StateServiceConfig, - - /// Listener for when the chain tip changes - chain_tip_change: zebra_state::ChainTipChange, - - /// Sync task handle. - sync_task_handle: Option>>, - - /// JsonRPC Client. - rpc_client: JsonRpSeeConnector, - - /// Core indexer. - indexer: NodeBackedChainIndex, - - /// Service metadata. - data: ServiceMetadata, - - /// Thread-safe status indicator. - status: NamedAtomicStatus, -} - -impl StateService { - #[cfg(feature = "test_dependencies")] - /// Helper for tests - pub fn read_state_service(&self) -> &ReadStateService { - &self.read_state_service - } -} - -impl Status for StateService { - fn status(&self) -> StatusType { - let current_status = self.status.load(); - if current_status == StatusType::Closing { - current_status - } else { - self.indexer.status() - } - } -} - -// #[allow(deprecated)] -impl ZcashService for StateService { - const BACKEND_TYPE: BackendType = BackendType::State; - - type Subscriber = StateServiceSubscriber; - type Config = StateServiceConfig; - - /// Initializes a new StateService instance and starts sync process. - #[instrument(name = "StateService::spawn", skip(config), fields(network = %config.common.network))] - async fn spawn(config: StateServiceConfig) -> Result { - info!( - rpc_address = %config.common.validator_rpc_address, - network = %config.common.network, - "Spawning State Service" - ); - - let rpc_client = JsonRpSeeConnector::new_from_config_parts( - &config.common.validator_rpc_address, - config.common.validator_rpc_user.clone(), - config.common.validator_rpc_password.clone(), - config.common.validator_cookie_path.clone(), - ) - .await?; - - let zebra_build_data = rpc_client.get_info().await?; - - let data = ServiceMetadata::new( - get_build_info(config.common.indexer_version.clone()), - config.common.network.to_zebra_network(), - zebra_build_data.build, - zebra_build_data.subversion, - ); - info!(build = %data.zebra_build(), subversion = %data.zebra_subversion(), "Connected to Zcash node"); - - info!( - grpc_address = %config.validator_grpc_address, - "Launching Chain Syncer" - ); - let (mut read_state_service, _latest_chain_tip, chain_tip_change, sync_task_handle) = - init_read_state_with_syncer( - config.validator_state_config.clone(), - &config.common.network.to_zebra_network(), - config.validator_grpc_address, - ) - .await??; - - info!("Chain syncer launched"); - - // Wait for ReadStateService to catch up to the validator's best chain tip. - // Height alone is insufficient during reorgs: the same height can refer to - // different blocks until JSON-RPC and ReadStateService agree on tip hash. - loop { - let blockchain_info = rpc_client.get_blockchain_info().await?; - let server_height = blockchain_info.blocks; - let server_tip_hash = blockchain_info.best_block_hash; - - let syncer_response = read_state_service - .ready() - .and_then(|service| service.call(ReadRequest::Tip)) - .await?; - let (syncer_height, syncer_tip_hash) = - expected_read_response!(syncer_response, Tip).ok_or( - RpcError::new_from_legacycode(LegacyCode::Misc, "no blocks in chain"), - )?; - - if server_height == syncer_height && server_tip_hash == syncer_tip_hash { - info!( - height = syncer_height.0, - tip_hash = %syncer_tip_hash, - "ReadStateService synced with Zebra" - ); - break; - } else { - info!( - syncer_height = syncer_height.0, - validator_height = server_height.0, - syncer_tip_hash = %syncer_tip_hash, - validator_tip_hash = %server_tip_hash, - "ReadStateService syncing with Zebra" - ); - tokio::time::sleep(std::time::Duration::from_millis(1000)).await; - continue; - } - } - - let mempool_source = ValidatorConnector::State(crate::chain_index::source::State { - read_state_service: read_state_service.clone(), - mempool_fetcher: rpc_client.clone(), - network: config.common.network, - }); - - let mempool = Mempool::spawn(mempool_source, None).await?; - - let chain_index = NodeBackedChainIndex::new( - ValidatorConnector::State(State { - read_state_service: read_state_service.clone(), - mempool_fetcher: rpc_client.clone(), - network: config.common.network, - }), - config.clone().into(), - ) - .await - .unwrap(); - - let state_service = Self { - chain_tip_change, - read_state_service, - sync_task_handle: Some(Arc::new(sync_task_handle)), - rpc_client: rpc_client.clone(), - mempool, - indexer: chain_index, - data, - config, - status: NamedAtomicStatus::new("StateService", StatusType::Spawning), - }; - - // wait for sync to complete, return error on sync fail. - loop { - match state_service.status() { - StatusType::Ready => { - state_service.status.store(StatusType::Ready); - break; - } - StatusType::CriticalError => { - return Err(StateServiceError::Critical( - "Chain index sync failed".to_string(), - )); - } - StatusType::Closing => break, - _ => { - tokio::time::sleep(tokio::time::Duration::from_millis(100)).await; - } - } - } - - Ok(state_service) - } - - fn get_subscriber(&self) -> IndexerSubscriber { - IndexerSubscriber::new(StateServiceSubscriber { - read_state_service: self.read_state_service.clone(), - rpc_client: self.rpc_client.clone(), - mempool: self.mempool.subscriber(), - indexer: self.indexer.subscriber(), - data: self.data.clone(), - config: self.config.clone(), - chain_tip_change: self.chain_tip_change.clone(), - }) - } - - /// Shuts down the StateService. - fn close(&mut self) { - if self.sync_task_handle.is_some() { - if let Some(handle) = self.sync_task_handle.take() { - handle.abort(); - } - } - } -} - -#[allow(deprecated)] -impl Drop for StateService { - fn drop(&mut self) { - self.close() - } -} - -/// A fetch service subscriber. -/// -/// Subscribers should be -#[derive(Debug, Clone)] -// #[deprecated] -pub struct StateServiceSubscriber { - /// Remote wrappper functionality for zebra's [`ReadStateService`]. - pub read_state_service: ReadStateService, - - /// Internal mempool. - pub mempool: MempoolSubscriber, - - /// StateService config data. - #[allow(deprecated)] - config: StateServiceConfig, - - /// Listener for when the chain tip changes - chain_tip_change: zebra_state::ChainTipChange, - - /// JsonRPC Client. - pub rpc_client: JsonRpSeeConnector, - - /// Core indexer. - pub indexer: NodeBackedChainIndexSubscriber, - - /// Service metadata. - pub data: ServiceMetadata, -} - -impl Status for StateServiceSubscriber { - fn status(&self) -> StatusType { - self.indexer.status() - } -} - -/// A subscriber to any chaintip updates -#[derive(Clone)] -pub struct ChainTipSubscriber { - monitor: zebra_state::ChainTipChange, -} - -impl ChainTipSubscriber { - /// Waits until the tip hash has changed (relative to the last time this method - /// was called), then returns the best tip's block hash. - pub async fn next_tip_hash( - &mut self, - ) -> Result { - self.monitor - .wait_for_tip_change() - .await - .map(|tip| tip.best_tip_hash()) - } -} - -/// Private RPC methods, which are used as helper methods by the public ones -/// -/// These would be simple to add to the public interface if -/// needed, there are currently no plans to do so. -// #[allow(deprecated)] -impl StateServiceSubscriber { - /// Gets a Subscriber to any updates to the latest chain tip - pub fn chaintip_update_subscriber(&self) -> ChainTipSubscriber { - ChainTipSubscriber { - monitor: self.chain_tip_change.clone(), - } - } - /// Returns the requested block header by hash or height, as a [`GetBlockHeader`] JSON string. - /// If the block is not in Zebra's state, - /// returns [error code `-8`.](https://github.com/zcash/zcash/issues/5758) - /// if a height was passed or -5 if a hash was passed. - /// - /// zcashd reference: [`getblockheader`](https://zcash.github.io/rpc/getblockheader.html) - /// method: post - /// tags: blockchain - /// - /// # Parameters - /// - /// - `hash_or_height`: (string, required, example="1") The hash or height - /// for the block to be returned. - /// - `verbose`: (bool, optional, default=false, example=true) false for hex encoded data, - /// true for a json object - /// - /// # Notes - /// - /// The undocumented `chainwork` field is not returned. - /// - /// This rpc is used by get_block(verbose), there is currently no - /// plan to offer this RPC publicly. - async fn get_block_header_inner( - state: &ReadStateService, - network: &Network, - hash_or_height: HashOrHeight, - verbose: Option, - ) -> Result { - let mut state = state.clone(); - let verbose = verbose.unwrap_or(true); - let network = network.clone(); - - let zebra_state::ReadResponse::BlockHeader { - header, - hash, - height, - next_block_hash, - } = state - .ready() - .and_then(|service| service.call(zebra_state::ReadRequest::BlockHeader(hash_or_height))) - .await - .map_err(|_| { - StateServiceError::RpcError(RpcError { - // Compatibility with zcashd. Note that since this function - // is reused by getblock(), we return the errors expected - // by it (they differ whether a hash or a height was passed) - code: LegacyCode::InvalidParameter as i64, - message: "block height not in best chain".to_string(), - data: None, - }) - })? - else { - return Err(StateServiceError::Custom( - "Unexpected response to BlockHeader request".to_string(), - )); - }; - - let response = if !verbose { - GetBlockHeaderZebra::Raw(HexData(header.zcash_serialize_to_vec()?)) - } else { - let zebra_state::ReadResponse::SaplingTree(sapling_tree) = state - .ready() - .and_then(|service| { - service.call(zebra_state::ReadRequest::SaplingTree(hash_or_height)) - }) - .await? - else { - return Err(StateServiceError::Custom( - "Unexpected response to SaplingTree request".to_string(), - )); - }; - // This could be `None` if there's a chain reorg between state queries. - let sapling_tree = sapling_tree.ok_or_else(|| { - StateServiceError::RpcError(zaino_fetch::jsonrpsee::connector::RpcError { - code: LegacyCode::InvalidParameter as i64, - message: "missing sapling tree for block".to_string(), - data: None, - }) - })?; - - let zebra_state::ReadResponse::Depth(depth) = state - .ready() - .and_then(|service| service.call(zebra_state::ReadRequest::Depth(hash))) - .await? - else { - return Err(StateServiceError::Custom( - "Unexpected response to Depth request".to_string(), - )); - }; - - // From - // TODO: Deduplicate const definition, consider - // refactoring this to avoid duplicate logic - const NOT_IN_BEST_CHAIN_CONFIRMATIONS: i64 = -1; - - // Confirmations are one more than the depth. - // Depth is limited by height, so it will never overflow an i64. - let confirmations = depth - .map(|depth| i64::from(depth) + 1) - .unwrap_or(NOT_IN_BEST_CHAIN_CONFIRMATIONS); - - let mut nonce = *header.nonce; - nonce.reverse(); - - let sapling_activation = crate::chain_index::ShieldedPool::Sapling - .activation_upgrade() - .activation_height(&network); - let sapling_tree_size = sapling_tree.count(); - let final_sapling_root: [u8; 32] = - if sapling_activation.is_some() && height >= sapling_activation.unwrap() { - let mut root: [u8; 32] = sapling_tree.root().into(); - root.reverse(); - root - } else { - [0; 32] - }; - - let difficulty = header.difficulty_threshold.relative_to_network(&network); - let block_commitments = - header_to_block_commitments(&header, &network, height, final_sapling_root)?; - - let block_header = GetBlockHeaderObject::new( - hash, - confirmations, - height, - header.version, - header.merkle_root, - block_commitments, - final_sapling_root, - sapling_tree_size, - header.time.timestamp(), - nonce, - header.solution, - header.difficulty_threshold, - difficulty, - header.previous_block_hash, - next_block_hash, - ); - - GetBlockHeaderZebra::Object(Box::new(block_header)) - }; - - Ok(response) - } - - /// Return a list of consecutive compact blocks. - #[allow(dead_code, deprecated)] - async fn get_block_range_inner( - &self, - request: BlockRange, - nullifiers_only: bool, - ) -> Result { - let validated_request = ValidatedBlockRangeRequest::new_from_block_range(&request) - .map_err(StateServiceError::from)?; - - let pool_type_filter = PoolTypeFilter::new_from_pool_types(&validated_request.pool_types()) - .map_err(GetBlockRangeError::PoolTypeArgumentError) - .map_err(StateServiceError::from)?; - - // Note conversion here is safe due to the use of [`ValidatedBlockRangeRequest::new_from_block_range`] - let start = validated_request.start() as u32; - let end = validated_request.end() as u32; - - let state_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - let snapshot = state_service_clone - .indexer - .snapshot_nonfinalized_state() - .await?; - - tokio::spawn(async move { - let timeout_result = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - // This method does not support passthrough. Just return. - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else {return}; - let chain_height = non_finalized_snapshot.best_tip.height.0; - - match state_service_clone - .indexer - .get_compact_block_stream( - &snapshot, - chain_types::Height(start), - chain_types::Height(end), - pool_type_filter.clone(), - ) - .await - { - Ok(Some(mut compact_block_stream)) => { - if nullifiers_only { - while let Some(stream_item) = compact_block_stream.next().await { - match stream_item { - Ok(block) => { - if channel_tx - .send(Ok(compact_block_to_nullifiers(block))) - .await - .is_err() - { - break; - } - } - Err(status) => { - if channel_tx.send(Err(status)).await.is_err() { - break; - } - } - } - } - } else { - while let Some(stream_item) = compact_block_stream.next().await { - if channel_tx.send(stream_item).await.is_err() { - break; - } - } - } - } - Ok(None) => { - // Per `get_compact_block_stream` semantics: `None` means at least one bound is above the tip. - let offending_height = if start > chain_height { start } else { end }; - - match channel_tx - .send(Err(tonic::Status::out_of_range(format!( - "Error: Height out of range [{offending_height}]. Height requested is greater than the best chain tip [{chain_height}].", - )))) - .await - { - Ok(_) => {} - Err(e) => { - warn!(%e, "GetBlockRange channel closed unexpectedly"); - } - } - } - Err(e) => { - // Preserve previous behaviour: if the request is above tip, surface OutOfRange; - // otherwise return the error (currently exposed for dev). - if start > chain_height || end > chain_height { - let offending_height = if start > chain_height { start } else { end }; - - match channel_tx - .send(Err(tonic::Status::out_of_range(format!( - "Error: Height out of range [{offending_height}]. Height requested is greater than the best chain tip [{chain_height}].", - )))) - .await - { - Ok(_) => {} - Err(e) => { - warn!(%e, "GetBlockRange channel closed unexpectedly"); - } - } - } else { - // TODO: Hide server error from clients before release. Currently useful for dev purposes. - if channel_tx - .send(Err(tonic::Status::unknown(e.to_string()))) - .await - .is_err() - { - warn!(%e, "GetBlockRangeStream closed unexpectedly"); - } - } - } - } - }, - ) - .await; - - if timeout_result.is_err() { - channel_tx - .send(Err(tonic::Status::deadline_exceeded( - "Error: get_block_range gRPC request timed out.", - ))) - .await - .ok(); - } - }); - - Ok(CompactBlockStream::new(channel_rx)) - } - - async fn error_get_block( - &self, - e: BlockCacheError, - height: u32, - ) -> Result { - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let chain_height = snapshot.max_serviceable_height().0; - Err(if height >= chain_height { - StateServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{height}]. Height requested \ - is greater than Zaino's best chain tip [{chain_height}].", - ))) - } else { - // TODO: Hide server error from clients before release. - // Currently useful for dev purposes. - StateServiceError::TonicStatusError(tonic::Status::unknown(format!( - "Error: Failed to retrieve block from node. Server Error: {e}", - ))) - }) - } - - pub(crate) async fn get_block_inner( - state: &ReadStateService, - network: &Network, - hash_or_height: HashOrHeight, - verbosity: Option, - ) -> Result { - let mut state_1 = state.clone(); - - let verbosity = verbosity.unwrap_or(1); - match verbosity { - 0 => { - let request = ReadRequest::Block(hash_or_height); - let response = state_1 - .ready() - .and_then(|service| service.call(request)) - .await?; - let block = expected_read_response!(response, Block); - block.map(SerializedBlock::from).map(GetBlock::Raw).ok_or( - StateServiceError::RpcError(RpcError::new_from_legacycode( - LegacyCode::InvalidParameter, - "block not found", - )), - ) - } - 1 | 2 => { - let state_2 = state.clone(); - let state_3 = state.clone(); - let state_4 = state.clone(); - let state_5 = state.clone(); - - let blockandsize_future = { - let req = ReadRequest::BlockAndSize(hash_or_height); - async move { state_1.ready().and_then(|service| service.call(req)).await } - }; - let orchard_future = { - let req = ReadRequest::OrchardTree(hash_or_height); - async move { - state_2 - .clone() - .ready() - .and_then(|service| service.call(req)) - .await - } - }; - let ironwood_future = { - let req = ReadRequest::IronwoodTree(hash_or_height); - async move { - state_5 - .clone() - .ready() - .and_then(|service| service.call(req)) - .await - } - }; - let block_info_future = { - let req = ReadRequest::BlockInfo(hash_or_height); - async move { - state_4 - .clone() - .ready() - .and_then(|service| service.call(req)) - .await - } - }; - let (fullblock, orchard_tree_response, ironwood_tree_response, header, block_info) = futures::join!( - blockandsize_future, - orchard_future, - ironwood_future, - StateServiceSubscriber::get_block_header_inner( - &state_3, - network, - hash_or_height, - Some(true) - ), - block_info_future - ); - - let header_obj = match header? { - GetBlockHeaderZebra::Raw(_hex_data) => unreachable!( - "`true` was passed to get_block_header, an object should be returned" - ), - GetBlockHeaderZebra::Object(get_block_header_object) => get_block_header_object, - }; - - let (transactions_response, size, block_info): (Vec, _, _) = - match (fullblock, block_info) { - ( - Ok(ReadResponse::BlockAndSize(Some((block, size)))), - Ok(ReadResponse::BlockInfo(Some(block_info))), - ) => Ok(( - block - .transactions - .iter() - .map(|transaction| { - match verbosity { - 1 => GetBlockTransaction::Hash(transaction.hash()), - 2 => GetBlockTransaction::Object(Box::new( - TransactionObject::from_transaction( - transaction.clone(), - Some(header_obj.height()), - Some(header_obj.confirmations()), - network, - DateTime::::from_timestamp( - header_obj.time(), - 0, - ), - Some(header_obj.hash()), - // block header has a non-optional height, which indicates - // a mainchain block. It is implied this method cannot return sidechain - // data, at least for now. This is subject to change: TODO - // return Some(true/false) after this assumption is resolved - None, - transaction.hash(), - ), - )), - _ => unreachable!("verbosity known to be 1 or 2"), - } - }) - .collect(), - size, - block_info, - )), - (Ok(ReadResponse::Block(None)), Ok(ReadResponse::BlockInfo(None))) => { - Err(StateServiceError::RpcError(RpcError::new_from_legacycode( - LegacyCode::InvalidParameter, - "block not found", - ))) - } - (Ok(unexpected), Ok(unexpected2)) => { - unreachable!("Unexpected responses from state service: {unexpected:?} {unexpected2:?}") - } - (Err(e), _) | (_, Err(e)) => Err(e.into()), - }?; - - let orchard_tree_response = orchard_tree_response?; - let orchard_tree = expected_read_response!(orchard_tree_response, OrchardTree) - .ok_or(StateServiceError::RpcError(RpcError::new_from_legacycode( - LegacyCode::Misc, - "missing orchard tree", - )))?; - - let final_orchard_root = match crate::chain_index::ShieldedPool::Orchard - .activation_upgrade() - .activation_height(network) - { - Some(activation_height) if header_obj.height() >= activation_height => { - Some(orchard_tree.root().into()) - } - _otherwise => None, - }; - - let ironwood_tree_response = ironwood_tree_response?; - let ironwood_tree = expected_read_response!(ironwood_tree_response, IronwoodTree) - .ok_or(StateServiceError::RpcError( - RpcError::new_from_legacycode(LegacyCode::Misc, "missing ironwood tree"), - ))?; - - let trees = GetBlockTrees::new( - header_obj.sapling_tree_size(), - orchard_tree.count(), - ironwood_tree.count(), - ); - - let (chain_supply, value_pools) = ( - GetBlockchainInfoBalance::chain_supply(*block_info.value_pools()), - GetBlockchainInfoBalance::value_pools(*block_info.value_pools(), None), - ); - let transaction_count = transactions_response.len(); - - Ok(GetBlock::Object(Box::new( - zebra_rpc::client::BlockObject::new( - header_obj.hash(), - header_obj.confirmations(), - Some(size as i64), - Some(header_obj.height()), - Some(header_obj.version()), - Some(header_obj.merkle_root()), - Some(header_obj.block_commitments()), - Some(header_obj.final_sapling_root()), - final_orchard_root, - transaction_count, - transactions_response, - Some(header_obj.time()), - Some(header_obj.nonce()), - Some(header_obj.solution()), - Some(header_obj.bits()), - Some(header_obj.difficulty()), - Some(chain_supply), - Some(value_pools), - trees, - Some(header_obj.previous_block_hash()), - header_obj.next_block_hash(), - ), - ))) - } - more_than_two => Err(StateServiceError::RpcError(RpcError::new_from_legacycode( - LegacyCode::InvalidParameter, - format!("invalid verbosity of {more_than_two}"), - ))), - } - } - - /// Returns the network type running. - #[allow(deprecated)] - pub fn network(&self) -> zaino_common::Network { - self.config.common.network - } - - /// Returns the median time of the last 11 blocks. - async fn median_time_past( - &self, - start: &zebra_rpc::client::BlockObject, - ) -> Result { - const MEDIAN_TIME_PAST_WINDOW: usize = 11; - - let mut times = Vec::with_capacity(MEDIAN_TIME_PAST_WINDOW); - - let start_hash = start.hash().to_string(); - let time_0 = start - .time() - .ok_or_else(|| MedianTimePast::StartMissingTime { - hash: start_hash.clone(), - })?; - times.push(time_0); - - let mut prev = start.previous_block_hash(); - - for _ in 0..(MEDIAN_TIME_PAST_WINDOW - 1) { - let hash = match prev { - Some(h) => h.to_string(), - None => break, // genesis - }; - - match self.z_get_block(hash.clone(), Some(1)).await { - Ok(GetBlock::Object(obj)) => { - if let Some(t) = obj.time() { - times.push(t); - } - prev = obj.previous_block_hash(); - } - Ok(GetBlock::Raw(_)) => { - return Err(MedianTimePast::UnexpectedRaw { hash }); - } - Err(_e) => { - // Use values up to this point - break; - } - } - } - - if times.is_empty() { - return Err(MedianTimePast::EmptyWindow); - } - - times.sort_unstable(); - Ok(times[times.len() / 2]) - } -} - -/// Extracts the diversifier and pk_d bytes from a validated Sapling -/// [`PaymentAddress`], returning pk_d in zcashd's big-endian byte order. -/// -/// # Deprecation -/// -/// See [`DEPRECATION_NOTICE`]. This function exists to support the -/// `z_validateaddress` RPC endpoint, which itself exists solely for zcashd -/// compatibility. The pk_d bytes are -/// reversed from `sapling-crypto`'s native little-endian representation to -/// match zcashd's big-endian hex output. -/// -/// # Precondition -/// -/// The caller must have obtained `s` through [`PaymentAddress::from_bytes`] or -/// equivalent (e.g. `ZcashAddress::convert_if_network`), which guarantees the -/// diversifier has a valid `g_d()` and pk_d is a non-identity Jubjub subgroup -/// point. No additional validation is performed here. -/// -/// # Layout -/// -/// `PaymentAddress::to_bytes()` returns 43 bytes: `diversifier (11) || pk_d (32)`. -/// `DiversifiedTransmissionKey::to_bytes()` is `pub(crate)` in `sapling-crypto`, -/// so we extract pk_d from the serialized form. -fn sapling_key_bytes(s: &sapling_crypto::PaymentAddress) -> ([u8; 11], [u8; 32]) { - let bytes = s.to_bytes(); - let diversifier: [u8; 11] = bytes[..11].try_into().unwrap(); - let mut pk_d: [u8; 32] = bytes[11..].try_into().unwrap(); - pk_d.reverse(); - (diversifier, pk_d) -} - -// #[allow(deprecated)] -impl ZcashIndexer for StateServiceSubscriber { - type Error = StateServiceError; - - async fn get_info(&self) -> Result { - // A number of these fields are difficult to access from the state service - // TODO: Fix this - self.rpc_client - .get_info() - .await - .map(GetInfo::from) - .map_err(|e| StateServiceError::Custom(e.to_string())) - } - - /// Returns all changes for an address. - /// - /// Returns information about all changes to the given transparent addresses within the given (inclusive) - /// - /// block height range, default is the full blockchain. - /// If start or end are not specified, they default to zero. - /// If start is greater than the latest block height, it's interpreted as that height. - /// - /// If end is zero, it's interpreted as the latest block height. - /// - /// [Original zcashd implementation](https://github.com/zcash/zcash/blob/18238d90cd0b810f5b07d5aaa1338126aa128c06/src/rpc/misc.cpp#L881) - /// - /// zcashd reference: [`getaddressdeltas`](https://zcash.github.io/rpc/getaddressdeltas.html) - /// method: post - /// tags: address - async fn get_address_deltas( - &self, - params: GetAddressDeltasParams, - ) -> Result { - Ok(self.indexer.get_address_deltas(params).await?) - } - - async fn get_difficulty(&self) -> Result { - chain_tip_difficulty( - self.config.common.network.to_zebra_network(), - self.read_state_service.clone(), - false, - ) - .await - .map_err(|e| { - StateServiceError::RpcError(RpcError::new_from_errorobject( - e, - "failed to get difficulty", - )) - }) - } - - async fn get_block_subsidy(&self, height: u32) -> Result { - self.rpc_client - .get_block_subsidy(height) - .await - .map_err(|e| StateServiceError::Custom(e.to_string())) - } - - async fn get_blockchain_info(&self) -> Result { - let mut state = self.read_state_service.clone(); - - let response = state - .ready() - .and_then(|service| service.call(ReadRequest::TipPoolValues)) - .await?; - let (height, hash, balance) = match response { - ReadResponse::TipPoolValues { - tip_height, - tip_hash, - value_balance, - } => (tip_height, tip_hash, value_balance), - unexpected => { - unreachable!("Unexpected response from state service: {unexpected:?}") - } - }; - - let usage_response = state - .ready() - .and_then(|service| service.call(ReadRequest::UsageInfo)) - .await?; - let size_on_disk = expected_read_response!(usage_response, UsageInfo); - - let request = zebra_state::ReadRequest::BlockHeader(hash.into()); - let response = state - .ready() - .and_then(|service| service.call(request)) - .await?; - let header = match response { - ReadResponse::BlockHeader { header, .. } => header, - unexpected => { - unreachable!("Unexpected response from state service: {unexpected:?}") - } - }; - - let now = Utc::now(); - let zebra_estimated_height = NetworkChainTipHeightEstimator::new( - header.time, - height, - &self.config.common.network.into(), - ) - .estimate_height_at(now); - let estimated_height = if header.time > now || zebra_estimated_height < height { - height - } else { - zebra_estimated_height - }; - - let upgrades = IndexMap::from_iter( - self.config - .common - .network - .to_zebra_network() - .full_activation_list() - .into_iter() - .filter_map(|(activation_height, network_upgrade)| { - // Zebra defines network upgrades based on incompatible consensus rule changes, - // but zcashd defines them based on ZIPs. - // - // All the network upgrades with a consensus branch ID - // are the same in Zebra and zcashd. - network_upgrade.branch_id().map(|branch_id| { - // zcashd's RPC seems to ignore Disabled network upgrades, - // so Zebra does too. - let status = if height >= activation_height { - NetworkUpgradeStatus::Active - } else { - NetworkUpgradeStatus::Pending - }; - - ( - ConsensusBranchIdHex::new(branch_id.into()), - NetworkUpgradeInfo::from_parts( - network_upgrade, - activation_height, - status, - ), - ) - }) - }), - ); - - let next_block_height = - (height + 1).expect("valid chain tips are a lot less than Height::MAX"); - let consensus = TipConsensusBranch::from_parts( - ConsensusBranchIdHex::new( - NetworkUpgrade::current(&self.config.common.network.into(), height) - .branch_id() - .unwrap_or(ConsensusBranchId::RPC_MISSING_ID) - .into(), - ) - .inner(), - ConsensusBranchIdHex::new( - NetworkUpgrade::current(&self.config.common.network.into(), next_block_height) - .branch_id() - .unwrap_or(ConsensusBranchId::RPC_MISSING_ID) - .into(), - ) - .inner(), - ); - - // TODO: Remove unwrap() - let difficulty = chain_tip_difficulty( - self.config.common.network.to_zebra_network(), - self.read_state_service.clone(), - false, - ) - .await - .unwrap(); - - let verification_progress = f64::from(height.0) / f64::from(zebra_estimated_height.0); - - Ok(GetBlockchainInfoResponse::new( - self.config - .common - .network - .to_zebra_network() - .bip70_network_name(), - height, - hash, - estimated_height, - zebra_rpc::client::GetBlockchainInfoBalance::chain_supply(balance), - // TODO: account for new delta_pools arg? - zebra_rpc::client::GetBlockchainInfoBalance::value_pools(balance, None), - upgrades, - consensus, - height, - difficulty, - verification_progress, - // TODO: store work in the finalized state for each height - // see https://github.com/ZcashFoundation/zebra/issues/7109 - 0, - false, - size_on_disk, - // TODO (copied from zebra): Investigate whether this needs to - // be implemented (it's sprout-only in zcashd) - 0, - )) - } - - /// Returns details on the active state of the TX memory pool. - /// In Zaino, this RPC call information is gathered from the local Zaino state instead of directly reflecting the full node's mempool. This state is populated from a gRPC stream, sourced from the full node. - /// There are no request parameters. - /// The Zcash source code is considered canonical: - /// [from the rpc definition](), [this function is called to produce the return value](>). - /// There are no required or optional parameters. - /// the `size` field is called by [this line of code](), and returns an int64. - /// `size` represents the number of transactions currently in the mempool. - /// the `bytes` field is called by [this line of code](), and returns an int64 from [this variable](). - /// `bytes` is the sum memory size in bytes of all transactions in the mempool: the sum of all transaction byte sizes. - /// the `usage` field is called by [this line of code](), and returns an int64 derived from the return of this function(), which includes a number of elements. - /// `usage` is the total memory usage for the mempool, in bytes. - /// the [optional `fullyNotified` field](), is only utilized for zcashd regtests, is deprecated, and is not included. - async fn get_mempool_info(&self) -> Result { - Ok(self.indexer.get_mempool_info().await.into()) - } - - async fn get_peer_info(&self) -> Result { - Ok(self.rpc_client.get_peer_info().await?) - } - - async fn z_get_address_balance( - &self, - address_strings: GetAddressBalanceRequest, - ) -> Result { - Ok(self.indexer.get_address_balance(address_strings).await?) - } - - async fn send_raw_transaction( - &self, - raw_transaction_hex: String, - ) -> Result { - validate_raw_transaction_hex(&raw_transaction_hex).map_err(StateServiceError::RpcError)?; - // Offload to the json rpc connector, as ReadStateService - // doesn't yet interface with the mempool - self.rpc_client - .send_raw_transaction(raw_transaction_hex) - .await - .map(SentTransactionHash::from) - .map_err(Into::into) - } - - async fn get_block_header( - &self, - hash: String, - verbose: bool, - ) -> Result { - self.rpc_client - .get_block_header(hash, verbose) - .await - .map_err(|e| StateServiceError::Custom(e.to_string())) - } - - async fn z_get_block( - &self, - hash_or_height_string: String, - verbosity: Option, - ) -> Result { - let hash_or_height = HashOrHeight::from_str(&hash_or_height_string); - - StateServiceSubscriber::get_block_inner( - &self.read_state_service.clone(), - &self.data.network(), - hash_or_height?, - verbosity, - ) - .await - } - - async fn get_block_deltas(&self, hash: String) -> Result { - // Get the block WITH the transaction data - let zblock = self.z_get_block(hash, Some(2)).await?; - - match zblock { - GetBlock::Object(boxed_block) => { - // Resolve each transparent spend's prevout ourselves: the - // verbosity-2 object from Zebra's stateless - // `TransactionObject::from_transaction` leaves input - // value/address `None`, so we look the spent output up via the - // best-chain `ReadRequest::Transaction` (finalized + - // non-finalized, so same-block prevouts resolve too). - let network = self.data.network(); - let mut state = self.read_state_service.clone(); - // Per-call cache: many inputs may reference the same prevtxid, - // so each previous transaction is fetched at most once. - let mut prevtx_cache: HashMap< - zebra_chain::transaction::Hash, - Arc, - > = HashMap::new(); - - let mut deltas = Vec::with_capacity(boxed_block.tx().len()); - for (tx_index, tx) in boxed_block.tx().iter().enumerate() { - let txo = match tx { - GetBlockTransaction::Object(txo) => txo, - GetBlockTransaction::Hash(_) => { - return Err(StateServiceError::Custom( - "Unexpected hash when expecting object".to_string(), - )) - } - }; - - let txid = txo.txid().to_string(); - - let mut inputs: Vec = Vec::new(); - for (i, vin) in txo.inputs().iter().enumerate() { - let (prevtxid, prevout) = match vin { - Input::Coinbase { .. } => continue, - Input::NonCoinbase { - txid: prevtxid, - vout: prevout, - .. - } => (prevtxid, *prevout), - }; - - let prev_hash = zebra_chain::transaction::Hash::from_str(prevtxid) - .map_err(|e| { - StateServiceError::Custom(format!( - "getblockdeltas: invalid prevout txid {prevtxid}: {e}" - )) - })?; - - let prev_tx = match prevtx_cache.get(&prev_hash) { - Some(prev_tx) => prev_tx.clone(), - None => { - let response = state - .ready() - .and_then(|service| { - service.call(ReadRequest::Transaction(prev_hash)) - }) - .await?; - let mined_tx = expected_read_response!(response, Transaction) - .ok_or_else(|| { - StateServiceError::Custom(format!( - "getblockdeltas: prevout tx {prevtxid} not in best chain" - )) - })?; - prevtx_cache.insert(prev_hash, mined_tx.tx.clone()); - mined_tx.tx - } - }; - - let output = prev_tx.outputs().get(prevout as usize).ok_or_else(|| { - StateServiceError::Custom(format!( - "getblockdeltas: prevout index {prevout} out of range for {prevtxid}" - )) - })?; - - // Nonstandard script ⇒ no derivable address ⇒ skip - // (matches the outputs branch). This is the only - // legitimate skip; it is not loss of a resolvable spend. - let address = match output.address(&network) { - Some(a) => a.to_string(), - None => continue, - }; - - // Inputs are debits, so the amount leaves the address. - let satoshis: Amount = - (-output.value().zatoshis()).try_into().map_err(|e| { - StateServiceError::Custom(format!( - "getblockdeltas: input amount out of range for {prevtxid}:{prevout}: {e}" - )) - })?; - - inputs.push(InputDelta { - address, - satoshis, - index: i as u32, - prevtxid: prevtxid.clone(), - prevout, - }); - } - - let outputs: Vec = txo - .outputs() - .iter() - .filter_map(|vout| { - let addr_opt = vout - .script_pub_key() - .addresses() - .as_ref() - .and_then(|v| if v.len() == 1 { v.first() } else { None }); - - let addr = addr_opt?.clone(); - - let output_amt: Amount = match vout.value_zat().try_into() - { - Ok(a) => a, - Err(_) => return None, - }; - - Some(OutputDelta { - address: addr, - satoshis: output_amt, - index: vout.n(), - }) - }) - .collect::>(); - - deltas.push(BlockDelta { - txid, - index: tx_index as u32, - inputs, - outputs, - }); - } - - Ok(BlockDeltas { - hash: boxed_block.hash().to_string(), - confirmations: boxed_block.confirmations(), - size: boxed_block.size().ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block size missing".into()) - })?, - height: boxed_block - .height() - .ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block height missing".into()) - })? - .0, - version: boxed_block.version().ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block version missing".into()) - })?, - merkle_root: boxed_block - .merkle_root() - .ok_or_else(|| { - StateServiceError::Custom( - "getblockdeltas: block merkle root missing".into(), - ) - })? - .encode_hex::(), - deltas, - time: boxed_block.time().ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block time missing".into()) - })?, - median_time: self.median_time_past(&boxed_block).await.map_err(|e| { - StateServiceError::Custom(format!("getblockdeltas: median_time_past: {e}")) - })?, - nonce: hex::encode(boxed_block.nonce().ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block nonce missing".into()) - })?), - bits: boxed_block - .bits() - .ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block bits missing".into()) - })? - .to_string(), - difficulty: boxed_block.difficulty().ok_or_else(|| { - StateServiceError::Custom("getblockdeltas: block difficulty missing".into()) - })?, - previous_block_hash: boxed_block - .previous_block_hash() - .map(|hash| hash.to_string()), - next_block_hash: boxed_block.next_block_hash().map(|h| h.to_string()), - }) - } - GetBlock::Raw(_serialized_block) => Err(StateServiceError::Custom( - "Unexpected raw block".to_string(), - )), - } - } - - async fn get_raw_mempool(&self) -> Result, Self::Error> { - Ok(self - .indexer - .get_mempool_txids() - .await? - .into_iter() - .map(|txid| txid.to_rpc_hex()) - .collect()) - } - - /// NOTE: This method currently has to fetch data from 2 places (get_treestate and get_indexed_block_by_*), - /// If `ValidatorConnector::GetTreeState` was updated to return the additional information - /// required, this second call could be removed, improving the performance of this method. - // Pre-existing lint: `StateServiceError` is a large error type; returning it by value here is - // flagged by `result_large_err`. Suppressed to satisfy `-D warnings` without an invasive - // boxing refactor of the shared error enum. - #[allow(clippy::result_large_err)] - async fn z_get_treestate( - &self, - hash_or_height: String, - ) -> Result { - let fallback_hash_or_height = hash_or_height.clone(); - let local_result: Result = async { - let hash_or_height_struct: HashOrHeight = HashOrHeight::from_str(&hash_or_height)?; - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - - let block_data = match hash_or_height_struct { - HashOrHeight::Hash(hash) => self - .indexer - .get_indexed_block_by_hash(&snapshot, &hash.into()) - .await? - .ok_or(StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to fetch block data.", - )))?, - HashOrHeight::Height(height) => self - .indexer - .get_indexed_block_by_height(&snapshot, &height.into()) - .await? - .ok_or(StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to fetch block data.", - )))?, - }; - - let treestates = self.indexer.get_treestate(block_data.hash()).await?; - let time: u32 = block_data.data().time().try_into().map_err(|_error| { - StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Block time is out of range for u32.", - )) - })?; - - Ok(super::build_treestate_response( - (*block_data.hash()).into(), - block_data.height().into(), - time, - treestates, - )) - } - .await; - - if let Ok(response) = local_result { - return Ok(response); - } - - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - if !self - .indexer - .hash_or_height_known_for_treestate(&snapshot, &fallback_hash_or_height) - .await? - { - return local_result; - } - - self.rpc_client - .get_treestate(fallback_hash_or_height) - .await - .map_err(|_error| { - StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to fetch treestate.", - )) - }) - .and_then(|treestate| { - treestate.try_into().map_err(|_error| { - StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to parse treestate.", - )) - }) - }) - } - - async fn get_mining_info(&self) -> Result { - Ok(self.rpc_client.get_mining_info().await?) - } - - /// Returns statistics about the unspent transaction output set. - /// - /// zcashd reference: [`gettxoutsetinfo`](https://zcash.github.io/rpc/gettxoutsetinfo.html) - /// method: post - /// tags: blockchain - async fn get_tx_out_set_info(&self) -> Result { - Ok(self.indexer.get_tx_out_set_info().await?) - } - - // No request parameters. - /// Return the hex encoded hash of the best (tip) block, in the longest block chain. - /// The Zcash source code is considered canonical: - /// [In the rpc definition](https://github.com/zcash/zcash/blob/654a8be2274aa98144c80c1ac459400eaf0eacbe/src/rpc/common.h#L48) there are no required params, or optional params. - /// [The function in rpc/blockchain.cpp](https://github.com/zcash/zcash/blob/654a8be2274aa98144c80c1ac459400eaf0eacbe/src/rpc/blockchain.cpp#L325) - /// where `return chainActive.Tip()->GetBlockHash().GetHex();` is the [return expression](https://github.com/zcash/zcash/blob/654a8be2274aa98144c80c1ac459400eaf0eacbe/src/rpc/blockchain.cpp#L339)returning a `std::string` - async fn get_best_blockhash(&self) -> Result { - // return should be valid hex encoded. - // Hash from zebra says: - // Return the hash bytes in big-endian byte-order suitable for printing out byte by byte. - // - // Zebra displays transaction and block hashes in big-endian byte-order, - // following the u256 convention set by Bitcoin and zcashd. - match self.read_state_service.best_tip() { - Some(x) => Ok(GetBlockHash::new(x.1)), - None => { - // try RPC if state read fails: - Ok(self.rpc_client.get_best_blockhash().await?.into()) - } - } - } - - /// Returns the current block count in the best valid block chain. - /// - /// zcashd reference: [`getblockcount`](https://zcash.github.io/rpc/getblockcount.html) - /// method: post - /// tags: blockchain - async fn get_block_count(&self) -> Result { - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // TODO: This probably shouldn't be an error. - // this is an improvement over previous behaviour of - // acting as if we are only synced to the genesis block - return Err(StateServiceError::UnavailableNotSyncedEnough); - }; - let h = non_finalized_snapshot.best_tip.height; - Ok(h.into()) - } - - async fn get_chain_tips(&self) -> Result { - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - return Ok(self.rpc_client.get_chain_tips().await?); - }; - - Ok(crate::chain_index::chain_tips_from_nonfinalized_snapshot( - non_finalized_snapshot, - )) - } - - async fn validate_address( - &self, - raw_address: String, - ) -> Result { - use zcash_transparent::address::TransparentAddress; - - let Ok(address) = raw_address.parse::() else { - return Ok(ValidateAddressResponse::invalid()); - }; - - let address = match address.convert_if_network::
( - match self.config.common.network.to_zebra_network().kind() { - NetworkKind::Mainnet => NetworkType::Main, - NetworkKind::Testnet => NetworkType::Test, - NetworkKind::Regtest => NetworkType::Regtest, - }, - ) { - Ok(address) => address, - Err(err) => { - tracing::debug!(?err, "conversion error"); - return Ok(ValidateAddressResponse::invalid()); - } - }; - - Ok(match address { - Address::Transparent(taddr) => ValidateAddressResponse::new( - true, - Some(raw_address), - Some(matches!(taddr, TransparentAddress::ScriptHash(_))), - ), - _ => ValidateAddressResponse::invalid(), - }) - } - - #[allow(deprecated)] - async fn z_validate_address( - &self, - address: String, - ) -> Result { - tracing::warn!("{}", Z_VALIDATE_DEPRECATION); - - let Ok(parsed_address) = address.parse::() else { - return Ok(ZValidateAddressResponse::Known( - KnownZValidateAddress::Invalid(InvalidZValidateAddress::new()), - )); - }; - - let converted_address = match parsed_address.convert_if_network::
( - match self.config.common.network.to_zebra_network().kind() { - NetworkKind::Mainnet => NetworkType::Main, - NetworkKind::Testnet => NetworkType::Test, - NetworkKind::Regtest => NetworkType::Regtest, - }, - ) { - Ok(address) => address, - Err(err) => { - tracing::debug!(?err, "conversion error"); - return Ok(ZValidateAddressResponse::Known( - KnownZValidateAddress::Invalid(InvalidZValidateAddress::new()), - )); - } - }; - - // Note: It could be the case that Zaino needs to support Sprout. For now, it's been disabled. - match converted_address { - Address::Transparent(t) => match t { - TransparentAddress::PublicKeyHash(_) => { - Ok(ZValidateAddressResponse::p2pkh(address)) - } - TransparentAddress::ScriptHash(_) => Ok(ZValidateAddressResponse::p2sh(address)), - }, - Address::Unified(u) => Ok(ZValidateAddressResponse::unified( - u.encode(&self.network().to_zebra_network()), - )), - Address::Sapling(s) => { - let (diversifier, pk_d) = sapling_key_bytes(&s); - Ok(ZValidateAddressResponse::sapling( - s.encode(&self.network().to_zebra_network()), - Some(hex::encode(diversifier)), - Some(hex::encode(pk_d)), - )) - } - _ => Ok(ZValidateAddressResponse::Known( - KnownZValidateAddress::Invalid(InvalidZValidateAddress::new()), - )), - } - } - - async fn z_get_subtrees_by_index( - &self, - pool: String, - start_index: NoteCommitmentSubtreeIndex, - limit: Option, - ) -> Result { - let mut state = self.read_state_service.clone(); - - match pool.as_str() { - "sapling" => { - let request = zebra_state::ReadRequest::SaplingSubtrees { start_index, limit }; - let response = state - .ready() - .and_then(|service| service.call(request)) - .await?; - let sapling_subtrees = expected_read_response!(response, SaplingSubtrees); - let subtrees = sapling_subtrees - .values() - .map(|subtree| { - SubtreeRpcData { - root: subtree.root.to_bytes().encode_hex(), - end_height: subtree.end_height, - } - .into() - }) - .collect(); - - Ok(GetSubtreesResponse { - pool, - start_index, - subtrees, - } - .into()) - } - "orchard" => { - let request = zebra_state::ReadRequest::OrchardSubtrees { start_index, limit }; - let response = state - .ready() - .and_then(|service| service.call(request)) - .await?; - let orchard_subtrees = expected_read_response!(response, OrchardSubtrees); - let subtrees = orchard_subtrees - .values() - .map(|subtree| { - SubtreeRpcData { - root: subtree.root.encode_hex(), - end_height: subtree.end_height, - } - .into() - }) - .collect(); - - Ok(GetSubtreesResponse { - pool, - start_index, - subtrees, - } - .into()) - } - otherwise => Err(StateServiceError::RpcError(RpcError::new_from_legacycode( - LegacyCode::Misc, - format!("invalid pool name \"{otherwise}\", must be \"sapling\" or \"orchard\""), - ))), - } - } - - async fn get_raw_transaction( - &self, - txid_hex: String, - verbose: Option, - ) -> Result { - #[allow(deprecated)] - let txid = TransactionHash::from_hex(&txid_hex).map_err(|error| { - StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidAddressOrKey, - error.to_string(), - )) - })?; - - #[allow(deprecated)] - let not_found_error = || { - StateServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidAddressOrKey, - "No such mempool or main chain transaction", - )) - }; - - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - - let Some((serialized_transaction, _consensus_branch_id)) = - self.indexer.get_raw_transaction(&snapshot, &txid).await? - else { - return Err(not_found_error()); - }; - - if verbose.is_none() { - return Ok(GetRawTransaction::Raw( - zebra_chain::transaction::SerializedTransaction::from(serialized_transaction), - )); - } - - let transaction = zebra_chain::transaction::Transaction::zcash_deserialize( - serialized_transaction.as_slice(), - ) - .map_err(|_| not_found_error())?; - - let (best_chain_location, _non_best_chain_locations) = self - .indexer - .get_transaction_status(&snapshot, &txid) - .await?; - - let (height, confirmations, block_hash, in_best_chain) = match best_chain_location { - Some(BestChainLocation::Block(block_hash, height)) => { - let confirmations: i64 = snapshot - .max_serviceable_height() - .0 - .saturating_sub(height.0) - .saturating_add(1) - .into(); - - ( - Some(zebra_chain::block::Height::from(height)), - Some(confirmations), - Some(zebra_chain::block::Hash::from(block_hash)), - Some(true), - ) - } - Some(BestChainLocation::Mempool(_height)) => (None, Some(0), None, Some(false)), - None => (None, None, None, Some(false)), - }; - - Ok(GetRawTransaction::Object(Box::new( - TransactionObject::from_transaction( - transaction.into(), - height, - confirmations, - #[allow(deprecated)] - &self.config.common.network.to_zebra_network(), - None, - block_hash, - in_best_chain, - zebra_chain::transaction::Hash::from(txid), - ), - ))) - } - - /// Returns details about an unspent transaction output. - /// - /// zcashd reference: [`gettxout`](https://zcash.github.io/rpc/gettxout.html) - /// method: post - /// tags: transaction - async fn get_tx_out( - &self, - txid: String, - n: u32, - include_mempool: Option, - ) -> Result { - Ok(self.rpc_client.get_tx_out(txid, n, include_mempool).await?) - } - - async fn get_spent_info( - &self, - request: GetSpentInfoRequest, - ) -> Result { - Ok(self.rpc_client.get_spent_info(request).await?) - } - - async fn get_address_tx_ids( - &self, - request: GetAddressTxIdsRequest, - ) -> Result, Self::Error> { - Ok(self - .indexer - .get_address_txids(request) - .await? - .into_iter() - .map(|transaction_hash| transaction_hash.to_rpc_hex()) - .collect()) - } - - async fn z_get_address_utxos( - &self, - address_strings: GetAddressBalanceRequest, - ) -> Result, Self::Error> { - Ok(self.indexer.get_address_utxos(address_strings).await?) - } - - /// Returns the estimated network solutions per second based on the last n blocks. - /// - /// zcashd reference: [`getnetworksolps`](https://zcash.github.io/rpc/getnetworksolps.html) - /// method: post - /// tags: blockchain - /// - /// This RPC is implemented in the [mining.cpp](https://github.com/zcash/zcash/blob/d00fc6f4365048339c83f463874e4d6c240b63af/src/rpc/mining.cpp#L104) - /// file of the Zcash repository. The Zebra implementation can be found [here](https://github.com/ZcashFoundation/zebra/blob/19bca3f1159f9cb9344c9944f7e1cb8d6a82a07f/zebra-rpc/src/methods.rs#L2687). - /// - /// # Parameters - /// - /// - `blocks`: (number, optional, default=120) Number of blocks, or -1 for blocks over difficulty averaging window. - /// - `height`: (number, optional, default=-1) To estimate network speed at the time of a specific block height. - async fn get_network_sol_ps( - &self, - blocks: Option, - height: Option, - ) -> Result { - self.rpc_client - .get_network_sol_ps(blocks, height) - .await - .map_err(|e| StateServiceError::Custom(e.to_string())) - } - - // Helper function, to get the chain height in rpc implementations - async fn chain_height(&self) -> Result { - let mut state = self.read_state_service.clone(); - let response = state - .ready() - .and_then(|service| service.call(ReadRequest::Tip)) - .await?; - let (chain_height, _chain_hash) = expected_read_response!(response, Tip).ok_or( - RpcError::new_from_legacycode(LegacyCode::Misc, "no blocks in chain"), - )?; - Ok(chain_height) - } -} - -// #[allow(deprecated)] -impl LightWalletIndexer for StateServiceSubscriber { - /// Return the height of the tip of the best chain - async fn get_latest_block(&self) -> Result { - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // TODO: This probably shouldn't be an error. - // this is an improvement over previous behaviour of - // acting as if we are only synced to the genesis block - return Err(StateServiceError::UnavailableNotSyncedEnough); - }; - Ok(non_finalized_snapshot.best_tip.to_wire()) - } - - /// Return the compact block corresponding to the given block identifier - async fn get_block(&self, request: BlockId) -> Result { - let hash_or_height = blockid_to_hashorheight(request).ok_or( - StateServiceError::TonicStatusError(tonic::Status::invalid_argument( - "Error: Invalid hash and/or height out of range. Failed to convert to u32.", - )), - )?; - - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - - // Convert HashOrHeight to chain_types::Height - let block_height = match hash_or_height { - HashOrHeight::Height(h) => chain_types::Height(h.0), - HashOrHeight::Hash(h) => self - .indexer - .get_block_height(&snapshot, chain_types::BlockHash(h.0)) - .await - .map_err(StateServiceError::ChainIndexError)? - .ok_or_else(|| { - StateServiceError::TonicStatusError(tonic::Status::not_found( - "Error: Block not found for given hash.", - )) - })?, - }; - - match self - .indexer - .get_compact_block(&snapshot, block_height, PoolTypeFilter::includes_all()) - .await - { - Ok(Some(block)) => Ok(block), - Ok(None) => { - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // TODO: This probably shouldn't be an error. - // this is an improvement over previous behaviour of - // acting as if we are only synced to the genesis block - return Err(StateServiceError::UnavailableNotSyncedEnough); - }; - let chain_height = non_finalized_snapshot.best_tip.height.0; - match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - StateServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ - is greater than the best chain tip [{chain_height}].", - ))), - ), - _otherwise => Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Failed to retrieve block from state.", - ))), - } - } - Err(e) => { - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // TODO: This probably shouldn't be an error. - // this is an improvement over previous behaviour of - // acting as if we are only synced to the genesis block - return Err(StateServiceError::UnavailableNotSyncedEnough); - }; - let chain_height = non_finalized_snapshot.best_tip.height.0; - match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - StateServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ - is greater than the best chain tip [{chain_height}].", - ))), - ), - _otherwise => - // TODO: Hide server error from clients before release. Currently useful for dev purposes. - { - Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - format!("Error: Failed to retrieve block from node. Server Error: {e}",), - ))) - } - } - } - } - } - - /// Same as GetBlock except actions contain only nullifiers, - /// and saling outputs are not returned (Sapling spends still are) - async fn get_block_nullifiers(&self, request: BlockId) -> Result { - let height: u32 = request.height.try_into().map_err(|_| { - StateServiceError::TonicStatusError(tonic::Status::invalid_argument( - "Error: Height out of range. Failed to convert to u32.", - )) - })?; - - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let block_height = chain_types::Height(height); - - match self - .indexer - .get_compact_block(&snapshot, block_height, PoolTypeFilter::includes_all()) - .await - { - Ok(Some(block)) => Ok(compact_block_to_nullifiers(block)), - Ok(None) => { - self.error_get_block( - BlockCacheError::Custom("Block not found".to_string()), - height, - ) - .await - } - Err(e) => Err(StateServiceError::ChainIndexError(e)), - } - } - - /// Return a list of consecutive compact blocks - async fn get_block_range( - &self, - blockrange: BlockRange, - ) -> Result { - self.get_block_range_inner(blockrange, false).await - } - /// Same as GetBlockRange except actions contain only nullifiers - async fn get_block_range_nullifiers( - &self, - request: BlockRange, - ) -> Result { - self.get_block_range_inner(request, true).await - } - - /// Return the requested full (not compact) transaction (as from zcashd) - async fn get_transaction(&self, request: TxFilter) -> Result { - let hash = zebra_chain::transaction::Hash::from( - <[u8; 32]>::try_from(request.hash).map_err(|_| { - StateServiceError::TonicStatusError(tonic::Status::invalid_argument( - "Error: Transaction hash incorrect", - )) - })?, - ); - let hex = hash.encode_hex(); - - // explicit over method call syntax to make it clear where this method is coming from - #[allow(clippy::result_large_err)] - ::get_raw_transaction(self, hex, Some(1)) - .await - .and_then(|grt| match grt { - GetRawTransaction::Raw(_serialized_transaction) => Err(StateServiceError::Custom( - "unreachable, verbose transaction expected".to_string(), - )), - GetRawTransaction::Object(transaction_object) => Ok(RawTransaction { - data: transaction_object.hex().as_ref().to_vec(), - height: transaction_object.height().unwrap_or(0) as u64, - }), - }) - } - - /// Submit the given transaction to the Zcash network - async fn send_transaction(&self, request: RawTransaction) -> Result { - let hex_tx = hex::encode(request.data); - let tx_output = self.send_raw_transaction(hex_tx).await?; - - Ok(SendResponse { - error_code: 0, - error_message: tx_output.hash().to_string(), - }) - } - - /// Return the transactions corresponding to the given t-address within the given block range - async fn get_taddress_transactions( - &self, - request: TransparentAddressBlockFilter, - ) -> Result { - let chain_height = self.chain_height().await?; - let txids = self.get_taddress_txids_helper(request).await?; - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (transmitter, receiver) = - mpsc::channel(self.config.common.service.channel_size as usize); - tokio::spawn(async move { - let timeout = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - for txid in txids { - let transaction = - fetch_service_clone.get_raw_transaction(txid, Some(1)).await; - if handle_raw_transaction::( - chain_height.0 as u64, - transaction, - transmitter.clone(), - ) - .await - .is_err() - { - break; - } - } - }, - ) - .await; - match timeout { - Ok(_) => {} - Err(_) => { - transmitter - .send(Err(tonic::Status::internal( - "Error: get_taddress_txids gRPC request timed out", - ))) - .await - .ok(); - } - } - }); - Ok(RawTransactionStream::new(receiver)) - } - - /// Return the txids corresponding to the given t-address within the given block range - /// This function is deprecated. Use `get_taddress_transactions`. - async fn get_taddress_txids( - &self, - request: TransparentAddressBlockFilter, - ) -> Result { - self.get_taddress_transactions(request).await - } - - /// Returns the total balance for a list of taddrs - async fn get_taddress_balance( - &self, - request: AddressList, - ) -> Result { - let taddrs = GetAddressBalanceRequest::new(request.addresses); - let balance = self.z_get_address_balance(taddrs).await?; - let checked_balance: i64 = match i64::try_from(balance.balance()) { - Ok(balance) => balance, - Err(_) => { - return Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Error converting balance from u64 to i64.", - ))); - } - }; - Ok(Balance { - value_zat: checked_balance, - }) - } - - /// Returns the total balance for a list of taddrs - /// - /// TODO: This is taken from fetch.rs, we could / probably should reconfigure into a trait implementation. - async fn get_taddress_balance_stream( - &self, - mut request: AddressStream, - ) -> Result { - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, mut channel_rx) = - mpsc::channel::(self.config.common.service.channel_size as usize); - let fetcher_task_handle = tokio::spawn(async move { - let fetcher_timeout = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - let mut total_balance: u64 = 0; - loop { - match channel_rx.recv().await { - Some(taddr) => { - let taddrs = GetAddressBalanceRequest::new(vec![taddr]); - let balance = - fetch_service_clone.z_get_address_balance(taddrs).await?; - total_balance += balance.balance(); - } - None => { - return Ok(total_balance); - } - } - } - }, - ) - .await; - match fetcher_timeout { - Ok(result) => result, - Err(_) => Err(tonic::Status::deadline_exceeded( - "Error: get_taddress_balance_stream request timed out.", - )), - } - }); - // NOTE: This timeout is so slow due to the blockcache not - // being implemented. This should be reduced to 30s once functionality is in place. - // TODO: Make [rpc_timout] a configurable system variable - // with [default = 30s] and [mempool_rpc_timout = 4*rpc_timeout] - let addr_recv_timeout = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - while let Some(address_result) = request.next().await { - // TODO: Hide server error from clients before release. - // Currently useful for dev purposes. - let address = address_result.map_err(|e| { - tonic::Status::unknown(format!("Failed to read from stream: {e}")) - })?; - if channel_tx.send(address.address).await.is_err() { - // TODO: Hide server error from clients before release. - // Currently useful for dev purposes. - return Err(tonic::Status::unknown( - "Error: Failed to send address to balance task.", - )); - } - } - drop(channel_tx); - Ok::<(), tonic::Status>(()) - }, - ) - .await; - match addr_recv_timeout { - Ok(Ok(())) => {} - Ok(Err(e)) => { - fetcher_task_handle.abort(); - return Err(StateServiceError::TonicStatusError(e)); - } - Err(_) => { - fetcher_task_handle.abort(); - return Err(StateServiceError::TonicStatusError( - tonic::Status::deadline_exceeded( - "Error: get_taddress_balance_stream request timed out in address loop.", - ), - )); - } - } - match fetcher_task_handle.await { - Ok(Ok(total_balance)) => { - let checked_balance: i64 = match i64::try_from(total_balance) { - Ok(balance) => balance, - Err(_) => { - // TODO: Hide server error from clients before release. - // Currently useful for dev purposes. - return Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Error converting balance from u64 to i64.", - ))); - } - }; - Ok(Balance { - value_zat: checked_balance, - }) - } - Ok(Err(e)) => Err(StateServiceError::TonicStatusError(e)), - // TODO: Hide server error from clients before release. - // Currently useful for dev purposes. - Err(e) => Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - format!("Fetcher Task failed: {e}"), - ))), - } - } - - /// Return the compact transactions currently in the mempool; the results - /// can be a few seconds out of date. If the Exclude list is empty, return - /// all transactions; otherwise return all *except* those in the Exclude list - /// (if any); this allows the client to avoid receiving transactions that it - /// already has (from an earlier call to this rpc). The transaction IDs in the - /// Exclude list can be shortened to any number of bytes to make the request - /// more bandwidth-efficient; if two or more transactions in the mempool - /// match a shortened txid, they are all sent (none is excluded). Transactions - /// in the exclude list that don't exist in the mempool are ignored. - async fn get_mempool_tx( - &self, - request: GetMempoolTxRequest, - ) -> Result { - let mut exclude_txids: Vec = vec![]; - - for (i, excluded_id) in request.exclude_txid_suffixes.iter().enumerate() { - if excluded_id.len() > 32 { - return Err(StateServiceError::TonicStatusError( - tonic::Status::invalid_argument(format!( - "Error: excluded txid {} is larger than 32 bytes", - i - )), - )); - } - - // NOTE: the TransactionHash methods cannot be used for this hex encoding as exclusions could be truncated to less than 32 bytes - let reversed_txid_bytes: Vec = excluded_id.iter().cloned().rev().collect(); - let hex_string_txid: String = hex::encode(&reversed_txid_bytes); - exclude_txids.push(hex_string_txid); - } - - let pool_types = match PoolTypeFilter::new_from_slice(&request.pool_types) { - Ok(pool_type_filter) => pool_type_filter, - Err(PoolTypeError::InvalidPoolType) => { - return Err(StateServiceError::TonicStatusError( - tonic::Status::invalid_argument( - "Error: An invalid `PoolType' was found".to_string(), - ), - )) - } - Err(PoolTypeError::UnknownPoolType(unknown_pool_type)) => { - return Err(StateServiceError::TonicStatusError( - tonic::Status::invalid_argument(format!( - "Error: Unknown `PoolType' {} was found", - unknown_pool_type - )), - )) - } - }; - - let mempool = self.mempool.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - tokio::spawn(async move { - let timeout = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - for (mempool_key, mempool_value) in - mempool.get_filtered_mempool(exclude_txids).await - { - let txid_bytes = match hex::decode(mempool_key.txid) { - Ok(bytes) => bytes, - Err(error) => { - if channel_tx - .send(Err(tonic::Status::unknown(error.to_string()))) - .await - .is_err() - { - break; - } else { - continue; - } - } - }; - match ::parse_from_slice( - mempool_value.serialized_tx.as_ref().as_ref(), - Some(vec![txid_bytes]), - None, - ) { - Ok(transaction) => { - // ParseFromSlice returns any data left after the conversion to a - // FullTransaction, If the conversion has succeeded this should be empty. - if transaction.0.is_empty() { - if channel_tx - .send( - transaction - .1 - .to_compact_tx(None, &pool_types) - .map_err(|e| tonic::Status::unknown(e.to_string())), - ) - .await - .is_err() - { - break; - } - } else { - // TODO: Hide server error from clients before release. Currently useful for dev purposes. - if channel_tx - .send(Err(tonic::Status::unknown("Error: "))) - .await - .is_err() - { - break; - } - } - } - Err(e) => { - // TODO: Hide server error from clients before release. Currently useful for dev purposes. - if channel_tx - .send(Err(tonic::Status::unknown(e.to_string()))) - .await - .is_err() - { - break; - } - } - } - } - }, - ) - .await; - match timeout { - Ok(_) => {} - Err(_) => { - channel_tx - .send(Err(tonic::Status::internal( - "Error: get_mempool_tx gRPC request timed out", - ))) - .await - .ok(); - } - } - }); - - Ok(CompactTransactionStream::new(channel_rx)) - } - - /// Return a stream of current Mempool transactions. This will keep the output stream open while - /// there are mempool transactions. It will close the returned stream when a new block is mined. - async fn get_mempool_stream(&self) -> Result { - let mut mempool = self.mempool.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // TODO: This probably shouldn't be an error. - // this is an improvement over previous behaviour of - // acting as if we are only synced to the genesis block - return Err(StateServiceError::UnavailableNotSyncedEnough); - }; - let mempool_height = non_finalized_snapshot.best_tip.height.0; - tokio::spawn(async move { - let timeout = timeout( - time::Duration::from_secs((service_timeout * 6) as u64), - async { - let (mut mempool_stream, _mempool_handle) = match mempool - .get_mempool_stream(None) - .await - { - Ok(stream) => stream, - Err(e) => { - warn!(?e, "error fetching mempool stream"); - channel_tx - .send(Err(tonic::Status::internal("Error getting mempool stream"))) - .await - .ok(); - return; - } - }; - while let Some(result) = mempool_stream.recv().await { - match result { - Ok((_mempool_key, mempool_value)) => { - if channel_tx - .send(Ok(RawTransaction { - data: mempool_value - .serialized_tx - .as_ref() - .as_ref() - .to_vec(), - height: mempool_height as u64, - })) - .await - .is_err() - { - break; - } - } - Err(e) => { - channel_tx - .send(Err(tonic::Status::internal(format!( - "Error in mempool stream: {e:?}" - )))) - .await - .ok(); - break; - } - } - } - }, - ) - .await; - match timeout { - Ok(_) => {} - Err(_) => { - channel_tx - .send(Err(tonic::Status::internal( - "Error: get_mempool_stream gRPC request timed out", - ))) - .await - .ok(); - } - } - }); - - Ok(RawTransactionStream::new(channel_rx)) - } - - /// GetTreeState returns the note commitment tree state corresponding to the given block. - /// See section 3.7 of the Zcash protocol specification. It returns several other useful - /// values also (even though they can be obtained using GetBlock). - /// The block can be specified by either height or hash. - async fn get_tree_state(&self, request: BlockId) -> Result { - let hash_or_height = blockid_to_hashorheight(request).ok_or( - crate::error::StateServiceError::TonicStatusError(tonic::Status::invalid_argument( - "Invalid hash or height", - )), - )?; - - let treestate_response = ::z_get_treestate( - self, - hash_or_height.to_string(), - ) - .await?; - - Ok(super::tree_state_from_treestate_response( - self.config - .common - .network - .to_zebra_network() - .bip70_network_name(), - treestate_response, - )) - } - - /// GetLatestTreeState returns the note commitment tree state corresponding to the chain tip. - async fn get_latest_tree_state(&self) -> Result { - let latest_block = self.chain_height().await?; - self.get_tree_state(BlockId { - height: latest_block.0 as u64, - hash: vec![], - }) - .await - } - - fn timeout_channel_size(&self) -> (u32, u32) { - ( - self.config.common.service.timeout, - self.config.common.service.channel_size, - ) - } - - /// Returns all unspent outputs for a list of addresses. - /// - /// Ignores all utxos below block height [GetAddressUtxosArg.start_height]. - /// Returns max [GetAddressUtxosArg.max_entries] utxos, or unrestricted if - /// [GetAddressUtxosArg.max_entries] = 0. - /// Utxos are collected and returned as a single Vec. - async fn get_address_utxos( - &self, - request: GetAddressUtxosArg, - ) -> Result { - super::validate_utxo_address_count(request.addresses.len())?; - let taddrs = GetAddressBalanceRequest::new(request.addresses); - let utxos = self.z_get_address_utxos(taddrs).await?; - let mut address_utxos: Vec = Vec::new(); - let mut entries: u32 = 0; - for utxo in utxos { - let (address, tx_hash, output_index, script, satoshis, height) = utxo.into_parts(); - if (height.0 as u64) < request.start_height { - continue; - } - entries += 1; - if request.max_entries > 0 && entries > request.max_entries { - break; - } - let checked_index = match i32::try_from(output_index.index()) { - Ok(index) => index, - Err(_) => { - return Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Index out of range. Failed to convert to i32.", - ))); - } - }; - let checked_satoshis = match i64::try_from(satoshis) { - Ok(satoshis) => satoshis, - Err(_) => { - return Err(StateServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Satoshis out of range. Failed to convert to i64.", - ))); - } - }; - let utxo_reply = GetAddressUtxosReply { - address: address.to_string(), - txid: tx_hash.0.to_vec(), - index: checked_index, - script: script.as_raw_bytes().to_vec(), - value_zat: checked_satoshis, - height: height.0 as u64, - }; - address_utxos.push(utxo_reply) - } - Ok(GetAddressUtxosReplyList { address_utxos }) - } - - /// Returns all unspent outputs for a list of addresses. - /// - /// Ignores all utxos below block height [GetAddressUtxosArg.start_height]. - /// Returns max [GetAddressUtxosArg.max_entries] utxos, or unrestricted if - /// [GetAddressUtxosArg.max_entries] = 0. - /// Utxos are returned in a stream. - async fn get_address_utxos_stream( - &self, - request: GetAddressUtxosArg, - ) -> Result { - super::validate_utxo_address_count(request.addresses.len())?; - let taddrs = GetAddressBalanceRequest::new(request.addresses); - let utxos = self.z_get_address_utxos(taddrs).await?; - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - tokio::spawn(async move { - let timeout = timeout( - time::Duration::from_secs((service_timeout * 4) as u64), - async { - let mut entries: u32 = 0; - for utxo in utxos { - let (address, tx_hash, output_index, script, satoshis, height) = - utxo.into_parts(); - if (height.0 as u64) < request.start_height { - continue; - } - entries += 1; - if request.max_entries > 0 && entries > request.max_entries { - break; - } - let checked_index = match i32::try_from(output_index.index()) { - Ok(index) => index, - Err(_) => { - let _ = channel_tx - .send(Err(tonic::Status::unknown( - "Error: Index out of range. Failed to convert to i32.", - ))) - .await; - return; - } - }; - let checked_satoshis = match i64::try_from(satoshis) { - Ok(satoshis) => satoshis, - Err(_) => { - let _ = channel_tx - .send(Err(tonic::Status::unknown( - "Error: Satoshis out of range. Failed to convert to i64.", - ))) - .await; - return; - } - }; - let utxo_reply = GetAddressUtxosReply { - address: address.to_string(), - txid: tx_hash.0.to_vec(), - index: checked_index, - script: script.as_raw_bytes().to_vec(), - value_zat: checked_satoshis, - height: height.0 as u64, - }; - if channel_tx.send(Ok(utxo_reply)).await.is_err() { - return; - } - } - }, - ) - .await; - match timeout { - Ok(_) => {} - Err(_) => { - channel_tx - .send(Err(tonic::Status::deadline_exceeded( - "Error: get_mempool_stream gRPC request timed out", - ))) - .await - .ok(); - } - } - }); - Ok(UtxoReplyStream::new(channel_rx)) - } - - /// Return information about this lightwalletd instance and the blockchain - /// - /// TODO: This could be made more efficient by fetching data directly (not using self.get_blockchain_info()) - async fn get_lightd_info(&self) -> Result { - let blockchain_info = self.get_blockchain_info().await?; - let sapling_id = zebra_rpc::methods::ConsensusBranchIdHex::new( - zebra_chain::parameters::ConsensusBranchId::from_hex("76b809bb") - .map_err(|_e| { - tonic::Status::internal( - "Internal Error - Consesnsus Branch ID hex conversion failed", - ) - })? - .into(), - ); - let sapling_activation_height = blockchain_info - .upgrades() - .get(&sapling_id) - .map_or(Height(1), |sapling_json| sapling_json.into_parts().1); - - let consensus_branch_id = zebra_chain::parameters::ConsensusBranchId::from( - blockchain_info.consensus().into_parts().0, - ) - .to_string(); - - let latest_upgrade = super::latest_network_upgrade(blockchain_info.upgrades()) - .map_err(StateServiceError::TonicStatusError)? - .into_parts(); - - let nu_name = latest_upgrade.0; - let nu_height = latest_upgrade.1; - - Ok(LightdInfo { - version: self.data.build_info().version(), - vendor: "ZingoLabs ZainoD".to_string(), - taddr_support: true, - chain_name: blockchain_info.chain().clone(), - sapling_activation_height: sapling_activation_height.0 as u64, - consensus_branch_id, - block_height: blockchain_info.blocks().0 as u64, - git_commit: self.data.build_info().commit_hash(), - branch: self.data.build_info().branch(), - build_date: self.data.build_info().build_date(), - build_user: self.data.build_info().build_user(), - estimated_height: blockchain_info.estimated_height().0 as u64, - zcashd_build: self.data.zebra_build(), - zcashd_subversion: self.data.zebra_subversion(), - donation_address: self - .config - .common - .donation_address - .as_ref() - .map(DonationAddress::encode) - .unwrap_or_default(), - upgrade_name: nu_name.to_string(), - upgrade_height: nu_height.0 as u64, - lightwallet_protocol_version: "v0.5.0".to_string(), - }) - } - - /// Testing-only, requires lightwalletd --ping-very-insecure (do not enable in production) - /// - /// NOTE: Currently unimplemented in Zaino. - async fn ping( - &self, - _request: zaino_proto::proto::service::Duration, - ) -> Result { - Err(crate::error::StateServiceError::TonicStatusError( - tonic::Status::unimplemented( - "Ping not yet implemented. If you require this RPC please open an \ - issue or PR at the Zaino github (https://github.com/zingolabs/zaino.git).", - ), - )) - } -} - -#[allow(clippy::result_large_err, deprecated)] -fn header_to_block_commitments( - header: &Header, - network: &Network, - height: Height, - final_sapling_root: [u8; 32], -) -> Result<[u8; 32], StateServiceError> { - let hash = match header.commitment(network, height).map_err(|e| { - StateServiceError::SerializationError( - zebra_chain::serialization::SerializationError::Parse( - // For some reason this error type takes a - // &'static str, and the the only way to create one - // dynamically is to leak a String. This shouldn't - // be a concern, as this error case should - // never occur when communing with a zebrad, which - // validates this field before serializing it - e.to_string().leak(), - ), - ) - })? { - zebra_chain::block::Commitment::PreSaplingReserved(bytes) => bytes, - zebra_chain::block::Commitment::FinalSaplingRoot(_root) => final_sapling_root, - zebra_chain::block::Commitment::ChainHistoryActivationReserved => [0; 32], - zebra_chain::block::Commitment::ChainHistoryRoot(root) => root.bytes_in_display_order(), - zebra_chain::block::Commitment::ChainHistoryBlockTxAuthCommitment(hash) => { - hash.bytes_in_display_order() - } - }; - Ok(hash) -} - -/// An error type for median time past calculation errors -#[derive(Debug, Clone)] -pub enum MedianTimePast { - /// The start block has no `time`. - StartMissingTime { hash: String }, - - /// Ignored verbosity. - UnexpectedRaw { hash: String }, - - /// No timestamps collected at all. - EmptyWindow, -} - -impl fmt::Display for MedianTimePast { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - MedianTimePast::StartMissingTime { hash } => { - write!(f, "start block {hash} is missing `time`") - } - MedianTimePast::UnexpectedRaw { hash } => { - write!(f, "unexpected raw payload for block {hash}") - } - MedianTimePast::EmptyWindow => { - write!(f, "no timestamps collected (empty MTP window)") - } - } - } -} - -impl Error for MedianTimePast {} - -#[cfg(test)] -mod tests { - /// Classifies the byte-level relationship between two slices. - #[derive(Debug, PartialEq)] - enum ByteRelation { - /// The slices are identical. - Equal, - /// `actual` fully byte-reversed equals `expected` (endian swap). - FullByteReversal, - /// Each byte's bits reversed maps `actual` to `expected`. - PerByteBitReversal, - /// Reversing bytes within 16-bit chunks maps `actual` to `expected`. - ChunkSwap16, - /// Reversing bytes within 32-bit chunks maps `actual` to `expected`. - ChunkSwap32, - /// Reversing bytes within 64-bit chunks maps `actual` to `expected`. - ChunkSwap64, - /// No recognized transformation. - Unrecognized, - } - - impl std::fmt::Display for ByteRelation { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - Self::Equal => write!(f, "equal"), - Self::FullByteReversal => write!(f, "full byte-reversal (endian swap)"), - Self::PerByteBitReversal => write!(f, "per-byte bit-reversal"), - Self::ChunkSwap16 => write!(f, "16-bit pairwise byte-swap"), - Self::ChunkSwap32 => write!(f, "32-bit chunk byte-reversal"), - Self::ChunkSwap64 => write!(f, "64-bit chunk byte-reversal"), - Self::Unrecognized => write!(f, "unrecognized mismatch"), - } - } - } - - /// Applies each candidate byte transformation to `actual` and returns - /// the first that produces `expected`, or [`ByteRelation::Unrecognized`]. - // `u32::is_multiple_of` is only stable from Rust 1.87; keep `% n == 0` for our older MSRV. - #[allow(clippy::manual_is_multiple_of)] - fn classify_byte_relation(actual: &[u8], expected: &[u8]) -> ByteRelation { - if actual == expected { - return ByteRelation::Equal; - } - - let chunk_swap = |size: usize| -> Vec { - actual - .chunks(size) - .flat_map(|c| c.iter().rev()) - .copied() - .collect() - }; - - let mut reversed = actual.to_vec(); - reversed.reverse(); - if reversed == expected { - return ByteRelation::FullByteReversal; - } - - let bit_reversed: Vec = actual.iter().map(|b| b.reverse_bits()).collect(); - if bit_reversed == expected { - return ByteRelation::PerByteBitReversal; - } - - if actual.len() % 2 == 0 && chunk_swap(2) == expected { - return ByteRelation::ChunkSwap16; - } - if actual.len() % 4 == 0 && chunk_swap(4) == expected { - return ByteRelation::ChunkSwap32; - } - if actual.len() % 8 == 0 && chunk_swap(8) == expected { - return ByteRelation::ChunkSwap64; - } - - ByteRelation::Unrecognized - } - - /// Verifies that our Sapling address parsing logic produces the same - /// diversifier and diversified transmission key (pk_d) hex strings as - /// zcashd's `z_validateaddress` RPC. - /// - /// # Guarantees - /// - /// - Exercises the production `sapling_key_bytes` function directly. - /// - The 11-byte diversifier matches the zcashd-derived test vector. - /// - The 32-byte pk_d (after the endian reversal inside `sapling_key_bytes`) - /// matches the zcashd-derived test vector. - /// - If the upstream serialization changes, the failure message - /// classifies the mismatch (endian swap, bit-reversal, chunk swap, - /// or unrecognized) to aid diagnosis. - /// - /// # Non-guarantees - /// - /// - Does not prove the test vector constants themselves are correct; - /// they were captured from zcashd and are trusted as ground truth. - /// - Does not exercise the full `z_validate_address` RPC path through - /// `StateService` — only the `sapling_key_bytes` extraction function. - /// - Does not verify behavior for malformed Sapling addresses or - /// addresses on other networks (mainnet, testnet). - #[test] - fn sapling_pk_d_byte_order_matches_test_vector() { - use super::sapling_key_bytes; - use zcash_keys::address::Address; - use zcash_protocol::consensus::NetworkType; - - // Canonical source: live-tests/clientless/src/lib.rs::rpc::json_rpc - // Tracked for DRY consolidation: https://github.com/zingolabs/zaino/issues/988 - const SAPLING_ADDRESS: &str = "zregtestsapling1jalqhycwumq3unfxlzyzcktq3n478n82k2wacvl8gwfxk6ahshkxmtp2034qj28n7gl92ka5wca"; - const EXPECTED_DIVERSIFIER: &str = "977e0b930ee6c11e4d26f8"; - const EXPECTED_PK_D: &str = - "553ef2f328096a7c2aac6dec85b76b6b9243e733dc9db2eacce3eb8c60592c88"; - - let parsed: zcash_address::ZcashAddress = SAPLING_ADDRESS.parse().unwrap(); - let converted = parsed - .convert_if_network::
(NetworkType::Regtest) - .unwrap(); - - let Address::Sapling(s) = converted else { - panic!("expected Sapling address"); - }; - - let (diversifier, pk_d) = sapling_key_bytes(&s); - - let expected_diversifier = hex::decode(EXPECTED_DIVERSIFIER).unwrap(); - let expected_pk_d = hex::decode(EXPECTED_PK_D).unwrap(); - - // Diversifier - match classify_byte_relation(&diversifier, &expected_diversifier) { - ByteRelation::Equal => {} - relation => panic!( - "diversifier mismatch.\n relation: {relation}\n actual: {}\n expected: {}", - hex::encode(diversifier), - hex::encode(expected_diversifier), - ), - } - - // pk_d (sapling_key_bytes already applies the endian reversal) - match classify_byte_relation(&pk_d, &expected_pk_d) { - ByteRelation::Equal => {} - relation => panic!( - "pk_d mismatch — upstream serialization may have changed.\ - \n relation: {relation}\n actual: {}\n expected: {}", - hex::encode(pk_d), - hex::encode(expected_pk_d), - ), - } - } - - #[test] - fn classify_byte_relation_detects_known_transforms() { - let original = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08]; - - assert_eq!( - classify_byte_relation(&original, &original), - ByteRelation::Equal, - ); - - let mut reversed = original.to_vec(); - reversed.reverse(); - assert_eq!( - classify_byte_relation(&original, &reversed), - ByteRelation::FullByteReversal, - ); - - let bit_rev: Vec = original.iter().map(|b| b.reverse_bits()).collect(); - assert_eq!( - classify_byte_relation(&original, &bit_rev), - ByteRelation::PerByteBitReversal, - ); - - let swapped_16: Vec = original - .chunks(2) - .flat_map(|c| c.iter().rev()) - .copied() - .collect(); - assert_eq!( - classify_byte_relation(&original, &swapped_16), - ByteRelation::ChunkSwap16, - ); - - let garbage = [0xFF; 8]; - assert_eq!( - classify_byte_relation(&original, &garbage), - ByteRelation::Unrecognized, - ); - } -} diff --git a/packages/zaino-state/src/chain_index.rs b/packages/zaino-state/src/chain_index.rs index 4609c0011..1571d3b8a 100644 --- a/packages/zaino-state/src/chain_index.rs +++ b/packages/zaino-state/src/chain_index.rs @@ -37,10 +37,17 @@ use source::{BlockchainSource, ValidatorConnector}; use tokio_stream::StreamExt; use tokio_util::sync::CancellationToken; use tracing::{info, instrument}; +use zaino_fetch::jsonrpsee::raw_transaction::validate_raw_transaction_hex; use zaino_fetch::jsonrpsee::response::{ address_deltas::{GetAddressDeltasParams, GetAddressDeltasResponse}, + block_deltas::BlockDeltas, + block_header::GetBlockHeader, + block_subsidy::GetBlockSubsidy, chain_tips::{ChainTip, ChainTipStatus, GetChainTipsResponse}, - EmptyTxOutSetInfo, GetTxOutSetInfo, GetTxOutSetInfoResponse, + mining_info::GetMiningInfoWire, + peer_info::GetPeerInfo, + EmptyTxOutSetInfo, GetNetworkSolPsResponse, GetSpentInfoRequest, GetSpentInfoResponse, + GetTxOutResponse, GetTxOutSetInfo, GetTxOutSetInfoResponse, }; use zaino_proto::proto::utils::{compact_block_with_pool_types, PoolTypeFilter}; use zebra_chain::parameters::ConsensusBranchId; @@ -48,7 +55,10 @@ pub use zebra_chain::parameters::Network as ZebraNetwork; use zebra_chain::serialization::ZcashSerialize; use zebra_rpc::{ client::{GetAddressBalanceRequest, GetAddressTxIdsRequest}, - methods::{AddressBalance, GetAddressUtxos}, + methods::{ + AddressBalance, GetAddressUtxos, GetBlock, GetBlockchainInfoResponse, GetInfo, + SentTransactionHash, + }, }; use zebra_state::HashOrHeight; @@ -313,6 +323,14 @@ fn branch_len_to_active_chain( /// /// When a call asks for info (e.g. a block), Zaino selects sources in this order: #[doc = simple_mermaid::mermaid!("chain_index_passthrough.mmd")] +/// +/// This trait holds the core methods required by the embedded wallet consumer +/// (zallet). RPC-server-only methods live on the [`ChainIndexRpcExt`] extension +/// trait. +/// +/// TODO: The core/extension split is a provisional first pass. It should be refined +/// into finer capability-based traits (zallet / lwd / block-explorer) in a follow-up +/// PR. pub trait ChainIndex { /// A snapshot of the nonfinalized state, needed for atomic access type Snapshot: NonFinalizedSnapshot; @@ -385,50 +403,6 @@ pub trait ChainIndex { end: Option, ) -> Option, Self::Error>>>; - /// Returns the *compact* block for the given height. - /// - /// Returns `None` if the specified `height` is greater than the snapshot's tip. - /// - /// ## Pool filtering - /// - /// - `pool_types` controls which per-transaction components are populated. - /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. - /// The original transaction index is preserved in `CompactTx.index`. - /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard - /// components are populated). - #[allow(clippy::type_complexity)] - fn get_compact_block( - &self, - nonfinalized_snapshot: &Self::Snapshot, - height: types::Height, - pool_types: PoolTypeFilter, - ) -> impl std::future::Future< - Output = Result, Self::Error>, - >; - - /// Streams *compact* blocks for an inclusive height range. - /// - /// Returns `None` if the requested range is entirely above the snapshot's tip. - /// - /// - The stream covers `[start_height, end_height]` (inclusive). - /// - If `start_height <= end_height` the stream is ascending; otherwise it is descending. - /// - /// ## Pool filtering - /// - /// - `pool_types` controls which per-transaction components are populated. - /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. - /// The original transaction index is preserved in `CompactTx.index`. - /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard - /// components are populated). - #[allow(clippy::type_complexity)] - fn get_compact_block_stream( - &self, - nonfinalized_snapshot: &Self::Snapshot, - start_height: types::Height, - end_height: types::Height, - pool_types: PoolTypeFilter, - ) -> impl std::future::Future, Self::Error>>; - // ********** Transaction methods ********** /// given a transaction id, returns the transaction, along with @@ -522,12 +496,6 @@ pub trait ChainIndex { // ********** Transparent address history methods ********** - /// Returns all changes for the given transparent addresses. - fn get_address_deltas( - &self, - params: GetAddressDeltasParams, - ) -> impl std::future::Future>; - /// Returns the total transparent balance for the given addresses. fn get_address_balance( &self, @@ -563,6 +531,169 @@ pub trait ChainIndex { outpoints: Vec, scope: types::ChainScope, ) -> impl std::future::Future>, Self::Error>>; +} + +/// RPC-server extension methods layered on top of [`ChainIndex`]. +/// +/// The core [`ChainIndex`] trait holds the subset required by the embedded wallet +/// consumer (zallet). This extension holds the additional functionality required by +/// the gRPC (lightwalletd) and JSON-RPC servers: compact-block serving, mempool +/// metadata, address deltas, and the block-explorer / node-passthrough RPCs. +/// +/// TODO: This two-way core/extension split is a provisional first pass. It should be +/// refined into finer capability-based traits (zallet / lwd / block-explorer) in a +/// follow-up PR, at which point methods will be redistributed to their narrowest +/// capability. +pub trait ChainIndexRpcExt: ChainIndex { + // ********** Block methods ********** + + /// Returns the *compact* block for the given height. + /// + /// Returns `None` if the specified `height` is greater than the snapshot's tip. + /// + /// ## Pool filtering + /// + /// - `pool_types` controls which per-transaction components are populated. + /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. + /// The original transaction index is preserved in `CompactTx.index`. + /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard + /// components are populated). + #[allow(clippy::type_complexity)] + fn get_compact_block( + &self, + nonfinalized_snapshot: &Self::Snapshot, + height: types::Height, + pool_types: PoolTypeFilter, + ) -> impl std::future::Future< + Output = Result, Self::Error>, + >; + + /// Streams *compact* blocks for an inclusive height range. + /// + /// Returns `None` if the requested range is entirely above the snapshot's tip. + /// + /// - The stream covers `[start_height, end_height]` (inclusive). + /// - If `start_height <= end_height` the stream is ascending; otherwise it is descending. + /// + /// ## Pool filtering + /// + /// - `pool_types` controls which per-transaction components are populated. + /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. + /// The original transaction index is preserved in `CompactTx.index`. + /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard + /// components are populated). + #[allow(clippy::type_complexity)] + fn get_compact_block_stream( + &self, + nonfinalized_snapshot: &Self::Snapshot, + start_height: types::Height, + end_height: types::Height, + pool_types: PoolTypeFilter, + ) -> impl std::future::Future, Self::Error>>; + + /// Returns the `getblock`-shaped block for the given hash-or-height string. + /// + /// `verbosity` follows the zcashd `getblock` convention (0 = raw, 1 = object with + /// txids, 2 = object with full transaction data). + /// + /// zcashd reference: [`getblock`](https://zcash.github.io/rpc/getblock.html) + fn z_get_block( + &self, + hash_or_height: String, + verbosity: Option, + ) -> impl std::future::Future>; + + /// Returns the `getblockheader`-shaped header for the given block hash. + /// + /// zcashd reference: [`getblockheader`](https://zcash.github.io/rpc/getblockheader.html) + fn get_block_header( + &self, + hash: String, + verbose: bool, + ) -> impl std::future::Future>; + + /// Returns the `getblockdeltas`-shaped transparent input/output deltas for the block + /// with the given hash. + /// + /// zcashd reference: [`getblockdeltas`](https://zcash.github.io/rpc/getblockdeltas.html) + fn get_block_deltas( + &self, + hash: String, + ) -> impl std::future::Future>; + + /// Returns the proof-of-work difficulty of the best chain as a multiple of the + /// minimum difficulty. + /// + /// zcashd reference: [`getdifficulty`](https://zcash.github.io/rpc/getdifficulty.html) + fn get_difficulty(&self) -> impl std::future::Future>; + + // ********** Node-passthrough methods ********** + // + // No local-index equivalent; always delegate to the backing validator. + + /// Returns the `getinfo` response. + fn get_info(&self) -> impl std::future::Future>; + + /// Returns the `getblockchaininfo` response. + fn get_blockchain_info( + &self, + ) -> impl std::future::Future>; + + /// Returns the `getpeerinfo` response. + fn get_peer_info(&self) -> impl std::future::Future>; + + /// Returns the `getblocksubsidy` response at the given height. + fn get_block_subsidy( + &self, + height: u32, + ) -> impl std::future::Future>; + + /// Returns the `getmininginfo` response. + fn get_mining_info( + &self, + ) -> impl std::future::Future>; + + /// Returns the `gettxout` response for the given outpoint. + fn get_tx_out( + &self, + txid: String, + n: u32, + include_mempool: Option, + ) -> impl std::future::Future>; + + /// Returns the `getspentinfo` response for the given request. + fn get_spent_info( + &self, + request: GetSpentInfoRequest, + ) -> impl std::future::Future>; + + /// Returns the `getnetworksolps` response. + fn get_network_sol_ps( + &self, + blocks: Option, + height: Option, + ) -> impl std::future::Future>; + + /// Submits a raw transaction to the network (`sendrawtransaction`). + fn send_raw_transaction( + &self, + raw_transaction_hex: String, + ) -> impl std::future::Future>; + + /// Returns the full `z_gettreestate` response for the given hash-or-height, via the + /// backing validator (node-passthrough fallback for treestates not locally serviceable). + fn get_treestate_by_id( + &self, + hash_or_height: String, + ) -> impl std::future::Future>; + + // ********** Transparent address history methods ********** + + /// Returns all changes for the given transparent addresses. + fn get_address_deltas( + &self, + params: GetAddressDeltasParams, + ) -> impl std::future::Future>; // ********** Metadata methods ********** @@ -809,7 +940,7 @@ impl NodeBackedChainIndex { finalized_db, sync_loop_handle: None, status: NamedAtomicStatus::new("ChainIndex", StatusType::Spawning), - network: config.network.to_zebra_network(), + network: config.network.clone(), source, sync_timings, cancel_token: CancellationToken::new(), @@ -845,11 +976,24 @@ impl NodeBackedChainIndex { /// the design we have. Cancelling first just removes the wasted /// failure-path round trip. pub async fn shutdown(&self) -> Result<(), FinalisedStateError> { + // The synchronous teardown (cancellation, mempool close, source + // release) runs before the fallible DB shutdown so a DB error cannot + // skip it — the source's Zebra syncer task must not outlive the index. + self.shutdown_sync_best_effort(); + self.finalized_db.shutdown().await + } + + /// Synchronous best-effort teardown for contexts that cannot run async + /// work (a `Drop` on a current-thread runtime, or on a thread with no + /// runtime at all): cancels the sync worker, closes the mempool, and + /// releases source-owned background work. The finalised DB's async + /// [`shutdown`](Self::shutdown) step is skipped; the DB's own `Drop` + /// remains responsible for releasing its resources. + pub(crate) fn shutdown_sync_best_effort(&self) { self.cancel_token.cancel(); self.status.store(StatusType::Closing); - self.finalized_db.shutdown().await?; self.mempool.close(); - Ok(()) + self.source.shutdown(); } /// Displays the status of the chain_index @@ -1075,7 +1219,11 @@ impl Drop for NodeBackedChainIndex { /// on `cancel_token.cancelled()` wrapping the iter body), the worker /// exits at its next await checkpoint instead. fn drop(&mut self) { - self.cancel_token.cancel(); + // The full synchronous teardown, not just the cancellation: the + // source release in particular must run here, or the `State` + // connector's Zebra syncer task outlives an index that is dropped + // without an explicit `shutdown()` call. + self.shutdown_sync_best_effort(); } } @@ -1180,10 +1328,20 @@ async fn compact_block_from_source( } impl NodeBackedChainIndexSubscriber { - fn source(&self) -> &Source { + pub(crate) fn source(&self) -> &Source { &self.source } + /// The indexer's mempool subscriber. + /// + /// Test-only escape hatch: live tests recompute expected `getmempoolinfo` + /// values directly off the mempool's entries. Production code goes through + /// the `ChainIndex` mempool API. + #[cfg(feature = "test_dependencies")] + pub(crate) fn mempool_subscriber(&self) -> &mempool::MempoolSubscriber { + &self.mempool + } + /// Returns the combined status of all chain index components. pub fn combined_status(&self) -> StatusType { let finalized_status = self.finalized_state.status(); @@ -1704,877 +1862,1063 @@ impl ChainIndex for NodeBackedChainIndexSubscriber Result, Option)>, Self::Error> { + // ChainIndex step 1 + if let Some(mempool_tx) = self + .mempool + .get_transaction(&mempool::MempoolKey { + txid: txid.to_rpc_hex(), + }) + .await + { + let bytes = mempool_tx.serialized_tx.as_ref().as_ref().to_vec(); + let mempool_branch_id = self.mempool_branch_id(snapshot); + + return Ok(Some((bytes, mempool_branch_id))); + } + + let Some((transaction, location)) = self + .source() + .get_transaction(*txid) + .await + .map_err(ChainIndexError::backing_validator)? + else { + return Ok(None); + }; + // as the reorg process cannot modify a transaction + // it's safe to serve nonfinalized state directly here + let height = match location { + GetTransactionLocation::BestChain(height) => height, + GetTransactionLocation::NonbestChain => { + // if the tranasction isn't on the best chain + // check our indexes. We need to find out the height from our index + // to determine the consensus branch ID + let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { + // If we don't have a block containing the transaction + // locally and the transaction's not on the validator's + // best chain, we can't determine its consensus branch ID + return Ok(None); + }; + + match self + .blocks_containing_transaction(non_finalized_snapshot, txid.0) + .await? + .next() + { + Some(block) => block.context.index.height.into(), + // As above Ok(None) + None => return Ok(None), + } + } + // We've already checked the mempool. Should be unreachable? + // todo: error here? + GetTransactionLocation::Mempool => return Ok(None), + }; + + Ok(Some(( + zebra_chain::transaction::SerializedTransaction::from(transaction) + .as_ref() + .to_vec(), + ConsensusBranchId::current(&self.network, height).map(u32::from), + ))) + } + + /// Given a transaction ID, returns all known blocks containing this transaction /// - /// **NOTE: This Method is currently not "passthrough aware", this should be added by - /// fetching block data from the backing validator when not locally available.** - async fn get_compact_block( + /// If the transaction is in the mempool, it will be in the `BestChainLocation` + /// if the mempool and snapshot are up-to-date, and the `NonBestChainLocation` set + /// if the snapshot is out-of-date compared to the mempool + async fn get_transaction_status( &self, snapshot: &Self::Snapshot, - height: types::Height, - pool_types: PoolTypeFilter, - ) -> Result, Self::Error> { + txid: &types::TransactionHash, + ) -> Result<(Option, HashSet), ChainIndexError> { match snapshot { ChainIndexSnapshot::NonFinalizedStateExists { non_finalized_snapshot, } => { - if height <= non_finalized_snapshot.best_tip.height { - Ok(Some(match snapshot.get_chainblock_by_height(&height) { - Some(block) => compact_block_with_pool_types( - block.to_compact_block(), - &pool_types.to_pool_types_vector(), - ), - None => { - match self - .finalized_state - .get_compact_block(height, pool_types.clone()) - .await - { - Ok(block) => block, - Err(_) => self - .get_compact_block_from_node(height, &pool_types) - .await? - .ok_or(ChainIndexError::database_hole(height, None))?, - } + let blocks_containing_transaction = self + .blocks_containing_transaction(non_finalized_snapshot, txid.0) + .await? + .collect::>(); + let Some(start_of_nonfinalized) = + non_finalized_snapshot.heights_to_hashes.keys().min() + else { + return Err(ChainIndexError::database_hole("no blocks", None)); + }; + let mut best_chain_block = blocks_containing_transaction + .iter() + .find(|block| { + non_finalized_snapshot + .heights_to_hashes + .get(&block.height()) + == Some(block.hash()) + || block.height() < *start_of_nonfinalized + // this block is either in the best chain ``heights_to_hashes`` or finalized. + }) + .map(|block| BestChainLocation::Block(*block.hash(), block.height())); + let mut non_best_chain_blocks: HashSet = + blocks_containing_transaction + .iter() + .filter(|block| { + non_finalized_snapshot + .heights_to_hashes + .get(&block.height()) + != Some(block.hash()) + && block.height() >= *start_of_nonfinalized + }) + .map(|block| NonBestChainLocation::Block(*block.hash(), block.height())) + .collect(); + let in_mempool = self + .mempool + .contains_txid(&mempool::MempoolKey { + txid: txid.to_rpc_hex(), + }) + .await; + if in_mempool { + let mempool_tip_hash = self.mempool.mempool_chain_tip(); + if mempool_tip_hash == non_finalized_snapshot.best_tip.hash { + if best_chain_block.is_some() { + return Err(ChainIndexError { + kind: ChainIndexErrorKind::InvalidSnapshot, + message: + "Best chain and up-to-date mempool both contain the same transaction" + .to_string(), + source: None, + }); + } else { + best_chain_block = Some(BestChainLocation::Mempool( + non_finalized_snapshot.best_tip.height + 1, + )); } - })) - } else { - Ok(None) + } else { + // the best chain and the mempool have divergent tip hashes + // get a new snapshot and use it to find the height of the mempool + if let ChainIndexSnapshot::NonFinalizedStateExists { + non_finalized_snapshot: new_snapshot, + } = self.snapshot_nonfinalized_state().await? + { + let target_height = + new_snapshot.blocks.iter().find_map(|(hash, block)| { + if *hash == mempool_tip_hash { + Some(block.height() + 1) + // found the block that is the tip that the mempool is hanging on to + } else { + None + } + }); + non_best_chain_blocks + .insert(NonBestChainLocation::Mempool(target_height)); + } + } } + Ok((best_chain_block, non_best_chain_blocks)) } - ChainIndexSnapshot::StillSyncingFinalizedState { - validator_finalized_height: _, - //TODO: Once we make chainwork an option field we should be able to - // support passthrougth for this - } => Ok(None), + validator_finalized_height, + } => { + if let Some((_transaction, GetTransactionLocation::BestChain(height))) = self + .source() + .get_transaction(*txid) + .await + .map_err(ChainIndexError::backing_validator)? + { + if height <= *validator_finalized_height { + if let Some(block) = self + .source() + .get_block(HashOrHeight::Height(height)) + .await + .map_err(ChainIndexError::backing_validator)? + { + return Ok(( + Some(BestChainLocation::Block(block.hash().into(), height.into())), + HashSet::new(), + )); + } + } + } + Ok((None, HashSet::new())) + } } } - /// Streams *compact* blocks for an inclusive height range. - /// - /// Returns `Ok(None)` if the request is descending and `start_height` exceeds the chain tip. - /// For ascending requests that exceed the tip, returns a stream that ends with an - /// `out_of_range` error after all available blocks have been sent. - /// - /// - The stream covers `[start_height, end_height]` (inclusive). - /// - If `start_height <= end_height` the stream is ascending; otherwise it is descending. - /// - /// ## Pool filtering + /// Returns all txids currently in the mempool. + async fn get_mempool_txids(&self) -> Result, Self::Error> { + self.mempool + .get_mempool() + .await + .into_iter() + .map(|(txid_key, _)| { + TransactionHash::from_hex(&txid_key.txid) + .map_err(ChainIndexError::backing_validator) + }) + .collect::>() + } + + /// Returns all transactions currently in the mempool, filtered by `exclude_list`. /// - /// - `pool_types` controls which per-transaction components are populated. - /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. - /// The original transaction index is preserved in `CompactTx.index`. - /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard - /// components are populated). + /// The `exclude_list` may contain shortened transaction ID hex prefixes (client-endian). + /// The transaction IDs in the Exclude list can be shortened to any number of bytes to make the request + /// more bandwidth-efficient; if two or more transactions in the mempool + /// match a shortened txid, they are all sent (none is excluded). Transactions + /// in the exclude list that don't exist in the mempool are ignored. + async fn get_mempool_transactions( + &self, + exclude_list: Vec, + ) -> Result>, Self::Error> { + // Use the mempool's own filtering (it already handles client-endian shortened prefixes). + let pairs: Vec<(mempool::MempoolKey, mempool::MempoolValue)> = + self.mempool.get_filtered_mempool(exclude_list).await; + + // Transform to the Vec> that the trait requires. + let bytes: Vec> = pairs + .into_iter() + .map(|(_, v)| v.serialized_tx.as_ref().as_ref().to_vec()) + .collect(); + + Ok(bytes) + } + + /// Returns a stream of mempool transactions, ending the stream when the chain tip block hash + /// changes (a new block is mined or a reorg occurs). /// - /// **NOTE: This Method is currently not "passthrough aware", this should be added by - /// fetching block data from the backing validator when not locally available.** - #[allow(clippy::type_complexity)] - async fn get_compact_block_stream( + /// If a snapshot is given and the chain tip has changed from the given spanshot, returns None. + fn get_mempool_stream( &self, - nonfinalized_snapshot: &Self::Snapshot, - start_height: types::Height, - end_height: types::Height, - pool_types: PoolTypeFilter, - ) -> Result, Self::Error> { - let chain_tip_height = self.best_chaintip(nonfinalized_snapshot).await?.height; + snapshot: Option<&Self::Snapshot>, + ) -> Option, Self::Error>>> { + let non_finalized_snapshot = match snapshot { + Some(s) => match s { + ChainIndexSnapshot::NonFinalizedStateExists { + non_finalized_snapshot, + } => Some(non_finalized_snapshot), + // If we're still syncing the finalized state, the chain tip + // is newer than the snapshot's tip. Return None. + ChainIndexSnapshot::StillSyncingFinalizedState { .. } => return None, + }, + None => None, + }; + let expected_chain_tip = non_finalized_snapshot.map(|snapshot| snapshot.best_tip.hash); + let mut subscriber = self.mempool.clone(); - // The finalised state serves heights up to `finalized_height_floor(tip)` - // (= `tip - OPERATIONAL_NFS_DEPTH`, saturating at genesis); the non-finalised cache serves - // everything above it, so the lowest non-finalised height is one past the finalised floor. - let lowest_nonfinalized_height = - types::Height(finalized_height_floor(chain_tip_height.0).0 + 1); + match subscriber + .get_mempool_stream(expected_chain_tip) + .now_or_never() + { + Some(Ok((in_rx, _handle))) => { + let (out_tx, out_rx) = + tokio::sync::mpsc::channel::, ChainIndexError>>(32); - let is_ascending = start_height <= end_height; + tokio::spawn(async move { + let mut in_stream = tokio_stream::wrappers::ReceiverStream::new(in_rx); + while let Some(item) = in_stream.next().await { + match item { + Ok((_key, value)) => { + let _ = out_tx + .send(Ok(value.serialized_tx.as_ref().as_ref().to_vec())) + .await; + } + Err(e) => { + let _ = out_tx + .send(Err(ChainIndexError::child_process_status_error( + "mempool", e, + ))) + .await; + break; + } + } + } + }); - // Descending: the first block we'd try to return is already above the tip → error immediately. - if !is_ascending && start_height > chain_tip_height { - return Ok(None); + Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) + } + Some(Err(crate::error::MempoolError::IncorrectChainTip { .. })) => None, + Some(Err(e)) => { + let (out_tx, out_rx) = + tokio::sync::mpsc::channel::, ChainIndexError>>(1); + let _ = out_tx.try_send(Err(e.into())); + Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) + } + None => { + // Should not happen because the inner tip check is synchronous, but fail safe. + let (out_tx, out_rx) = + tokio::sync::mpsc::channel::, ChainIndexError>>(1); + let _ = out_tx.try_send(Err(ChainIndexError::child_process_status_error( + "mempool", + crate::error::StatusError { + server_status: crate::StatusType::RecoverableError, + }, + ))); + Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) + } } + } - let pool_types_vector = pool_types.to_pool_types_vector(); + // ********** Chain methods ********** - // For ascending requests that extend past the tip: cap the streaming range at the tip, - // then append a trailing out_of_range error after all valid blocks have been sent. - let needs_out_of_range = is_ascending && end_height > chain_tip_height; - let capped_end_height = if needs_out_of_range { - chain_tip_height - } else { - end_height - }; - - // Pre-create any finalized-state stream(s) we will need so that errors are returned - // from this method (not deferred into the spawned task). - let finalized_stream: Option = if is_ascending { - if start_height < lowest_nonfinalized_height { - let finalized_end_height = types::Height(std::cmp::min( - capped_end_height.0, - lowest_nonfinalized_height.0.saturating_sub(1), - )); - - if start_height <= finalized_end_height { - Some( - self.finalized_state - .get_compact_block_stream( - start_height, - finalized_end_height, - pool_types.clone(), - ) - .await - .map_err(ChainIndexError::from)?, - ) - } else { - None - } - } else { - None - } - // Serve in reverse order. - } else if end_height < lowest_nonfinalized_height { - let finalized_start_height = if start_height < lowest_nonfinalized_height { - start_height - } else { - types::Height(lowest_nonfinalized_height.0.saturating_sub(1)) - }; - - Some( - self.finalized_state - .get_compact_block_stream( - finalized_start_height, - end_height, - pool_types.clone(), - ) - .await - .map_err(ChainIndexError::from)?, - ) - } else { - None - }; - - let nonfinalized_snapshot = nonfinalized_snapshot.clone(); - let source = self.source.clone(); - let network = self.network.clone(); - let pool_types_for_node = pool_types.clone(); - // TODO: Investigate whether channel size should be changed, added to config, or set dynamically based on resources. - let (channel_sender, channel_receiver) = tokio::sync::mpsc::channel(128); + /// For a given block, + /// find its newest main-chain ancestor, + /// or the block itself if it is on the main-chain. + /// Returns Ok(None) if no fork point found. This is not an error, + /// as zaino does not guarentee knowledge of all sidechain data. + async fn find_fork_point( + &self, + snapshot: &Self::Snapshot, + hash: &types::BlockHash, + ) -> Result, Self::Error> { + // ChainIndex step 1: Skip + // mempool blocks have no canon height, guaranteed to return None + // todo: possible efficiency boost by checking mempool for a negative? - tokio::spawn(async move { - if is_ascending { - // 1) Finalized segment (if any), ascending. - if let Some(mut finalized_stream) = finalized_stream { - while let Some(stream_item) = finalized_stream.next().await { - if channel_sender.send(stream_item).await.is_err() { - return; + match snapshot { + ChainIndexSnapshot::NonFinalizedStateExists { + non_finalized_snapshot, + } => { + match non_finalized_snapshot.get_chainblock_by_hash(hash) { + Some(block) => { + // At this point, we know that + // The block is non-FINALIZED in the INDEXER + // ChainIndex step 3: + if non_finalized_snapshot + .heights_to_hashes + .get(&block.height()) + == Some(block.hash()) + { + // The block is in the best chain. + Ok(Some((*block.hash(), block.height()))) + } else { + // Otherwise, it's non-best chain! Grab its parent, and recurse + Box::pin(self.find_fork_point(snapshot, &block.context.parent_hash)) + .await + // gotta pin recursive async functions to prevent infinite-sized + // Future-implementing types } } - } - - // 2) Nonfinalized segment, ascending. - let nonfinalized_start_height = - types::Height(std::cmp::max(start_height.0, lowest_nonfinalized_height.0)); - - for height_value in nonfinalized_start_height.0..=capped_end_height.0 { - let Some(indexed_block) = nonfinalized_snapshot - .get_chainblock_by_height(&types::Height(height_value)) - else { - match compact_block_from_source( - &source, - network.clone(), - types::Height(height_value), - &pool_types_for_node, - ) - .await - { - Ok(Some(compact_block)) => { - if channel_sender.send(Ok(compact_block)).await.is_err() { - return; - } - continue; - } - Ok(None) => { - let _ = channel_sender - .send(Err(tonic::Status::internal(format!( - "Internal error, missing nonfinalized block at height [{height_value}].", - )))) - .await; - return; - } - Err(error) => { - let _ = channel_sender - .send(Err(tonic::Status::internal(error.to_string()))) - .await; - return; + None => { + // At this point, we know that + // the block is NOT non-FINALIZED in the INDEXER. + // as the non finalzed state is known to be populated, + // we now check the finalized state + match self.finalized_state.get_block_height(*hash).await { + Ok(Some(height)) => { + // the block is FINALIZED in the INDEXER + Ok(Some((*hash, height))) } + Err(e) => Err(ChainIndexError::database_hole(hash, Some(Box::new(e)))), + Ok(None) => Ok(None), } - }; - let compact_block = compact_block_with_pool_types( - indexed_block.to_compact_block(), - &pool_types_vector, - ); - if channel_sender.send(Ok(compact_block)).await.is_err() { - return; } } - // If the original end_height was above the tip, signal out_of_range after all valid blocks. - if needs_out_of_range { - let _ = channel_sender - .send(Err(tonic::Status::out_of_range(format!( - "Error: Height out of range [{}]. Height requested is greater than the best chain tip [{}].", - end_height.0, chain_tip_height.0, - )))) - .await; - } - } else { - // 1) Nonfinalized segment, descending. - if start_height >= lowest_nonfinalized_height { - let nonfinalized_end_height = - types::Height(std::cmp::max(end_height.0, lowest_nonfinalized_height.0)); - - for height_value in (nonfinalized_end_height.0..=start_height.0).rev() { - let Some(indexed_block) = nonfinalized_snapshot - .get_chainblock_by_height(&types::Height(height_value)) - else { - match compact_block_from_source( - &source, - network.clone(), - types::Height(height_value), - &pool_types_for_node, - ) - .await - { - Ok(Some(compact_block)) => { - if channel_sender.send(Ok(compact_block)).await.is_err() { - return; - } - continue; - } - Ok(None) => { - let _ = channel_sender - .send(Err(tonic::Status::internal(format!( - "Internal error, missing nonfinalized block at height [{height_value}].", - )))) - .await; - return; - } - Err(error) => { - let _ = channel_sender - .send(Err(tonic::Status::internal(error.to_string()))) - .await; - return; + } + ChainIndexSnapshot::StillSyncingFinalizedState { + validator_finalized_height, + } => { + // We're not fully synced, so we pass through. + // Now, we ask the VALIDATOR. + // ChainIndex step 5 + match self + .source() + .get_block(HashOrHeight::Hash(zebra_chain::block::Hash::from(*hash))) + .await + { + Ok(Some(block)) => { + // At this point, we know that + // the block is in the VALIDATOR. + match block.coinbase_height() { + None => { + // the block is in the VALIDATOR. but doesnt have a height. That would imply a bug. + Err(ChainIndexError::validator_data_error_block_coinbase_height_missing()) + } + Some(height) => { + // The VALIDATOR returned a block with a height. + // However, there is as of yet no guaranteed the Block is FINALIZED + if height <= *validator_finalized_height { + Ok(Some(( + types::BlockHash::from(block.hash()), + types::Height::from(height), + ))) + } else { + // non-finalized block + // no passthrough + Ok(None) } } - }; - let compact_block = compact_block_with_pool_types( - indexed_block.to_compact_block(), - &pool_types_vector, - ); - if channel_sender.send(Ok(compact_block)).await.is_err() { - return; } } - } - // 2) Finalized segment (if any), descending. - if let Some(mut finalized_stream) = finalized_stream { - while let Some(stream_item) = finalized_stream.next().await { - if channel_sender.send(stream_item).await.is_err() { - return; - } + Ok(None) => { + // At this point, we know that + // the block is NOT FINALIZED in the VALIDATOR. + // Return Ok(None) = no block found. + Ok(None) } + Err(e) => Err(ChainIndexError::backing_validator(e)), } } - }); + } + } - Ok(Some(CompactBlockStream::new(channel_receiver))) - } + /// Returns the block commitment tree data by hash. + async fn get_treestate( + &self, + hash: &types::BlockHash, + ) -> Result< + ( + Option, + Option, + Option, + ), + Self::Error, + > { + let snapshot = self.snapshot_nonfinalized_state().await?; + if !self.block_hash_known_for_treestate(&snapshot, hash).await? { + return Err(ChainIndexError::internal(format!( + "block hash {hash} not found in local chain index" + ))); + } - // ********** Transaction methods ********** + match self.source().get_treestate(*hash).await { + Ok(resp) => Ok(resp), + Err(e) => Err(ChainIndexError { + kind: ChainIndexErrorKind::InternalServerError, + message: "failed to fetch treestate from validator".to_string(), + source: Some(Box::new(e)), + }), + } + } - /// given a transaction id, returns the transaction - /// and the consensus branch ID for the block the transaction - /// is in - async fn get_raw_transaction( + /// Gets the subtree roots of a given pool and the end heights of each root, + /// starting at the provided index, up to an optional maximum number of roots. + async fn get_subtree_roots( &self, - snapshot: &Self::Snapshot, - txid: &types::TransactionHash, - ) -> Result, Option)>, Self::Error> { - // ChainIndex step 1 - if let Some(mempool_tx) = self - .mempool - .get_transaction(&mempool::MempoolKey { - txid: txid.to_rpc_hex(), - }) + pool: ShieldedPool, + start_index: u16, + max_entries: Option, + ) -> Result, Self::Error> { + self.source() + .get_subtree_roots(pool, start_index, max_entries) .await - { - let bytes = mempool_tx.serialized_tx.as_ref().as_ref().to_vec(); - let mempool_branch_id = self.mempool_branch_id(snapshot); + .map_err(ChainIndexError::backing_validator) + } - return Ok(Some((bytes, mempool_branch_id))); - } + // ********** Transparent address history methods ********** - let Some((transaction, location)) = self - .source() - .get_transaction(*txid) + /// Returns the total transparent balance for the given addresses. + async fn get_address_balance( + &self, + address_strings: GetAddressBalanceRequest, + ) -> Result { + self.source() + .get_address_balance(address_strings) .await - .map_err(ChainIndexError::backing_validator)? - else { - return Ok(None); - }; - // as the reorg process cannot modify a transaction - // it's safe to serve nonfinalized state directly here - let height = match location { - GetTransactionLocation::BestChain(height) => height, - GetTransactionLocation::NonbestChain => { - // if the tranasction isn't on the best chain - // check our indexes. We need to find out the height from our index - // to determine the consensus branch ID - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - // If we don't have a block containing the transaction - // locally and the transaction's not on the validator's - // best chain, we can't determine its consensus branch ID - return Ok(None); - }; + .map_err(ChainIndexError::backing_validator) + } - match self - .blocks_containing_transaction(non_finalized_snapshot, txid.0) - .await? - .next() - { - Some(block) => block.context.index.height.into(), - // As above Ok(None) - None => return Ok(None), - } - } - // We've already checked the mempool. Should be unreachable? - // todo: error here? - GetTransactionLocation::Mempool => return Ok(None), - }; + /// Returns the transaction ids made by the given transparent addresses. + async fn get_address_txids( + &self, + request: GetAddressTxIdsRequest, + ) -> Result, Self::Error> { + self.source() + .get_address_txids(request) + .await + .map_err(ChainIndexError::backing_validator) + } - Ok(Some(( - zebra_chain::transaction::SerializedTransaction::from(transaction) - .as_ref() - .to_vec(), - ConsensusBranchId::current(&self.network, height).map(u32::from), - ))) + /// Returns all unspent transparent outputs for the given addresses. + async fn get_address_utxos( + &self, + address_strings: GetAddressBalanceRequest, + ) -> Result, Self::Error> { + self.source() + .get_address_utxos(address_strings) + .await + .map_err(ChainIndexError::backing_validator) } - /// Given a transaction ID, returns all known blocks containing this transaction - /// - /// If the transaction is in the mempool, it will be in the `BestChainLocation` - /// if the mempool and snapshot are up-to-date, and the `NonBestChainLocation` set - /// if the snapshot is out-of-date compared to the mempool - async fn get_transaction_status( + async fn get_outpoint_spenders( &self, snapshot: &Self::Snapshot, - txid: &types::TransactionHash, - ) -> Result<(Option, HashSet), ChainIndexError> { - match snapshot { + outpoints: Vec, + scope: types::ChainScope, + ) -> Result>, Self::Error> { + use std::collections::HashMap; + + let mut result: Vec> = vec![None; outpoints.len()]; + + // 1) Non-finalised best chain (FullChain scope only). Scan only the blocks reachable + // via `heights_to_hashes` (the `blocks` map also holds reorged-away blocks, which + // must not count). One pass builds an outpoint -> spending-txid map regardless of + // how many outpoints we look up. Under `Finalised` scope this is skipped so results + // are reorg-stable. + if let ( + types::ChainScope::FullChain, ChainIndexSnapshot::NonFinalizedStateExists { non_finalized_snapshot, - } => { - let blocks_containing_transaction = self - .blocks_containing_transaction(non_finalized_snapshot, txid.0) - .await? - .collect::>(); - let Some(start_of_nonfinalized) = - non_finalized_snapshot.heights_to_hashes.keys().min() - else { - return Err(ChainIndexError::database_hole("no blocks", None)); + }, + ) = (scope, snapshot) + { + let mut nfs_spenders: HashMap = HashMap::new(); + for hash in non_finalized_snapshot.heights_to_hashes.values() { + let Some(block) = non_finalized_snapshot.blocks.get(hash) else { + continue; }; - let mut best_chain_block = blocks_containing_transaction - .iter() - .find(|block| { - non_finalized_snapshot - .heights_to_hashes - .get(&block.height()) - == Some(block.hash()) - || block.height() < *start_of_nonfinalized - // this block is either in the best chain ``heights_to_hashes`` or finalized. - }) - .map(|block| BestChainLocation::Block(*block.hash(), block.height())); - let mut non_best_chain_blocks: HashSet = - blocks_containing_transaction - .iter() - .filter(|block| { - non_finalized_snapshot - .heights_to_hashes - .get(&block.height()) - != Some(block.hash()) - && block.height() >= *start_of_nonfinalized - }) - .map(|block| NonBestChainLocation::Block(*block.hash(), block.height())) - .collect(); - let in_mempool = self - .mempool - .contains_txid(&mempool::MempoolKey { - txid: txid.to_rpc_hex(), - }) - .await; - if in_mempool { - let mempool_tip_hash = self.mempool.mempool_chain_tip(); - if mempool_tip_hash == non_finalized_snapshot.best_tip.hash { - if best_chain_block.is_some() { - return Err(ChainIndexError { - kind: ChainIndexErrorKind::InvalidSnapshot, - message: - "Best chain and up-to-date mempool both contain the same transaction" - .to_string(), - source: None, - }); - } else { - best_chain_block = Some(BestChainLocation::Mempool( - non_finalized_snapshot.best_tip.height + 1, - )); - } - } else { - // the best chain and the mempool have divergent tip hashes - // get a new snapshot and use it to find the height of the mempool - if let ChainIndexSnapshot::NonFinalizedStateExists { - non_finalized_snapshot: new_snapshot, - } = self.snapshot_nonfinalized_state().await? - { - let target_height = - new_snapshot.blocks.iter().find_map(|(hash, block)| { - if *hash == mempool_tip_hash { - Some(block.height() + 1) - // found the block that is the tip that the mempool is hanging on to - } else { - None - } - }); - non_best_chain_blocks - .insert(NonBestChainLocation::Mempool(target_height)); - } + for tx in block.transactions() { + let txid = *tx.txid(); + // `spent_outpoints` already skips coinbase null prevouts and builds each + // `Outpoint`, keeping the construction in one place (see #1332). + for outpoint in tx.transparent().spent_outpoints() { + nfs_spenders.insert(outpoint, txid); } } - Ok((best_chain_block, non_best_chain_blocks)) } - ChainIndexSnapshot::StillSyncingFinalizedState { - validator_finalized_height, - } => { - if let Some((_transaction, GetTransactionLocation::BestChain(height))) = self - .source() - .get_transaction(*txid) - .await - .map_err(ChainIndexError::backing_validator)? - { - if height <= *validator_finalized_height { - if let Some(block) = self - .source() - .get_block(HashOrHeight::Height(height)) - .await - .map_err(ChainIndexError::backing_validator)? - { - return Ok(( - Some(BestChainLocation::Block(block.hash().into(), height.into())), - HashSet::new(), - )); - } - } + for (i, outpoint) in outpoints.iter().enumerate() { + if let Some(txid) = nfs_spenders.get(outpoint) { + result[i] = Some(*txid); } - Ok((None, HashSet::new())) } } - } - - /// Returns all txids currently in the mempool. - async fn get_mempool_txids(&self) -> Result, Self::Error> { - self.mempool - .get_mempool() - .await - .into_iter() - .map(|(txid_key, _)| { - TransactionHash::from_hex(&txid_key.txid) - .map_err(ChainIndexError::backing_validator) - }) - .collect::>() - } - - /// Returns all transactions currently in the mempool, filtered by `exclude_list`. - /// - /// The `exclude_list` may contain shortened transaction ID hex prefixes (client-endian). - /// The transaction IDs in the Exclude list can be shortened to any number of bytes to make the request - /// more bandwidth-efficient; if two or more transactions in the mempool - /// match a shortened txid, they are all sent (none is excluded). Transactions - /// in the exclude list that don't exist in the mempool are ignored. - async fn get_mempool_transactions( - &self, - exclude_list: Vec, - ) -> Result>, Self::Error> { - // Use the mempool's own filtering (it already handles client-endian shortened prefixes). - let pairs: Vec<(mempool::MempoolKey, mempool::MempoolValue)> = - self.mempool.get_filtered_mempool(exclude_list).await; - // Transform to the Vec> that the trait requires. - let bytes: Vec> = pairs - .into_iter() - .map(|(_, v)| v.serialized_tx.as_ref().as_ref().to_vec()) + // 2) Finalised lookup for the still-unresolved outpoints, batched into one DB call. + let unresolved_indices: Vec = result + .iter() + .enumerate() + .filter_map(|(i, r)| r.is_none().then_some(i)) .collect(); + if unresolved_indices.is_empty() { + return Ok(result); + } + let unresolved_outpoints: Vec = + unresolved_indices.iter().map(|&i| outpoints[i]).collect(); + let locations = self + .finalized_state + .get_outpoint_spenders(unresolved_outpoints) + .await?; - Ok(bytes) - } + // 3) Resolve each finalised `TxLocation` to a txid. Dedup identical locations so a + // block spending several queried outpoints is only fetched once. `get_txid` is a + // single keyed lookup, far cheaper than reconstructing the whole block. + let mut slots_by_location: HashMap> = HashMap::new(); + for (slot, location) in unresolved_indices.into_iter().zip(locations) { + if let Some(location) = location { + slots_by_location.entry(location).or_default().push(slot); + } + } + for (location, slots) in slots_by_location { + let txid = self.finalized_state.get_txid(location).await?; + for slot in slots { + result[slot] = Some(txid); + } + } - /// Returns a stream of mempool transactions, ending the stream when the chain tip block hash - /// changes (a new block is mined or a reorg occurs). - /// - /// If a snapshot is given and the chain tip has changed from the given spanshot, returns None. - fn get_mempool_stream( - &self, - snapshot: Option<&Self::Snapshot>, - ) -> Option, Self::Error>>> { - let non_finalized_snapshot = match snapshot { - Some(s) => match s { - ChainIndexSnapshot::NonFinalizedStateExists { - non_finalized_snapshot, - } => Some(non_finalized_snapshot), - // If we're still syncing the finalized state, the chain tip - // is newer than the snapshot's tip. Return None. - ChainIndexSnapshot::StillSyncingFinalizedState { .. } => return None, - }, - None => None, - }; - let expected_chain_tip = non_finalized_snapshot.map(|snapshot| snapshot.best_tip.hash); - let mut subscriber = self.mempool.clone(); + Ok(result) + } - match subscriber - .get_mempool_stream(expected_chain_tip) - .now_or_never() - { - Some(Ok((in_rx, _handle))) => { - let (out_tx, out_rx) = - tokio::sync::mpsc::channel::, ChainIndexError>>(32); + // ********** Metadata methods ********** - tokio::spawn(async move { - let mut in_stream = tokio_stream::wrappers::ReceiverStream::new(in_rx); - while let Some(item) = in_stream.next().await { - match item { - Ok((_key, value)) => { - let _ = out_tx - .send(Ok(value.serialized_tx.as_ref().as_ref().to_vec())) - .await; - } - Err(e) => { - let _ = out_tx - .send(Err(ChainIndexError::child_process_status_error( - "mempool", e, - ))) - .await; - break; - } - } - } - }); + async fn best_chaintip(&self, snapshot: &Self::Snapshot) -> Result { + Ok(match snapshot { + ChainIndexSnapshot::NonFinalizedStateExists { + non_finalized_snapshot, + } => non_finalized_snapshot.best_tip, - Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) - } - Some(Err(crate::error::MempoolError::IncorrectChainTip { .. })) => None, - Some(Err(e)) => { - let (out_tx, out_rx) = - tokio::sync::mpsc::channel::, ChainIndexError>>(1); - let _ = out_tx.try_send(Err(e.into())); - Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) - } - None => { - // Should not happen because the inner tip check is synchronous, but fail safe. - let (out_tx, out_rx) = - tokio::sync::mpsc::channel::, ChainIndexError>>(1); - let _ = out_tx.try_send(Err(ChainIndexError::child_process_status_error( - "mempool", - crate::error::StatusError { - server_status: crate::StatusType::RecoverableError, - }, - ))); - Some(tokio_stream::wrappers::ReceiverStream::new(out_rx)) + ChainIndexSnapshot::StillSyncingFinalizedState { + validator_finalized_height, + } => { + BlockIndex { + height: *validator_finalized_height, + hash: self + .source() + // TODO: do something more efficient than getting the whole block + .get_block(HashOrHeight::Height((*validator_finalized_height).into())) + .await + .map_err(|e| { + ChainIndexError::database_hole( + validator_finalized_height, + Some(Box::new(e)), + ) + })? + .ok_or(ChainIndexError::database_hole( + validator_finalized_height, + None, + ))? + .hash() + .into(), + } } - } + }) } +} - // ********** Chain methods ********** - - /// For a given block, - /// find its newest main-chain ancestor, - /// or the block itself if it is on the main-chain. - /// Returns Ok(None) if no fork point found. This is not an error, - /// as zaino does not guarentee knowledge of all sidechain data. - async fn find_fork_point( +impl ChainIndexRpcExt for NodeBackedChainIndexSubscriber { + /// Returns the *compact* block for the given height. + /// + /// Returns `None` if the specified `height` is greater than the snapshot's tip. + /// + /// ## Pool filtering + /// + /// - `pool_types` controls which per-transaction components are populated. + /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. + /// The original transaction index is preserved in `CompactTx.index`. + /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard + /// components are populated). + /// + /// Returns None if the specified height + /// is greater than the snapshot's tip + /// + /// **NOTE: This Method is currently not "passthrough aware", this should be added by + /// fetching block data from the backing validator when not locally available.** + async fn get_compact_block( &self, snapshot: &Self::Snapshot, - hash: &types::BlockHash, - ) -> Result, Self::Error> { - // ChainIndex step 1: Skip - // mempool blocks have no canon height, guaranteed to return None - // todo: possible efficiency boost by checking mempool for a negative? - + height: types::Height, + pool_types: PoolTypeFilter, + ) -> Result, Self::Error> { match snapshot { ChainIndexSnapshot::NonFinalizedStateExists { non_finalized_snapshot, } => { - match non_finalized_snapshot.get_chainblock_by_hash(hash) { - Some(block) => { - // At this point, we know that - // The block is non-FINALIZED in the INDEXER - // ChainIndex step 3: - if non_finalized_snapshot - .heights_to_hashes - .get(&block.height()) - == Some(block.hash()) - { - // The block is in the best chain. - Ok(Some((*block.hash(), block.height()))) - } else { - // Otherwise, it's non-best chain! Grab its parent, and recurse - Box::pin(self.find_fork_point(snapshot, &block.context.parent_hash)) + if height <= non_finalized_snapshot.best_tip.height { + Ok(Some(match snapshot.get_chainblock_by_height(&height) { + Some(block) => compact_block_with_pool_types( + block.to_compact_block(), + &pool_types.to_pool_types_vector(), + ), + None => { + match self + .finalized_state + .get_compact_block(height, pool_types.clone()) .await - // gotta pin recursive async functions to prevent infinite-sized - // Future-implementing types - } - } - None => { - // At this point, we know that - // the block is NOT non-FINALIZED in the INDEXER. - // as the non finalzed state is known to be populated, - // we now check the finalized state - match self.finalized_state.get_block_height(*hash).await { - Ok(Some(height)) => { - // the block is FINALIZED in the INDEXER - Ok(Some((*hash, height))) + { + Ok(block) => block, + Err(_) => self + .get_compact_block_from_node(height, &pool_types) + .await? + .ok_or(ChainIndexError::database_hole(height, None))?, } - Err(e) => Err(ChainIndexError::database_hole(hash, Some(Box::new(e)))), - Ok(None) => Ok(None), } - } + })) + } else { + Ok(None) } } + ChainIndexSnapshot::StillSyncingFinalizedState { - validator_finalized_height, - } => { - // We're not fully synced, so we pass through. - // Now, we ask the VALIDATOR. - // ChainIndex step 5 - match self - .source() - .get_block(HashOrHeight::Hash(zebra_chain::block::Hash::from(*hash))) + validator_finalized_height: _, + //TODO: Once we make chainwork an option field we should be able to + // support passthrougth for this + } => Ok(None), + } + } + + /// Streams *compact* blocks for an inclusive height range. + /// + /// Returns `Ok(None)` if the request is descending and `start_height` exceeds the chain tip. + /// For ascending requests that exceed the tip, returns a stream that ends with an + /// `out_of_range` error after all available blocks have been sent. + /// + /// - The stream covers `[start_height, end_height]` (inclusive). + /// - If `start_height <= end_height` the stream is ascending; otherwise it is descending. + /// + /// ## Pool filtering + /// + /// - `pool_types` controls which per-transaction components are populated. + /// - Transactions that contain no elements in any requested pool are omitted from `vtx`. + /// The original transaction index is preserved in `CompactTx.index`. + /// - `PoolTypeFilter::default()` preserves the legacy behaviour (only Sapling and Orchard + /// components are populated). + /// + /// **NOTE: This Method is currently not "passthrough aware", this should be added by + /// fetching block data from the backing validator when not locally available.** + #[allow(clippy::type_complexity)] + async fn get_compact_block_stream( + &self, + nonfinalized_snapshot: &Self::Snapshot, + start_height: types::Height, + end_height: types::Height, + pool_types: PoolTypeFilter, + ) -> Result, Self::Error> { + let chain_tip_height = self.best_chaintip(nonfinalized_snapshot).await?.height; + + // The finalised state serves heights up to `finalized_height_floor(tip)` + // (= `tip - OPERATIONAL_NFS_DEPTH`, saturating at genesis); the non-finalised cache serves + // everything above it, so the lowest non-finalised height is one past the finalised floor. + let lowest_nonfinalized_height = + types::Height(finalized_height_floor(chain_tip_height.0).0 + 1); + + let is_ascending = start_height <= end_height; + + // Descending: the first block we'd try to return is already above the tip → error immediately. + if !is_ascending && start_height > chain_tip_height { + return Ok(None); + } + + let pool_types_vector = pool_types.to_pool_types_vector(); + + // For ascending requests that extend past the tip: cap the streaming range at the tip, + // then append a trailing out_of_range error after all valid blocks have been sent. + let needs_out_of_range = is_ascending && end_height > chain_tip_height; + let capped_end_height = if needs_out_of_range { + chain_tip_height + } else { + end_height + }; + + // Pre-create any finalized-state stream(s) we will need so that errors are returned + // from this method (not deferred into the spawned task). + let finalized_stream: Option = if is_ascending { + if start_height < lowest_nonfinalized_height { + let finalized_end_height = types::Height(std::cmp::min( + capped_end_height.0, + lowest_nonfinalized_height.0.saturating_sub(1), + )); + + if start_height <= finalized_end_height { + Some( + self.finalized_state + .get_compact_block_stream( + start_height, + finalized_end_height, + pool_types.clone(), + ) + .await + .map_err(ChainIndexError::from)?, + ) + } else { + None + } + } else { + None + } + // Serve in reverse order. + } else if end_height < lowest_nonfinalized_height { + let finalized_start_height = if start_height < lowest_nonfinalized_height { + start_height + } else { + types::Height(lowest_nonfinalized_height.0.saturating_sub(1)) + }; + + Some( + self.finalized_state + .get_compact_block_stream( + finalized_start_height, + end_height, + pool_types.clone(), + ) .await - { - Ok(Some(block)) => { - // At this point, we know that - // the block is in the VALIDATOR. - match block.coinbase_height() { - None => { - // the block is in the VALIDATOR. but doesnt have a height. That would imply a bug. - Err(ChainIndexError::validator_data_error_block_coinbase_height_missing()) + .map_err(ChainIndexError::from)?, + ) + } else { + None + }; + + let nonfinalized_snapshot = nonfinalized_snapshot.clone(); + let source = self.source.clone(); + let network = self.network.clone(); + let pool_types_for_node = pool_types.clone(); + // TODO: Investigate whether channel size should be changed, added to config, or set dynamically based on resources. + let (channel_sender, channel_receiver) = tokio::sync::mpsc::channel(128); + + tokio::spawn(async move { + if is_ascending { + // 1) Finalized segment (if any), ascending. + if let Some(mut finalized_stream) = finalized_stream { + while let Some(stream_item) = finalized_stream.next().await { + if channel_sender.send(stream_item).await.is_err() { + return; + } + } + } + + // 2) Nonfinalized segment, ascending. + let nonfinalized_start_height = + types::Height(std::cmp::max(start_height.0, lowest_nonfinalized_height.0)); + + for height_value in nonfinalized_start_height.0..=capped_end_height.0 { + let Some(indexed_block) = nonfinalized_snapshot + .get_chainblock_by_height(&types::Height(height_value)) + else { + match compact_block_from_source( + &source, + network.clone(), + types::Height(height_value), + &pool_types_for_node, + ) + .await + { + Ok(Some(compact_block)) => { + if channel_sender.send(Ok(compact_block)).await.is_err() { + return; + } + continue; } - Some(height) => { - // The VALIDATOR returned a block with a height. - // However, there is as of yet no guaranteed the Block is FINALIZED - if height <= *validator_finalized_height { - Ok(Some(( - types::BlockHash::from(block.hash()), - types::Height::from(height), - ))) - } else { - // non-finalized block - // no passthrough - Ok(None) + Ok(None) => { + let _ = channel_sender + .send(Err(tonic::Status::internal(format!( + "Internal error, missing nonfinalized block at height [{height_value}].", + )))) + .await; + return; + } + Err(error) => { + let _ = channel_sender + .send(Err(tonic::Status::internal(error.to_string()))) + .await; + return; + } + } + }; + let compact_block = compact_block_with_pool_types( + indexed_block.to_compact_block(), + &pool_types_vector, + ); + if channel_sender.send(Ok(compact_block)).await.is_err() { + return; + } + } + // If the original end_height was above the tip, signal out_of_range after all valid blocks. + if needs_out_of_range { + let _ = channel_sender + .send(Err(tonic::Status::out_of_range(format!( + "Error: Height out of range [{}]. Height requested is greater than the best chain tip [{}].", + end_height.0, chain_tip_height.0, + )))) + .await; + } + } else { + // 1) Nonfinalized segment, descending. + if start_height >= lowest_nonfinalized_height { + let nonfinalized_end_height = + types::Height(std::cmp::max(end_height.0, lowest_nonfinalized_height.0)); + + for height_value in (nonfinalized_end_height.0..=start_height.0).rev() { + let Some(indexed_block) = nonfinalized_snapshot + .get_chainblock_by_height(&types::Height(height_value)) + else { + match compact_block_from_source( + &source, + network.clone(), + types::Height(height_value), + &pool_types_for_node, + ) + .await + { + Ok(Some(compact_block)) => { + if channel_sender.send(Ok(compact_block)).await.is_err() { + return; + } + continue; + } + Ok(None) => { + let _ = channel_sender + .send(Err(tonic::Status::internal(format!( + "Internal error, missing nonfinalized block at height [{height_value}].", + )))) + .await; + return; + } + Err(error) => { + let _ = channel_sender + .send(Err(tonic::Status::internal(error.to_string()))) + .await; + return; } } + }; + let compact_block = compact_block_with_pool_types( + indexed_block.to_compact_block(), + &pool_types_vector, + ); + if channel_sender.send(Ok(compact_block)).await.is_err() { + return; } } + } - Ok(None) => { - // At this point, we know that - // the block is NOT FINALIZED in the VALIDATOR. - // Return Ok(None) = no block found. - Ok(None) + // 2) Finalized segment (if any), descending. + if let Some(mut finalized_stream) = finalized_stream { + while let Some(stream_item) = finalized_stream.next().await { + if channel_sender.send(stream_item).await.is_err() { + return; + } } - Err(e) => Err(ChainIndexError::backing_validator(e)), } } - } + }); + + Ok(Some(CompactBlockStream::new(channel_receiver))) } - /// Returns the block commitment tree data by hash. - async fn get_treestate( + async fn z_get_block( &self, - hash: &types::BlockHash, - ) -> Result< - ( - Option, - Option, - Option, - ), - Self::Error, - > { + hash_or_height: String, + verbosity: Option, + ) -> Result { + // Resolve tip-relative negative heights against the best chaintip, + // matching zebra's own `getblock` semantics (`-1` is the tip). A + // rejected identifier carries zcashd's legacy InvalidParameter code + // as a typed `RpcError` source, which the serve layer recovers by + // downcast-walking the error chain. let snapshot = self.snapshot_nonfinalized_state().await?; - if !self.block_hash_known_for_treestate(&snapshot, hash).await? { - return Err(ChainIndexError::internal(format!( - "block hash {hash} not found in local chain index" - ))); - } - - match self.source().get_treestate(*hash).await { - Ok(resp) => Ok(resp), - Err(e) => Err(ChainIndexError { - kind: ChainIndexErrorKind::InternalServerError, - message: "failed to fetch treestate from validator".to_string(), - source: Some(Box::new(e)), - }), - } + let tip = self.best_chaintip(&snapshot).await?; + let id = HashOrHeight::new(&hash_or_height, Some(tip.height.into())).map_err(|error| { + ChainIndexError::internal_from( + zaino_fetch::jsonrpsee::connector::RpcError::new_from_legacycode( + zebra_rpc::server::error::LegacyCode::InvalidParameter, + error, + ), + ) + })?; + self.source() + .get_block_verbose(id, verbosity) + .await + .map_err(ChainIndexError::backing_validator) } - /// Gets the subtree roots of a given pool and the end heights of each root, - /// starting at the provided index, up to an optional maximum number of roots. - async fn get_subtree_roots( + async fn get_block_header( &self, - pool: ShieldedPool, - start_index: u16, - max_entries: Option, - ) -> Result, Self::Error> { + hash: String, + verbose: bool, + ) -> Result { self.source() - .get_subtree_roots(pool, start_index, max_entries) + .get_block_header(hash, verbose) .await .map_err(ChainIndexError::backing_validator) } - // ********** Transparent address history methods ********** + // TODO(internal-first): `getblockdeltas` is buildable from the indexed chainblocks + // + finalised/non-finalised prevout resolution. Build it internally by default once + // an internal prevout resolver (spanning non-finalised + finalised, reconstructing + // addresses from `TxOutCompact`) exists, keeping this source call as the fallback. + async fn get_block_deltas(&self, hash: String) -> Result { + self.source() + .get_block_deltas(hash) + .await + .map_err(ChainIndexError::backing_validator) + } - /// Returns all changes for the given transparent addresses. - async fn get_address_deltas( - &self, - params: GetAddressDeltasParams, - ) -> Result { + // `getdifficulty` is the difficulty-adjusted expected difficulty (over a block + // window), not the tip block's stored bits, so it cannot be built from indexed data: + // always delegate to the backing validator. + async fn get_difficulty(&self) -> Result { self.source() - .get_address_deltas(params) + .get_difficulty() .await .map_err(ChainIndexError::backing_validator) } - /// Returns the total transparent balance for the given addresses. - async fn get_address_balance( - &self, - address_strings: GetAddressBalanceRequest, - ) -> Result { + async fn get_info(&self) -> Result { self.source() - .get_address_balance(address_strings) + .get_info() .await .map_err(ChainIndexError::backing_validator) } - /// Returns the transaction ids made by the given transparent addresses. - async fn get_address_txids( - &self, - request: GetAddressTxIdsRequest, - ) -> Result, Self::Error> { + // `getblockchaininfo` needs cumulative pool value balances (TipPoolValues) and on-disk + // size, which are not in the ChainIndex's indexed data, so it cannot be built + // internally: always delegate to the backing validator. + async fn get_blockchain_info(&self) -> Result { self.source() - .get_address_txids(request) + .get_blockchain_info() .await .map_err(ChainIndexError::backing_validator) } - /// Returns all unspent transparent outputs for the given addresses. - async fn get_address_utxos( - &self, - address_strings: GetAddressBalanceRequest, - ) -> Result, Self::Error> { + async fn get_peer_info(&self) -> Result { self.source() - .get_address_utxos(address_strings) + .get_peer_info() .await .map_err(ChainIndexError::backing_validator) } - async fn get_outpoint_spenders( - &self, - snapshot: &Self::Snapshot, - outpoints: Vec, - scope: types::ChainScope, - ) -> Result>, Self::Error> { - use std::collections::HashMap; + async fn get_block_subsidy(&self, height: u32) -> Result { + self.source() + .get_block_subsidy(height) + .await + .map_err(ChainIndexError::backing_validator) + } - let mut result: Vec> = vec![None; outpoints.len()]; + async fn get_mining_info(&self) -> Result { + self.source() + .get_mining_info() + .await + .map_err(ChainIndexError::backing_validator) + } - // 1) Non-finalised best chain (FullChain scope only). Scan only the blocks reachable - // via `heights_to_hashes` (the `blocks` map also holds reorged-away blocks, which - // must not count). One pass builds an outpoint -> spending-txid map regardless of - // how many outpoints we look up. Under `Finalised` scope this is skipped so results - // are reorg-stable. - if let ( - types::ChainScope::FullChain, - ChainIndexSnapshot::NonFinalizedStateExists { - non_finalized_snapshot, - }, - ) = (scope, snapshot) - { - let mut nfs_spenders: HashMap = HashMap::new(); - for hash in non_finalized_snapshot.heights_to_hashes.values() { - let Some(block) = non_finalized_snapshot.blocks.get(hash) else { - continue; - }; - for tx in block.transactions() { - let txid = *tx.txid(); - // `spent_outpoints` already skips coinbase null prevouts and builds each - // `Outpoint`, keeping the construction in one place (see #1332). - for outpoint in tx.transparent().spent_outpoints() { - nfs_spenders.insert(outpoint, txid); - } - } - } - for (i, outpoint) in outpoints.iter().enumerate() { - if let Some(txid) = nfs_spenders.get(outpoint) { - result[i] = Some(*txid); - } - } - } + async fn get_tx_out( + &self, + txid: String, + n: u32, + include_mempool: Option, + ) -> Result { + self.source() + .get_tx_out(txid, n, include_mempool) + .await + .map_err(ChainIndexError::backing_validator) + } - // 2) Finalised lookup for the still-unresolved outpoints, batched into one DB call. - let unresolved_indices: Vec = result - .iter() - .enumerate() - .filter_map(|(i, r)| r.is_none().then_some(i)) - .collect(); - if unresolved_indices.is_empty() { - return Ok(result); - } - let unresolved_outpoints: Vec = - unresolved_indices.iter().map(|&i| outpoints[i]).collect(); - let locations = self - .finalized_state - .get_outpoint_spenders(unresolved_outpoints) - .await?; + async fn get_spent_info( + &self, + request: GetSpentInfoRequest, + ) -> Result { + self.source() + .get_spent_info(request) + .await + .map_err(ChainIndexError::backing_validator) + } - // 3) Resolve each finalised `TxLocation` to a txid. Dedup identical locations so a - // block spending several queried outpoints is only fetched once. `get_txid` is a - // single keyed lookup, far cheaper than reconstructing the whole block. - let mut slots_by_location: HashMap> = HashMap::new(); - for (slot, location) in unresolved_indices.into_iter().zip(locations) { - if let Some(location) = location { - slots_by_location.entry(location).or_default().push(slot); - } - } - for (location, slots) in slots_by_location { - let txid = self.finalized_state.get_txid(location).await?; - for slot in slots { - result[slot] = Some(txid); - } - } + async fn get_network_sol_ps( + &self, + blocks: Option, + height: Option, + ) -> Result { + self.source() + .get_network_sol_ps(blocks, height) + .await + .map_err(ChainIndexError::backing_validator) + } - Ok(result) + async fn send_raw_transaction( + &self, + raw_transaction_hex: String, + ) -> Result { + validate_raw_transaction_hex(&raw_transaction_hex) + .map_err(ChainIndexError::internal_from)?; + self.source() + .send_raw_transaction(raw_transaction_hex) + .await + .map_err(ChainIndexError::backing_validator) } - // ********** Metadata methods ********** + async fn get_treestate_by_id( + &self, + hash_or_height: String, + ) -> Result { + self.source() + .get_treestate_by_id(hash_or_height) + .await + .map_err(ChainIndexError::backing_validator) + } + + /// Returns all changes for the given transparent addresses. + async fn get_address_deltas( + &self, + params: GetAddressDeltasParams, + ) -> Result { + self.source() + .get_address_deltas(params) + .await + .map_err(ChainIndexError::backing_validator) + } /// Returns Information about the mempool state: /// - size: Current tx count @@ -2584,39 +2928,6 @@ impl ChainIndex for NodeBackedChainIndexSubscriber Result { - Ok(match snapshot { - ChainIndexSnapshot::NonFinalizedStateExists { - non_finalized_snapshot, - } => non_finalized_snapshot.best_tip, - - ChainIndexSnapshot::StillSyncingFinalizedState { - validator_finalized_height, - } => { - BlockIndex { - height: *validator_finalized_height, - hash: self - .source() - // TODO: do something more efficient than getting the whole block - .get_block(HashOrHeight::Height((*validator_finalized_height).into())) - .await - .map_err(|e| { - ChainIndexError::database_hole( - validator_finalized_height, - Some(Box::new(e)), - ) - })? - .ok_or(ChainIndexError::database_hole( - validator_finalized_height, - None, - ))? - .hash() - .into(), - } - } - }) - } - async fn get_tx_out_set_info(&self) -> Result { use crate::chain_index::types::db::metadata::{ is_unspendable_tx_out, ZAINO_TXOUTSET_ENTRY_LEN, diff --git a/packages/zaino-state/src/chain_index/finalised_state.rs b/packages/zaino-state/src/chain_index/finalised_state.rs index 11c824ec1..1a1915e82 100644 --- a/packages/zaino-state/src/chain_index/finalised_state.rs +++ b/packages/zaino-state/src/chain_index/finalised_state.rs @@ -243,6 +243,33 @@ use crate::{ use std::{sync::Arc, time::Duration}; use tokio::time::{interval, MissedTickBehavior}; +/// The activation heights of the three shielded pools whose data +/// [`build_indexed_block_from_source`] assembles, resolved once per run. +/// +/// Both the ingestion loop ([`capability::DbWrite::write_blocks_to_height`]) and the v1.2.1 → +/// v1.3.0 migration backfill need exactly this set of heights; resolving them in one place keeps +/// the two call sites from drifting apart. A `None` height means the pool's network upgrade has no +/// activation height on the given network. +struct PoolActivationHeights { + sapling: Option, + nu5: Option, + nu6_3: Option, +} + +impl PoolActivationHeights { + /// Resolves the Sapling, NU5 (Orchard), and NU6.3 (Ironwood) activation heights on + /// `zebra_network`. + fn resolve(zebra_network: &zebra_chain::parameters::Network) -> Self { + let activation_height = + |pool: super::ShieldedPool| pool.activation_upgrade().activation_height(zebra_network); + Self { + sapling: activation_height(super::ShieldedPool::Sapling), + nu5: activation_height(super::ShieldedPool::Orchard), + nu6_3: activation_height(super::ShieldedPool::Ironwood), + } + } +} + /// Fetches the block at `height_int` from `source` and builds its [`IndexedBlock`], threading /// `parent_chainwork` into the block metadata. /// @@ -251,7 +278,7 @@ use tokio::time::{interval, MissedTickBehavior}; /// owns the loop. pub(crate) async fn build_indexed_block_from_source( source: &S, - network: zaino_common::Network, + network: zebra_chain::parameters::Network, sapling_activation_height: zebra_chain::block::Height, nu5_activation_height: Option, nu6_3_activation_height: Option, @@ -312,7 +339,7 @@ pub(crate) async fn build_indexed_block_from_source( orchard_size, ironwood, parent_chainwork, - network.to_zebra_network(), + network, ); let block_with_metadata = BlockWithMetadata::new(block.as_ref(), metadata); @@ -464,7 +491,7 @@ impl FinalisedState { return Ok(Self { db: Arc::new(Router::new(Arc::new(FinalisedSource::ephemeral( source, - cfg.network.into(), + cfg.network.clone(), None, )))), cfg, @@ -543,12 +570,25 @@ impl FinalisedState { source: migration_source, }; - if let Err(error) = migration_manager.migrate().await { - tracing::error!("FinalisedState migration failed: {error}"); + match migration_manager.migrate().await { + Ok(()) => { + // Start the background validator only now that every migration has + // finished: its initial scan reads tables a migration populates (e.g. + // `commitment_tree_data_1_3_0`), so starting it earlier would race the + // migration and fail on a not-yet-written row. + migration_router.primary_backend().start_validator(); + } + Err(error) => { + tracing::error!("FinalisedState migration failed: {error}"); - migration_router.store_primary_status(StatusType::CriticalError); + migration_router.store_primary_status(StatusType::CriticalError); + } } }); + } else { + // No migration to run, so the on-disk tables the validator scans are already at the + // current schema: start it immediately. + router.primary_backend().start_validator(); } Ok(Self { db: router, cfg }) @@ -658,7 +698,7 @@ impl FinalisedState { /// - `Some(version)` if a compatible database directory is found, /// - `None` if no database is detected (fresh DB creation case). async fn try_find_current_db_version(cfg: &ChainIndexConfig) -> Option { - let legacy_dir = match cfg.network.to_zebra_network().kind() { + let legacy_dir = match cfg.network.kind() { NetworkKind::Mainnet => "live", NetworkKind::Testnet => "test", NetworkKind::Regtest => "local", @@ -668,7 +708,7 @@ impl FinalisedState { return Some(0); } - let net_dir = match cfg.network.to_zebra_network().kind() { + let net_dir = match cfg.network.kind() { NetworkKind::Mainnet => "mainnet", NetworkKind::Testnet => "testnet", NetworkKind::Regtest => "regtest", @@ -786,7 +826,7 @@ impl FinalisedState { let ephemeral_reference = router .init_or_take_ephemeral( source.clone(), - cfg.network.to_zebra_network(), + cfg.network.clone(), EphemeralMode::ReadOnly, db_height_opt, ) @@ -1053,6 +1093,14 @@ impl FinalisedState { migration_manager.migrate().await?; } + // This test helper builds a fixture at an arbitrary (often intermediate) version for + // inspection, so it deliberately does NOT start the validator: the validator only validates + // against the current schema and would fail on an intermediate-version database. The + // foreground migration is already complete, so mark the primary `Ready` directly (as + // `spawn_v1_0_0` does) to give callers a settled backend. Validation is exercised through + // the production `FinalisedState::spawn` path, which always targets the current schema. + router.store_primary_status(StatusType::Ready); + let metadata = router.get_metadata().await?; if metadata.version() != target_version { return Err(FinalisedStateError::Custom(format!( @@ -1084,7 +1132,6 @@ impl FinalisedState { source: T, ) -> Result, FinalisedStateError> { let db = FinalisedSource::spawn_v1_0_0(cfg).await?; - db.wait_until_ready().await; let tip = source.get_best_block_height().await?.ok_or_else(|| { FinalisedStateError::BlockchainSourceError(BlockchainSourceError::Unrecoverable( @@ -1098,7 +1145,7 @@ impl FinalisedState { // defaults — mirroring `build_indexed_block_from_source`. let nu6_3_activation_height = super::ShieldedPool::Ironwood .activation_upgrade() - .activation_height(&cfg.network.to_zebra_network()); + .activation_height(&cfg.network); let mut parent_chainwork: Option = None; @@ -1145,7 +1192,7 @@ impl FinalisedState { orchard_size, ironwood, parent_chainwork, - cfg.network.to_zebra_network(), + cfg.network.clone(), ); let block_with_metadata = BlockWithMetadata::new(block.as_ref(), metadata); let chain_block = IndexedBlock::try_from(block_with_metadata).map_err(|_| { diff --git a/packages/zaino-state/src/chain_index/finalised_state/finalised_source.rs b/packages/zaino-state/src/chain_index/finalised_state/finalised_source.rs index cae4c96e3..d2aeb1cce 100644 --- a/packages/zaino-state/src/chain_index/finalised_state/finalised_source.rs +++ b/packages/zaino-state/src/chain_index/finalised_state/finalised_source.rs @@ -283,6 +283,18 @@ impl FinalisedSource { } } + /// Start the background validator on the primary v1 backend (no-op for the ephemeral + /// passthrough). + /// + /// Called by the orchestrator only once all pending migrations have completed, so the + /// validator never races a migration that populates the tables its initial scan reads. + pub(super) fn start_validator(&self) { + match self { + Self::V1(db) => db.start_validator(), + Self::Ephemeral(_) => {} + } + } + /// Stores a new runtime status in the concrete backend. /// /// This is used by router-level background orchestration, for example to report an asynchronous @@ -368,6 +380,12 @@ impl FinalisedSource { .require_v1("v1 commitment_tree_data db")? .commitment_tree_data_db()) } + + /// Provides access to the (v1.3.0) `ironwood` table, required for Migration1_2_1To1_3_0 to + /// backfill ironwood rows from validator-fetched block data. + pub(super) fn ironwood_db(&self) -> Result { + Ok(self.require_v1("v1 ironwood db")?.ironwood_db()) + } } impl From for FinalisedSource { diff --git a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1.rs b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1.rs index 73ed8b717..bd241b01b 100644 --- a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1.rs +++ b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1.rs @@ -405,16 +405,22 @@ pub(crate) struct DbV1 { /// - validated read fetchers used by the capability trait implementations, and /// - internal validation / indexing helpers. impl DbV1 { - /// Spawns a new [`DbV1`] and opens (or creates) the LMDB environment for the configured network. + /// Opens (and heals) the v1 database for the configured network **without** starting the + /// background validator. /// /// This method: /// - chooses a versioned path suffix (`...//v1`), /// - configures LMDB map size and reader slots, - /// - opens or creates all V1 named databases, - /// - validates or initializes the `"metadata"` record (schema hash + version), and - /// - spawns the background validator / maintenance task. + /// - opens or creates all V1 named databases, and + /// - validates or initializes the `"metadata"` record (schema hash + version). + /// + /// The validator is started separately via [`DbV1::start_validator`]. This split exists so the + /// orchestrator can guarantee that any pending schema migration finishes *before* validation + /// runs: the validator's `initial_block_scan` reads tables (e.g. `commitment_tree_data_1_3_0`) + /// that a migration populates, so starting it concurrently with a migration races the migration + /// and can fail on a not-yet-written row. pub(crate) async fn spawn(config: &ChainIndexConfig) -> Result { - let mut zaino_db = Self::open_env_and_dbs(config).await?; + let zaino_db = Self::open_env_and_dbs(config).await?; // Validate (or initialise) the metadata entry before we touch any tables. zaino_db.check_schema_version().await?; @@ -424,9 +430,6 @@ impl DbV1 { // on a quiescent database. zaino_db.reconcile_alpha_txid_location_index().await?; - // Spawn handler task to perform background validation and trailing tx cleanup. - zaino_db.spawn_handler().await?; - Ok(zaino_db) } @@ -453,7 +456,7 @@ impl DbV1 { // Prepare database details and path. let db_size_bytes = config.storage.database.size.to_byte_count(); - let db_path_dir = match config.network.to_zebra_network().kind() { + let db_path_dir = match config.network.kind() { NetworkKind::Mainnet => "mainnet", NetworkKind::Testnet => "testnet", NetworkKind::Regtest => "regtest", @@ -598,7 +601,12 @@ impl DbV1 { /// `initial_block_scan`). /// - **Steady state:** periodically attempts to validate the next height after `validated_tip`. /// Separately, it performs periodic trailing-reader cleanup via `clean_trailing()`. - async fn spawn_handler(&mut self) -> Result<(), FinalisedStateError> { + /// + /// Kept separate from [`DbV1::spawn`] so the orchestrator starts it only once all pending + /// migrations have finished (the validator reads tables a migration populates). Takes `&self` + /// (the join handle lives behind a `Mutex`) so it can be driven through the shared + /// `Arc` the router holds after spawn. + pub(super) fn start_validator(&self) { // Clone everything the task needs so we can move it into the async block. let zaino_db = self.detached_handle(); @@ -700,7 +708,6 @@ impl DbV1 { }); *self.db_handler.lock().expect("db_handler mutex poisoned") = Some(handle); - Ok(()) } /// Validates every stored spent-outpoint entry (`Outpoint` -> `TxLocation`) by checksum. @@ -843,6 +850,12 @@ impl DbV1 { self.commitment_tree_data } + /// Provides access to the (v1.3.0) `ironwood` table, required for Migration1_2_1To1_3_0 to + /// backfill ironwood rows for post-NU6.3 blocks from validator-fetched block data. + pub(super) fn ironwood_db(&self) -> Database { + self.ironwood + } + /// Provudes access to the spent DB table, required for Migration1_1_0To1_2_0. pub(crate) fn spent_db(&self) -> Database { self.spent @@ -1016,9 +1029,16 @@ impl DbV1 { // (see the method doc) — that is the behavioural difference from `spawn`. zaino_db.write_v1_0_0_metadata()?; - // Spawn handler task to perform background validation and trailing tx cleanup. - let mut zaino_db = zaino_db; - zaino_db.spawn_handler().await?; + // Deliberately does NOT start the background validator. This builds a *pre-migration* + // v1.0.0 fixture; the validator validates against the current (v1.3.0) schema — it reads + // `commitment_tree_data_1_3_0` and the `ironwood` table — so it must run only after the + // database has been migrated to the newest schema. Callers build the fixture with direct + // v1.0.0 writes, shut it down, then reopen through `FinalisedState::spawn`, which migrates + // first and starts the validator afterwards. + // + // With no validator to advance it, mark the empty fixture `Ready` directly so callers see a + // settled backend. + zaino_db.status.store(StatusType::Ready); Ok(zaino_db) } @@ -1108,7 +1128,7 @@ mod tests { async fn initial_spent_scan_reports_corrupt_value() { use lmdb::{Transaction as _, WriteFlags}; use zaino_common::network::ActivationHeights; - use zaino_common::{DatabaseConfig, Network, StorageConfig}; + use zaino_common::{DatabaseConfig, StorageConfig}; let temp_dir = tempfile::tempdir().expect("tempdir"); let config = ChainIndexConfig { @@ -1121,7 +1141,7 @@ mod tests { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let db = DbV1::spawn(&config).await.expect("spawn empty v1 db"); diff --git a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/tx_out_set_accumulator.rs b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/tx_out_set_accumulator.rs index e254e70f0..df3aad9ca 100644 --- a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/tx_out_set_accumulator.rs +++ b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/tx_out_set_accumulator.rs @@ -1710,7 +1710,7 @@ mod tests { use std::sync::Arc; use tempfile::TempDir; use zaino_common::network::ActivationHeights; - use zaino_common::{DatabaseConfig, Network, StorageConfig, SyncWriteBatchSize}; + use zaino_common::{DatabaseConfig, StorageConfig, SyncWriteBatchSize}; fn p2pkh_out(value: u64) -> TxOutCompact { TxOutCompact::new(value, [0x11; 20], 0).expect("P2PKH script_type should be valid") @@ -2111,7 +2111,7 @@ mod tests { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let zaino_db = FinalisedState::spawn(config, source.clone()).await.unwrap(); @@ -2186,7 +2186,7 @@ mod tests { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let zaino_db = FinalisedState::spawn(config, source.clone()).await.unwrap(); diff --git a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/write_core.rs b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/write_core.rs index 212ac75d3..c833d7abc 100644 --- a/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/write_core.rs +++ b/packages/zaino-state/src/chain_index/finalised_state/finalised_source/v1/write_core.rs @@ -2,8 +2,6 @@ use super::*; -use crate::chain_index::ShieldedPool; - #[cfg(feature = "prometheus")] use crate::metric_names::*; @@ -188,13 +186,35 @@ fn build_block_row_entries( ), sapling_entry: StoredEntryVar::new(block_height_bytes, SaplingTxList::new(sapling)), orchard_entry: StoredEntryVar::new(block_height_bytes, OrchardTxList::new(orchard)), - ironwood_entry: ironwood - .iter() - .any(Option::is_some) - .then(|| StoredEntryVar::new(block_height_bytes, OrchardTxList::new(ironwood))), + ironwood_entry: ironwood_entry(ironwood, block_height_bytes), } } +/// Builds the sparse ironwood row entry for a block's per-transaction ironwood list: `Some` only +/// when at least one transaction carries ironwood data, so a block with none stores no ironwood row +/// (readers treat an absent row as "no ironwood data"). +fn ironwood_entry( + ironwood: Vec>, + block_height_bytes: &[u8], +) -> Option> { + ironwood + .iter() + .any(Option::is_some) + .then(|| StoredEntryVar::new(block_height_bytes, OrchardTxList::new(ironwood))) +} + +/// Builds the sparse ironwood row entry for `block` (the same value the write path stores). Used by +/// the v1.2.1 → v1.3.0 migration to backfill the ironwood table from validator-fetched blocks. +pub(crate) fn build_block_ironwood_entry( + block: &IndexedBlock, + block_height_bytes: &[u8], +) -> Result>, FinalisedStateError> { + Ok(ironwood_entry( + extract_block_pool_lists(block)?.ironwood, + block_height_bytes, + )) +} + impl DbWrite for DbV1 { async fn write_block(&self, block: IndexedBlock) -> Result<(), FinalisedStateError> { self.write_block(block).await @@ -209,20 +229,17 @@ impl DbWrite for DbV1 { height: Height, source: &S, ) -> Result<(), FinalisedStateError> { - use crate::chain_index::finalised_state::build_indexed_block_from_source; + use crate::chain_index::finalised_state::{ + build_indexed_block_from_source, PoolActivationHeights, + }; - let network = self.config.network; - let zebra_network = network.to_zebra_network(); - let sapling_activation_height = ShieldedPool::Sapling - .activation_upgrade() - .activation_height(&zebra_network) + let zebra_network = self.config.network.clone(); + let pool_activations = PoolActivationHeights::resolve(&zebra_network); + let sapling_activation_height = pool_activations + .sapling .expect("Sapling activation height must be set"); - let nu5_activation_height = ShieldedPool::Orchard - .activation_upgrade() - .activation_height(&zebra_network); - let nu6_3_activation_height = ShieldedPool::Ironwood - .activation_upgrade() - .activation_height(&zebra_network); + let nu5_activation_height = pool_activations.nu5; + let nu6_3_activation_height = pool_activations.nu6_3; // Seed `parent_chainwork` from the current tip header (the block before the first one we // write). On an empty database this is genesis with zero chainwork. Read raw rather than via @@ -264,7 +281,7 @@ impl DbWrite for DbV1 { info!( start_height, target = height.0, - ?network, + ?zebra_network, "write_blocks_to_height: syncing finalised blocks" ); @@ -309,7 +326,7 @@ impl DbWrite for DbV1 { let build_start = std::time::Instant::now(); let block = build_indexed_block_from_source( source, - network, + zebra_network.clone(), sapling_activation_height, nu5_activation_height, nu6_3_activation_height, @@ -336,7 +353,7 @@ impl DbWrite for DbV1 { info!( current = next - 1, target = height.0, - ?network, + ?zebra_network, "write_blocks_to_height: syncing" ); last_progress_log = std::time::Instant::now(); diff --git a/packages/zaino-state/src/chain_index/finalised_state/migrations.rs b/packages/zaino-state/src/chain_index/finalised_state/migrations.rs index b88009194..3e5f5bd51 100644 --- a/packages/zaino-state/src/chain_index/finalised_state/migrations.rs +++ b/packages/zaino-state/src/chain_index/finalised_state/migrations.rs @@ -343,7 +343,7 @@ impl MigrationManager { .router .init_or_take_ephemeral( self.source.clone(), - self.cfg.network.to_zebra_network(), + self.cfg.network.clone(), EphemeralMode::Full, db_height, ) @@ -491,6 +491,34 @@ impl Migration for Migration1_0_0To1_1_0 { /// - No shadow database. struct Migration1_1_0To1_2_0; +/// Writes `value` under `key` with `NO_OVERWRITE`, tolerating an existing byte-identical row. +/// +/// Migrations use this to stay idempotent on crash-resume: a resumed pass may revisit rows it has +/// already committed, and each such row must match the rebuilt bytes exactly. Any difference — +/// a conflicting rebuild or a corrupt existing row — aborts the migration with the message +/// `describe` produces. `describe` runs only on that error path, so the per-row success path +/// allocates nothing. +fn put_idempotent( + txn: &mut lmdb::RwTransaction<'_>, + db: lmdb::Database, + key: &[u8], + value: &[u8], + describe: impl FnOnce() -> String, +) -> Result<(), FinalisedStateError> { + match txn.put(db, &key, &value, WriteFlags::NO_OVERWRITE) { + Ok(()) => Ok(()), + Err(lmdb::Error::KeyExist) => { + let existing = txn.get(db, &key).map_err(FinalisedStateError::LmdbError)?; + if existing == value { + Ok(()) + } else { + Err(FinalisedStateError::Custom(describe())) + } + } + Err(error) => Err(FinalisedStateError::LmdbError(error)), + } +} + /// Flushes a buffered batch of `spent` index entries in sorted key order, then commits them /// together with the Stage B progress watermark and fsyncs. /// @@ -511,26 +539,12 @@ fn flush_migration_spent_batch( let mut txn = env.begin_rw_txn()?; for (outpoint_bytes, tx_location) in buffer.iter() { let entry_bytes = StoredEntryFixed::new(outpoint_bytes, *tx_location).to_bytes()?; - match txn.put( - spent_db, - outpoint_bytes, - &entry_bytes, - WriteFlags::NO_OVERWRITE, - ) { - Ok(()) => {} - Err(lmdb::Error::KeyExist) => { - let existing = txn - .get(spent_db, outpoint_bytes) - .map_err(FinalisedStateError::LmdbError)?; - if existing != entry_bytes { - return Err(FinalisedStateError::Custom(format!( - "conflicting existing spent entry during batched migration for outpoint {}", - hex::encode(outpoint_bytes) - ))); - } - } - Err(error) => return Err(FinalisedStateError::LmdbError(error)), - } + put_idempotent(&mut txn, spent_db, outpoint_bytes, &entry_bytes, || { + format!( + "conflicting existing spent entry during batched migration for outpoint {}", + hex::encode(outpoint_bytes) + ) + })?; } let progress = StoredEntryFixed::new(progress_key, up_to_height + 1); @@ -689,42 +703,20 @@ impl Migration for Migration1_1_0To1_2_0 { let entry_bytes = StoredEntryFixed::new(txid_bytes, *tx_location).to_bytes()?; - match txn.put( + // Idempotent on resume: an existing entry must match byte-for-byte, which + // also rejects a corrupt existing row (its checksum bytes would differ). + put_idempotent( + &mut txn, txid_location_db, txid_bytes, &entry_bytes, - WriteFlags::NO_OVERWRITE, - ) { - Ok(()) => {} - - // Idempotent on resume: an existing entry must match exactly. - Err(lmdb::Error::KeyExist) => { - let existing_bytes = txn - .get(txid_location_db, txid_bytes) - .map_err(FinalisedStateError::LmdbError)?; - let existing_entry = - StoredEntryFixed::::from_bytes(existing_bytes) - .map_err(|error| { - FinalisedStateError::Custom(format!( - "corrupt existing txid_location entry: {error}" - )) - })?; - if !existing_entry.verify(txid_bytes) { - return Err(FinalisedStateError::Custom( - "existing txid_location entry checksum mismatch" - .to_string(), - )); - } - if existing_entry.inner() != tx_location { - return Err(FinalisedStateError::Custom(format!( - "conflicting txid_location entry at height {}", - height.0 - ))); - } - } - - Err(error) => return Err(FinalisedStateError::LmdbError(error)), - } + || { + format!( + "conflicting or corrupt existing txid_location entry at height {}", + height.0 + ) + }, + )?; } let progress = @@ -1017,27 +1009,28 @@ impl Migration for Migration1_2_0To1_2_1 { /// Minor migration: v1.2.1 → v1.3.0. /// /// Introduces the Ironwood (NU6.3) shielded pool to the finalised state: -/// - a new per-height `ironwood` table (`ironwood_1_3_0`), created empty by `DbV1::spawn`; new -/// blocks populate it via the write path. No backfill is needed because every block already on -/// disk predates NU6.3 (see the guard below). +/// - a new per-height `ironwood` table (`ironwood_1_3_0`), created empty by `DbV1::spawn`. New +/// blocks populate it via the write path; here it is backfilled for any stored block at or above +/// NU6.3 activation (see below). /// - the `commitment_tree_data` table is rebuilt from the legacy fixed-length /// `StoredEntryFixed` (V1) rows into the variable-length /// `StoredEntryVar` (V2) layout, which carries the optional Ironwood root and -/// size. The rebuild reads the legacy `commitment_tree_data_1_0_0` table and writes the new -/// `commitment_tree_data_1_3_0` table (opened as the primary's `commitment_tree_data` handle), -/// then clears the legacy table. +/// size. The new rows are written to `commitment_tree_data_1_3_0` (the primary's +/// `commitment_tree_data` handle), then the legacy table is cleared. /// -/// This is a **rebuild from existing on-disk data** (no validator refetch), correct only while every -/// stored block predates Ironwood. The migration therefore asserts the database tip is below NU6.3 -/// activation before running; a database already synced past NU6.3 under the old schema would be -/// missing ironwood data that cannot be reconstructed from existing rows and must be re-indexed from -/// the validator instead. +/// Per-height rebuild strategy, branching on NU6.3 activation: +/// - **Below activation:** rebuilt in place from the legacy on-disk commitment row (Ironwood +/// root/size default to `None`/`0` via `CommitmentTreeData` V1 decode). No validator access. +/// - **At or above activation:** the legacy data predates Ironwood, so both the commitment row +/// (now carrying the Ironwood root/size) and the sparse ironwood row are rebuilt from +/// validator-fetched block data via [`build_indexed_block_from_source`]. This lets the migration +/// run on a database already synced past NU6.3, rather than forcing a full re-index. /// /// Safety and resumability: -/// - Deterministic: each rebuilt row is derived only from the matching legacy row (Ironwood -/// root/size default to `None`/`0` via `CommitmentTreeData` V1 decode). +/// - Deterministic: a below-activation row is derived only from its legacy row; an at/above-activation +/// row is refetched from immutable finalised history, so a resumed rebuild reproduces the same bytes. /// - Resumable: the next height to rebuild is stored in the metadata DB under a temporary key. -/// - Crash-safe: each height's rebuilt row and the progress update commit in the same transaction; +/// - Crash-safe: each height's rebuilt rows and the progress update commit in the same transaction; /// idempotent on resume (`NO_OVERWRITE` + verify-match). struct Migration1_2_1To1_3_0; @@ -1062,10 +1055,15 @@ impl Migration for Migration1_2_1To1_3_0 { &self, router: Arc>, cfg: ChainIndexConfig, - _source: T, + source: T, ) -> Result<(), FinalisedStateError> { use lmdb::DatabaseFlags; + use crate::chain_index::finalised_state::{ + build_indexed_block_from_source, + finalised_source::v1::write_core::build_block_ironwood_entry, PoolActivationHeights, + }; + // Temporary metadata entry recording the next height to rebuild, removed on completion. const MIGRATION_CTD_PROGRESS_KEY: &[u8] = b"_migration_commitment_tree_data_progress_1_3_0_next_height"; @@ -1077,8 +1075,10 @@ impl Migration for Migration1_2_1To1_3_0 { let backend = router.primary_backend(); let env = backend.env()?; let metadata_db = backend.metadata_db()?; - // The primary's `commitment_tree_data` handle is the new `commitment_tree_data_1_3_0` table. + // The primary's `commitment_tree_data` handle is the new `commitment_tree_data_1_3_0` table; + // `ironwood_db` is the new (v1.3.0) sparse ironwood table backfilled below. let new_ctd_db = backend.commitment_tree_data_db()?; + let ironwood_db = backend.ironwood_db()?; // Open the legacy fixed-length commitment table by name. On a pre-v1.3.0 database it already // exists; `open_or_create_db` creating it empty on an unexpected fresh DB is harmless (the @@ -1091,12 +1091,15 @@ impl Migration for Migration1_2_1To1_3_0 { ) .await?; - // Guard: Ironwood data is only expected from NU6.3 activation. A rebuild-from-existing-data - // migration cannot reconstruct ironwood roots/sizes for post-NU6.3 blocks. - let zebra_network = cfg.network.to_zebra_network(); - let nu6_3_activation_height = crate::chain_index::ShieldedPool::Ironwood - .activation_upgrade() - .activation_height(&zebra_network); + // Network-upgrade activation heights (shared with `write_blocks_to_height`). Blocks at or + // above NU6.3 have their ironwood root/size and ironwood tx list rebuilt from the validator + // (the legacy on-disk data predates ironwood); blocks below it are rebuilt in place from + // the legacy commitment row, with no ironwood. + let network = cfg.network.clone(); + let pool_activations = PoolActivationHeights::resolve(&network); + let sapling_activation_height = pool_activations.sapling; + let nu5_activation_height = pool_activations.nu5; + let nu6_3_activation_height = pool_activations.nu6_3; // Mark migration in progress (observability only; resumption uses the progress key). { @@ -1133,17 +1136,6 @@ impl Migration for Migration1_2_1To1_3_0 { if let Some(db_tip) = backend.db_height().await? { let db_tip = db_tip.0; - if let Some(activation) = nu6_3_activation_height { - if db_tip >= activation.0 { - return Err(FinalisedStateError::Custom(format!( - "cannot migrate finalised state to v1.3.0: tip {db_tip} is at or above NU6.3 \ - activation {} but was built without Ironwood data; wipe and re-index the \ - finalised state from the validator", - activation.0 - ))); - } - } - let mut next_height = read_progress(MIGRATION_CTD_PROGRESS_KEY)?.unwrap_or(GENESIS_HEIGHT.0); @@ -1159,53 +1151,97 @@ impl Migration for Migration1_2_1To1_3_0 { .map_err(|error| FinalisedStateError::Custom(error.to_string()))?; let height_bytes = height.to_bytes()?; - // Read + verify the legacy fixed-length row. - let commitment_tree_data: CommitmentTreeData = { - let txn = env.begin_ro_txn()?; - let raw = txn - .get(legacy_ctd_db, &height_bytes) - .map_err(FinalisedStateError::LmdbError)?; - let entry = StoredEntryFixed::::from_bytes(raw).map_err( - |error| { - FinalisedStateError::Custom(format!( - "legacy commitment_tree_data corrupt data: {error}" - )) - }, - )?; - if !entry.verify(&height_bytes) { - return Err(FinalisedStateError::Custom( - "legacy commitment_tree_data checksum mismatch".to_string(), - )); - } - *entry.inner() - }; + let ironwood_active = + nu6_3_activation_height.is_some_and(|activation| next_height >= activation.0); + + // Prepare the rows to write. Reads / validator fetches happen before the write txn + // so the commit stays short. + let commitment_bytes: Vec; + let ironwood_bytes: Option>; + + if ironwood_active { + // Post-NU6.3: the legacy row carries no ironwood root/size and the ironwood + // table has no row, so rebuild both from validator-fetched block data. + let sapling_activation_height = sapling_activation_height.ok_or_else(|| { + FinalisedStateError::Custom( + "Sapling activation height must be set to backfill ironwood" + .to_string(), + ) + })?; + let block = build_indexed_block_from_source( + &source, + network.clone(), + sapling_activation_height, + nu5_activation_height, + nu6_3_activation_height, + next_height, + // Chainwork is irrelevant here: only the commitment-tree and ironwood rows + // are extracted, and neither depends on it. + None, + ) + .await + .map_err(|error| { + FinalisedStateError::Custom(format!( + "v1.3.0 ironwood backfill failed at height {next_height}: {error}. \ + This backfill refetches every stored block from NU6.3 activation \ + through the database tip; ensure the backing validator serves that \ + range, or wipe the finalised-state directory and re-index from the \ + validator." + )) + })?; - // Re-wrap in the V2 `StoredEntryVar` layout and advance progress atomically. - let new_entry_bytes = - StoredEntryVar::new(&height_bytes, commitment_tree_data).to_bytes()?; + commitment_bytes = + StoredEntryVar::new(&height_bytes, *block.commitment_tree_data()) + .to_bytes()?; + ironwood_bytes = match build_block_ironwood_entry(&block, &height_bytes)? { + Some(entry) => Some(entry.to_bytes()?), + None => None, + }; + } else { + // Pre-NU6.3: rebuild the commitment row in place from the legacy fixed-length + // row (ironwood defaults to none). + let commitment_tree_data: CommitmentTreeData = { + let txn = env.begin_ro_txn()?; + let raw = txn + .get(legacy_ctd_db, &height_bytes) + .map_err(FinalisedStateError::LmdbError)?; + let entry = StoredEntryFixed::::from_bytes(raw) + .map_err(|error| { + FinalisedStateError::Custom(format!( + "legacy commitment_tree_data corrupt data: {error}" + )) + })?; + if !entry.verify(&height_bytes) { + return Err(FinalisedStateError::Custom( + "legacy commitment_tree_data checksum mismatch".to_string(), + )); + } + *entry.inner() + }; + commitment_bytes = + StoredEntryVar::new(&height_bytes, commitment_tree_data).to_bytes()?; + ironwood_bytes = None; + } + // Write commitment (+ ironwood) and advance progress atomically. { let mut txn = env.begin_rw_txn()?; - match txn.put( + put_idempotent( + &mut txn, new_ctd_db, &height_bytes, - &new_entry_bytes, - WriteFlags::NO_OVERWRITE, - ) { - Ok(()) => {} - // Idempotent on resume: an existing rebuilt row must match exactly. - Err(lmdb::Error::KeyExist) => { - let existing = txn - .get(new_ctd_db, &height_bytes) - .map_err(FinalisedStateError::LmdbError)?; - if existing != new_entry_bytes.as_slice() { - return Err(FinalisedStateError::Custom(format!( - "conflicting rebuilt commitment_tree_data at height {}", - height.0 - ))); - } - } - Err(error) => return Err(FinalisedStateError::LmdbError(error)), + &commitment_bytes, + || { + format!( + "conflicting rebuilt commitment_tree_data at height {}", + height.0 + ) + }, + )?; + if let Some(bytes) = &ironwood_bytes { + put_idempotent(&mut txn, ironwood_db, &height_bytes, bytes, || { + format!("conflicting rebuilt ironwood at height {}", height.0) + })?; } let progress = StoredEntryFixed::new(MIGRATION_CTD_PROGRESS_KEY, height + 1); diff --git a/packages/zaino-state/src/chain_index/source.rs b/packages/zaino-state/src/chain_index/source.rs index 9b5ea63f0..b1fa80e76 100644 --- a/packages/zaino-state/src/chain_index/source.rs +++ b/packages/zaino-state/src/chain_index/source.rs @@ -1,6 +1,6 @@ //! Traits and types for the blockchain source thats serves zaino, commonly a validator connection. -use std::{error::Error, str::FromStr as _, sync::Arc}; +use std::{error::Error, sync::Arc}; use crate::chain_index::{ types::{BlockHash, TransactionHash}, @@ -10,12 +10,16 @@ use crate::SendFut; use futures::TryFutureExt as _; use incrementalmerkletree::frontier::CommitmentTree; use tower::{Service, ServiceExt as _}; -use zaino_common::Network; use zaino_fetch::jsonrpsee::{ connector::{JsonRpSeeConnector, RpcRequestError}, response::{ address_deltas::{GetAddressDeltasParams, GetAddressDeltasResponse}, - GetBlockError, GetBlockResponse, GetTransactionResponse, GetTreestateResponse, + block_header::GetBlockHeader, + block_subsidy::GetBlockSubsidy, + mining_info::GetMiningInfoWire, + peer_info::GetPeerInfo, + GetBlockError, GetBlockResponse, GetNetworkSolPsResponse, GetSpentInfoRequest, + GetSpentInfoResponse, GetTransactionResponse, GetTreestateResponse, GetTxOutResponse, }, }; use zcash_primitives::merkle_tree::{read_commitment_tree, write_commitment_tree}; @@ -24,7 +28,9 @@ use zebra_chain::{ }; use zebra_rpc::{ client::{GetAddressBalanceRequest, GetAddressTxIdsRequest}, - methods::{AddressBalance, GetAddressUtxos}, + methods::{ + AddressBalance, GetAddressUtxos, GetBlockchainInfoResponse, GetInfo, SentTransactionHash, + }, }; use zebra_state::{HashOrHeight, ReadRequest, ReadResponse, ReadStateService}; @@ -75,6 +81,33 @@ pub trait BlockchainSource: Clone + Send + Sync + 'static { id: HashOrHeight, ) -> impl SendFut>>>; + /// Returns the `getblock`-shaped verbose block for the given hash or height. + /// + /// `verbosity` follows the zcashd `getblock` convention (0 = raw, 1 = object with + /// txids, 2 = object with full transaction data). + fn get_block_verbose( + &self, + hash_or_height: HashOrHeight, + verbosity: Option, + ) -> impl SendFut>; + + /// Returns the `getblockheader`-shaped block header for the given block hash. + /// + /// When `verbose` is false the header is returned in raw hex form; when true it is + /// returned as a structured object. + fn get_block_header( + &self, + hash: String, + verbose: bool, + ) -> impl SendFut>; + + /// Returns the `getblockdeltas`-shaped transparent input/output deltas for the block + /// with the given hash. + fn get_block_deltas( + &self, + hash: String, + ) -> impl SendFut>; + // ********** Transaction methods ********** /// Returns the transaction by txid @@ -107,6 +140,88 @@ pub trait BlockchainSource: Clone + Send + Sync + 'static { &self, ) -> impl SendFut>>; + /// Returns the proof-of-work difficulty of the best chain as a multiple of the + /// minimum difficulty (the `getdifficulty` RPC value). + fn get_difficulty(&self) -> impl SendFut>; + + /// A watch stream of the source's chain-tip changes, when the source owns + /// one locally. Only the `Direct` validator connection (which drives its + /// own Zebra syncer) does; every other source observes tips by polling, + /// and inherits this `None` default. + fn chain_tip_change(&self) -> Option { + None + } + + /// Returns the `getblockchaininfo` response. + fn get_blockchain_info( + &self, + ) -> impl SendFut>; + + // ********** Node-passthrough methods ********** + // + // These have no local-index equivalent and always proxy to the backing validator's + // JSON-RPC interface. + + /// Returns the `getinfo` response. + fn get_info(&self) -> impl SendFut>; + + /// Returns the `getpeerinfo` response. + fn get_peer_info(&self) -> impl SendFut>; + + /// Returns the validator's `getchaintips` response. Serves as the + /// `getchaintips` fallback while the local index is still building its + /// finalised state and has no non-finalised snapshot to answer from. + fn get_chain_tips( + &self, + ) -> impl SendFut< + BlockchainSourceResult, + >; + + /// Returns the `getblocksubsidy` response at the given height. + fn get_block_subsidy( + &self, + height: u32, + ) -> impl SendFut>; + + /// Returns the `getmininginfo` response. + fn get_mining_info(&self) -> impl SendFut>; + + /// Returns the `gettxout` response for the given outpoint. + fn get_tx_out( + &self, + txid: String, + n: u32, + include_mempool: Option, + ) -> impl SendFut>; + + /// Returns the `getspentinfo` response for the given request. + fn get_spent_info( + &self, + request: GetSpentInfoRequest, + ) -> impl SendFut>; + + /// Returns the `getnetworksolps` response. + fn get_network_sol_ps( + &self, + blocks: Option, + height: Option, + ) -> impl SendFut>; + + /// Submits a raw transaction to the network via the validator's mempool + /// (`sendrawtransaction`). + fn send_raw_transaction( + &self, + raw_transaction_hex: String, + ) -> impl SendFut>; + + /// Returns the full `z_gettreestate` response for the given hash-or-height string. + /// + /// Node-passthrough fallback for treestates not locally serviceable by the index. + fn get_treestate_by_id( + &self, + hash_or_height: String, + ) -> impl SendFut>; + /// Returns the sapling and orchard treestate by hash fn get_treestate(&self, id: BlockHash) -> impl SendFut>; @@ -243,6 +358,15 @@ pub trait BlockchainSource: Clone + Send + Sync + 'static { fn subscribe_to_blocks_received(&self) -> Option> { None } + + /// Release any long-lived resources the source owns (e.g. a background + /// syncer task feeding a `ReadStateService`). + /// + /// Default is a no-op — poll-only sources (`JsonRpSeeConnector`) and test + /// mockchains own nothing to tear down. Sources that spawn their own + /// validator plumbing (the `State` arm of `ValidatorConnector`, which owns + /// the Zebra syncer task) override this to abort that task on shutdown. + fn shutdown(&self) {} } /// Sleep up to `duration`, but return early if `change_rx` resolves first. @@ -271,11 +395,54 @@ pub(super) async fn wait_or_source_change( /// An error originating from a blockchain source. #[derive(Debug, thiserror::Error)] pub enum BlockchainSourceError { - /// Unrecoverable error. + /// Unrecoverable error described only by a message (no underlying + /// typed error exists, e.g. an unexpected response shape). // TODO: Add logic for handling recoverable errors if any are identified // one candidate may be ephemerable network hiccoughs #[error("critical error in backing block source: {0}")] Unrecoverable(String), + /// Unrecoverable error whose typed cause is preserved as + /// [`std::error::Error::source`]. zaino-serve recovers zcashd-compatible + /// RPC error codes by downcast-walking `source()` chains, so errors that + /// wrap a typed transport or RPC error must use this variant rather than + /// [`Self::Unrecoverable`] with a stringified cause. + #[error("critical error in backing block source: {message}")] + UnrecoverableWithSource { + /// Rendered description, including the cause's `Display` output so + /// top-level log lines match the previous stringified form. + message: String, + /// The typed cause, available to `source()`-chain walks. + #[source] + source: Box, + }, +} + +impl BlockchainSourceError { + /// Wraps a typed error, preserving it for `source()`-chain recovery. + /// Accepts both concrete error types and already-boxed errors (e.g. + /// zebra's `BoxError`). + pub(crate) fn unrecoverable( + error: impl Into>, + ) -> Self { + let source = error.into(); + Self::UnrecoverableWithSource { + message: source.to_string(), + source, + } + } + + /// Wraps a typed error with a context prefix, preserving the error for + /// `source()`-chain recovery. + pub(crate) fn unrecoverable_context( + context: impl std::fmt::Display, + error: impl Into>, + ) -> Self { + let source = error.into(); + Self::UnrecoverableWithSource { + message: format!("{context}: {source}"), + source, + } + } } /// Error type returned when invalid data is returned by the validator. diff --git a/packages/zaino-state/src/chain_index/source/mockchain_source.rs b/packages/zaino-state/src/chain_index/source/mockchain_source.rs index 00cab444d..a6bc290e8 100644 --- a/packages/zaino-state/src/chain_index/source/mockchain_source.rs +++ b/packages/zaino-state/src/chain_index/source/mockchain_source.rs @@ -1,21 +1,32 @@ //! Mock BlockchainSourceResult implementation. +use super::validator_connector::{ + assemble_block_deltas, build_block_header_object, build_verbose_block, + confirmations_from_depth, final_orchard_root, final_sapling_root, median_of_block_times, + zebra_block_header_to_wire, +}; use super::*; use std::collections::{HashMap, HashSet}; +use std::str::FromStr as _; use std::sync::{ atomic::{AtomicU32, Ordering}, Arc, Mutex, }; use zaino_common::network::ActivationHeights; -use zaino_fetch::jsonrpsee::response::address_deltas::BlockInfo; +use zaino_fetch::jsonrpsee::response::{ + address_deltas::BlockInfo, block_deltas::BlockDeltas, block_header::GetBlockHeader, +}; use zebra_chain::{block::Block, orchard::tree as orchard, sapling::tree as sapling}; use zebra_chain::{ - block::Height, + block::{Height, SerializedBlock}, parameters::NetworkKind, - serialization::BytesInDisplayOrder as _, + serialization::{BytesInDisplayOrder as _, ZcashSerialize as _}, transparent::{Address, OutPoint, Output, OutputIndex}, }; -use zebra_rpc::{client::TransactionObject, methods::ValidateAddresses as _}; +use zebra_rpc::{ + client::{BlockObject, HexData, Input, TransactionObject}, + methods::{GetBlock, GetBlockHeaderResponse, GetBlockTransaction, ValidateAddresses as _}, +}; use zebra_state::HashOrHeight; /// Build the txid → (height, tx) lookup map used by @@ -125,7 +136,7 @@ fn normalize_requested_addresses_for_network( /// transparent address prefixes, so output-derived transparent addresses use /// `NetworkKind::Testnet`. fn mockchain_network() -> zebra_chain::parameters::Network { - zaino_common::Network::Regtest(ActivationHeights::default()).to_zebra_network() + ActivationHeights::default().to_regtest_network() } /// A test-only mock implementation of BlockchainReader using ordered lists by height. @@ -169,6 +180,9 @@ pub(crate) struct MockchainSource { /// neither know nor care how many `mine_blocks` events occurred /// between wakes. blocks_received_broadcaster: tokio::sync::watch::Sender<()>, + /// Records whether [`BlockchainSource::shutdown`] ran, so teardown tests + /// can assert the index releases its source. Shared across clones. + shutdown_called: Arc, } impl MockchainSource { @@ -235,9 +249,16 @@ impl MockchainSource { )), get_block_hook: Arc::new(Mutex::new(None)), blocks_received_broadcaster: tokio::sync::watch::channel(()).0, + shutdown_called: Arc::new(std::sync::atomic::AtomicBool::new(false)), } } + /// Whether [`BlockchainSource::shutdown`] has run on this source (or any + /// clone of it). + pub(crate) fn shutdown_called(&self) -> bool { + self.shutdown_called.load(Ordering::SeqCst) + } + /// When set to true, `get_best_block_height` and `get_best_block_hash` /// return `BlockchainSourceError::Unrecoverable`. pub(crate) fn set_failing(&self, fail: bool) { @@ -332,6 +353,98 @@ impl MockchainSource { self.active_height() as usize } + /// Resolves a hash-or-height request to a block index within the active chain. + fn resolve_index(&self, id: &HashOrHeight) -> Option { + match id { + HashOrHeight::Height(height) => self.valid_height(height.0), + HashOrHeight::Hash(hash) => self.valid_hash(hash), + } + } + + /// The zebra hash of the block after `height_index`, if it is within the active chain. + fn next_block_hash(&self, height_index: usize) -> Option { + let next = height_index + 1; + (next <= self.active_chain_height_as_usize()).then(|| self.blocks[next].hash()) + } + + /// Builds the zebra `getblockheader` response for the block at `height_index` from the + /// test-vector block and tree-root data, using the same builders as the validator path. + fn block_header_response_at( + &self, + height_index: usize, + verbose: bool, + ) -> BlockchainSourceResult { + let block = &self.blocks[height_index]; + let header = &block.header; + if !verbose { + return Ok(GetBlockHeaderResponse::Raw(HexData( + header + .zcash_serialize_to_vec() + .map_err(BlockchainSourceError::unrecoverable)?, + ))); + } + + let network = mockchain_network(); + let hash = block.hash(); + let height = block.coinbase_height().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("block missing coinbase height".to_string()) + })?; + let depth = self.active_height().checked_sub(height.0); + let confirmations = confirmations_from_depth(depth); + + let (sapling_root_bytes, sapling_tree_size) = match &self.roots[height_index].0 { + Some((root, size)) => (<[u8; 32]>::from(*root), *size), + None => ([0u8; 32], 0), + }; + let final_sapling_root = final_sapling_root(sapling_root_bytes, height, &network); + let next_block_hash = self.next_block_hash(height_index); + + let header_obj = build_block_header_object( + header, + hash, + height, + confirmations, + final_sapling_root, + sapling_tree_size, + next_block_hash, + &network, + )?; + Ok(GetBlockHeaderResponse::Object(Box::new(header_obj))) + } + + /// Median time past over the 11-block window ending at `start`, walking backwards via + /// verbosity-1 `getblock` lookups against the mock vectors. + async fn median_time_past(&self, start: &BlockObject) -> BlockchainSourceResult { + const MEDIAN_TIME_PAST_WINDOW: usize = 11; + let mut times = Vec::with_capacity(MEDIAN_TIME_PAST_WINDOW); + let start_time = start.time().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("getblockdeltas: start block missing time".into()) + })?; + times.push(start_time); + + let mut prev = start.previous_block_hash(); + for _ in 0..(MEDIAN_TIME_PAST_WINDOW - 1) { + let Some(hash) = prev else { + break; // genesis + }; + match self + .get_block_verbose(HashOrHeight::Hash(hash), Some(1)) + .await + { + Ok(GetBlock::Object(object)) => { + if let Some(time) = object.time() { + times.push(time); + } + prev = object.previous_block_hash(); + } + Ok(GetBlock::Raw(_)) => break, + Err(_) => break, + } + } + + median_of_block_times(times) + } + fn block_height_at_index(&self, block_index: usize) -> Height { self.blocks[block_index] .coinbase_height() @@ -450,6 +563,263 @@ impl BlockchainSource for MockchainSource { } } + async fn get_block_verbose( + &self, + hash_or_height: HashOrHeight, + verbosity: Option, + ) -> BlockchainSourceResult { + let verbosity = verbosity.unwrap_or(1); + let height_index = self + .resolve_index(&hash_or_height) + .ok_or_else(|| BlockchainSourceError::Unrecoverable("block not found".to_string()))?; + + match verbosity { + 0 => Ok(GetBlock::Raw(SerializedBlock::from(Arc::clone( + &self.blocks[height_index], + )))), + 1 | 2 => { + let block = &self.blocks[height_index]; + let network = mockchain_network(); + + let GetBlockHeaderResponse::Object(header_obj) = + self.block_header_response_at(height_index, true)? + else { + unreachable!("`true` yields an object") + }; + + let height = block.coinbase_height().ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "block missing coinbase height".to_string(), + ) + })?; + let (orchard_root_bytes, orchard_tree_size) = match &self.roots[height_index].1 { + Some((root, size)) => (<[u8; 32]>::from(*root), *size), + None => ([0u8; 32], 0), + }; + let final_orchard_root = final_orchard_root(orchard_root_bytes, height, &network); + + // The verbose block reports the block's serialized byte size; the mock has the + // full block, so serialize it to measure. + let size = block + .zcash_serialize_to_vec() + .map_err(BlockchainSourceError::unrecoverable)? + .len() as i64; + + // `chain_supply` / `value_pools` are cumulative pool balances that the test + // vectors do not carry, so they are `None` for the mock. + Ok(build_verbose_block( + &header_obj, + block, + verbosity, + size, + final_orchard_root, + orchard_tree_size, + // The mock's test vectors do not carry an ironwood tree size. + 0, + None, + None, + &network, + )) + } + more_than_two => Err(BlockchainSourceError::Unrecoverable(format!( + "invalid verbosity of {more_than_two}" + ))), + } + } + + async fn get_block_header( + &self, + hash: String, + verbose: bool, + ) -> BlockchainSourceResult { + let hash_or_height = + HashOrHeight::from_str(&hash).map_err(BlockchainSourceError::unrecoverable)?; + let height_index = self.resolve_index(&hash_or_height).ok_or_else(|| { + BlockchainSourceError::Unrecoverable("block height not in best chain".to_string()) + })?; + let header = self.block_header_response_at(height_index, verbose)?; + zebra_block_header_to_wire(header) + } + + async fn get_block_deltas(&self, hash: String) -> BlockchainSourceResult { + let hash_or_height = + HashOrHeight::from_str(&hash).map_err(BlockchainSourceError::unrecoverable)?; + let GetBlock::Object(object) = self.get_block_verbose(hash_or_height, Some(2)).await? + else { + return Err(BlockchainSourceError::Unrecoverable( + "getblockdeltas: unexpected raw block".to_string(), + )); + }; + + // The mock holds every transaction, so prevout resolution is a direct index lookup. + let mut prevtx_cache: HashMap< + zebra_chain::transaction::Hash, + Arc, + > = HashMap::new(); + for tx in object.tx() { + let GetBlockTransaction::Object(txo) = tx else { + continue; + }; + for input in txo.inputs() { + let Input::NonCoinbase { txid: prevtxid, .. } = input else { + continue; + }; + let prev_hash = zebra_chain::transaction::Hash::from_str(prevtxid) + .map_err(BlockchainSourceError::unrecoverable)?; + if prevtx_cache.contains_key(&prev_hash) { + continue; + } + let (_height, prev_tx) = self.txid_index.get(&prev_hash).ok_or_else(|| { + BlockchainSourceError::Unrecoverable(format!( + "getblockdeltas: prevout tx {prevtxid} not found in mock chain" + )) + })?; + prevtx_cache.insert(prev_hash, Arc::clone(prev_tx)); + } + } + + let median_time = self.median_time_past(&object).await?; + assemble_block_deltas(&object, &prevtx_cache, median_time, &mockchain_network()) + } + + // ********** Chain methods ********** + + async fn get_difficulty(&self) -> BlockchainSourceResult { + let tip_index = self.active_chain_height_as_usize(); + let tip_block = self.blocks.get(tip_index).ok_or_else(|| { + BlockchainSourceError::Unrecoverable("mock chain has no tip block".to_string()) + })?; + Ok(tip_block + .header + .difficulty_threshold + .relative_to_network(&mockchain_network())) + } + + async fn get_blockchain_info( + &self, + ) -> BlockchainSourceResult { + // Needs cumulative pool value balances (TipPoolValues) and on-disk size, which the + // vectors don't carry. Test vectors must be extended to serve this method; tracked + // by the update-test-vectors follow-up (see "Future work"). + unimplemented!( + "MockchainSource cannot serve get_blockchain_info until test vectors are extended" + ) + } + + // ********** Node-passthrough methods ********** + // + // These are node-only RPCs with no chain data in the vectors. Test vectors must be + // extended to let MockchainSource serve them; tracked by the update-test-vectors + // follow-up (see "Future work"). + + async fn get_info(&self) -> BlockchainSourceResult { + unimplemented!("MockchainSource cannot serve get_info until test vectors are extended") + } + + async fn get_peer_info( + &self, + ) -> BlockchainSourceResult { + unimplemented!("MockchainSource cannot serve get_peer_info until test vectors are extended") + } + + /// Records the release so teardown tests can assert the index shuts its + /// source down (the mock owns no real background work). + fn shutdown(&self) { + self.shutdown_called.store(true, Ordering::SeqCst); + } + + /// A single active tip at the mockchain's current active height, matching + /// what a validator with no side branches reports. + async fn get_chain_tips( + &self, + ) -> BlockchainSourceResult + { + if self + .force_requests_against_source_to_fail + .load(Ordering::SeqCst) + { + return Err(BlockchainSourceError::Unrecoverable( + "forced source failure".into(), + )); + } + let height = self.active_height(); + let Some(index) = self.valid_height(height) else { + return Ok(vec![]); + }; + Ok(vec![ + zaino_fetch::jsonrpsee::response::chain_tips::ChainTip::new( + height, + self.blocks[index].hash().to_string(), + 0, + zaino_fetch::jsonrpsee::response::chain_tips::ChainTipStatus::Active, + ), + ]) + } + + async fn get_block_subsidy( + &self, + _height: u32, + ) -> BlockchainSourceResult + { + unimplemented!( + "MockchainSource cannot serve get_block_subsidy until test vectors are extended" + ) + } + + async fn get_mining_info( + &self, + ) -> BlockchainSourceResult + { + unimplemented!( + "MockchainSource cannot serve get_mining_info until test vectors are extended" + ) + } + + async fn get_tx_out( + &self, + _txid: String, + _n: u32, + _include_mempool: Option, + ) -> BlockchainSourceResult { + unimplemented!("MockchainSource cannot serve get_tx_out until test vectors are extended") + } + + async fn get_spent_info( + &self, + _request: zaino_fetch::jsonrpsee::response::GetSpentInfoRequest, + ) -> BlockchainSourceResult { + unimplemented!( + "MockchainSource cannot serve get_spent_info until test vectors are extended" + ) + } + + async fn get_network_sol_ps( + &self, + _blocks: Option, + _height: Option, + ) -> BlockchainSourceResult { + unimplemented!( + "MockchainSource cannot serve get_network_sol_ps until test vectors are extended" + ) + } + + async fn send_raw_transaction( + &self, + _raw_transaction_hex: String, + ) -> BlockchainSourceResult { + // The mock chain has no mempool to accept submissions. + unimplemented!("MockchainSource cannot serve send_raw_transaction") + } + + async fn get_treestate_by_id( + &self, + _hash_or_height: String, + ) -> BlockchainSourceResult { + // The `z_get_treestate` local path serves the mock; the node-passthrough fallback + // is never reached, so this is left unimplemented. + unimplemented!("MockchainSource cannot serve the get_treestate_by_id passthrough") + } + // ********** Transaction methods ********** async fn get_transaction( diff --git a/packages/zaino-state/src/chain_index/source/validator_connector.rs b/packages/zaino-state/src/chain_index/source/validator_connector.rs index cf7908a94..e2130c218 100644 --- a/packages/zaino-state/src/chain_index/source/validator_connector.rs +++ b/packages/zaino-state/src/chain_index/source/validator_connector.rs @@ -1,10 +1,42 @@ //! validator connected blockchain source. +use std::collections::HashMap; +use std::str::FromStr as _; + +use chrono::{DateTime, Utc}; use futures::future::join3; -use hex::FromHex as _; -use zaino_fetch::jsonrpsee::response::address_deltas::BlockInfo; -use zebra_chain::serialization::BytesInDisplayOrder as _; -use zebra_rpc::methods::ValidateAddresses as _; +use hex::{FromHex as _, ToHex as _}; +use indexmap::IndexMap; +use tracing::info; +use zaino_fetch::jsonrpsee::response::{ + address_deltas::BlockInfo, + block_deltas::{BlockDelta, BlockDeltas, InputDelta, OutputDelta}, + block_header::GetBlockHeader, + block_subsidy::GetBlockSubsidy, + mining_info::GetMiningInfoWire, + peer_info::GetPeerInfo, + GetInfoResponse, GetNetworkSolPsResponse, GetSpentInfoRequest, GetSpentInfoResponse, + GetTxOutResponse, +}; +use zebra_rpc::sync::init_read_state_with_syncer; + +use crate::config::{CommonBackendConfig, DirectConnectionConfig}; +use zebra_chain::{ + amount::{Amount, NonNegative}, + block::{Header, SerializedBlock}, + chain_tip::NetworkChainTipHeightEstimator, + parameters::{ConsensusBranchId, NetworkUpgrade}, + serialization::{BytesInDisplayOrder as _, ZcashSerialize as _}, +}; +use zebra_rpc::{ + client::{BlockObject, GetBlockchainInfoBalance, HexData, Input, TransactionObject}, + methods::{ + chain_tip_difficulty, ConsensusBranchIdHex, GetBlock, GetBlockHeaderObject, + GetBlockHeaderResponse, GetBlockTransaction, GetBlockTrees, GetBlockchainInfoResponse, + GetInfo, NetworkUpgradeInfo, NetworkUpgradeStatus, SentTransactionHash, TipConsensusBranch, + ValidateAddresses as _, + }, +}; use crate::Height; @@ -34,8 +66,14 @@ pub struct State { pub read_state_service: ReadStateService, /// Temporarily used to fetch mempool data. pub mempool_fetcher: JsonRpSeeConnector, - /// Current network type being run. - pub network: Network, + /// The runtime network (activation schedule adopted from the validator). + pub network: zebra_chain::parameters::Network, + /// Watches the Zebra syncer's chain-tip changes; served to consumers via + /// [`ValidatorConnector::chain_tip_change`]. + pub chain_tip_change: zebra_state::ChainTipChange, + /// Handle to the Zebra `ReadStateService` sync task, kept alive for the lifetime of + /// the connector and aborted on [`ValidatorConnector::shutdown`]. Shared across clones. + pub sync_task_handle: Option>>, } /// A connection to a validator. @@ -51,6 +89,301 @@ pub enum ValidatorConnector { Fetch(JsonRpSeeConnector), } +impl ValidatorConnector { + /// The JSON-RPC connector for this validator, used by the node-passthrough RPCs that + /// have no local-index equivalent. The `State` variant proxies these through its + /// `mempool_fetcher` until the `ReadStateService` can serve them. + fn json_rpc_connector(&self) -> &JsonRpSeeConnector { + match self { + ValidatorConnector::State(state) => &state.mempool_fetcher, + ValidatorConnector::Fetch(fetch) => fetch, + } + } + + /// Spawns a JSON-RPC-backed [`ValidatorConnector::Fetch`] from the common backend + /// config, returning the connector, the validator's `getinfo` response (used by + /// the backend to build its `ServiceMetadata`), and the runtime network whose + /// activation schedule was adopted from the validator (zaino#1076). + /// + /// Owns the `JsonRpSeeConnector` setup that previously lived in `FetchService::spawn`. + pub(crate) async fn spawn_fetch( + common: &CommonBackendConfig, + ) -> Result<(Self, GetInfoResponse, zebra_chain::parameters::Network), BlockchainSourceError> + { + let fetcher = JsonRpSeeConnector::new_from_config_parts( + &common.validator_rpc_address, + common.validator_rpc_user.clone(), + common.validator_rpc_password.clone(), + common.validator_cookie_path.clone(), + ) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + + let info = fetcher + .get_info() + .await + .map_err(BlockchainSourceError::unrecoverable)?; + + // A freshly started validator answers RPC before it has fetched and + // committed its first block: zebra serves getblockchaininfo and an + // empty getrawmempool while getbestblockhash returns "No blocks in + // state" until genesis arrives, which can take minutes of peer + // discovery. Everything downstream assumes a servable tip + // (Mempool::spawn returns Critical on get_best_block_hash and the + // ChainIndex sync loop escalates to CriticalError), so wait here + // instead of failing spawn and exit-looping the whole process. + while let Err(e) = fetcher.get_best_blockhash().await { + info!(%e, "Waiting for validator to serve its first block"); + tokio::time::sleep(std::time::Duration::from_secs(3)).await; + } + + let network = adopt_network(common, &fetcher).await?; + + Ok((ValidatorConnector::Fetch(fetcher), info, network)) + } + + /// Spawns a `ReadStateService`-backed [`ValidatorConnector::State`] from the common + /// backend config plus the [`DirectConnectionConfig`], returning the connector plus + /// the validator's `getinfo` response. + /// + /// Owns the JSON-RPC + Zebra chain-syncer setup for the `Direct` connection: builds + /// the mempool JSON-RPC fetcher, launches the syncer and `ReadStateService`, then + /// blocks until the syncer has caught up to the validator's best chain tip (comparing + /// tip *hash* as well as height so a reorg mid-sync cannot report a false match). The + /// syncer task handle is retained on the `State` so [`ValidatorConnector::shutdown`] + /// can abort it. + pub(crate) async fn spawn_state( + common: &CommonBackendConfig, + direct: &DirectConnectionConfig, + ) -> Result<(Self, GetInfoResponse, zebra_chain::parameters::Network), BlockchainSourceError> + { + let map_err = + |error: &dyn std::fmt::Display| BlockchainSourceError::Unrecoverable(error.to_string()); + + let rpc_client = JsonRpSeeConnector::new_from_config_parts( + &common.validator_rpc_address, + common.validator_rpc_user.clone(), + common.validator_rpc_password.clone(), + common.validator_cookie_path.clone(), + ) + .await + .map_err(|error| map_err(&error))?; + + let info = rpc_client + .get_info() + .await + .map_err(|error| map_err(&error))?; + + let network = adopt_network(common, &rpc_client).await?; + + info!( + grpc_address = %direct.validator_grpc_address, + "Launching Chain Syncer" + ); + let (mut read_state_service, _latest_chain_tip, chain_tip_change, sync_task_handle) = + init_read_state_with_syncer( + direct.validator_state_config.clone(), + &network, + direct.validator_grpc_address, + ) + .await + .map_err(|error| map_err(&error))? + .map_err(|error| map_err(&error))?; + + info!("Chain syncer launched"); + + // Wait for ReadStateService to catch up to the validator's best chain tip. + // Height alone is insufficient during reorgs: the same height can refer to + // different blocks until JSON-RPC and ReadStateService agree on tip hash. + loop { + let blockchain_info = rpc_client + .get_blockchain_info() + .await + .map_err(|error| map_err(&error))?; + let server_height = blockchain_info.blocks; + let server_tip_hash = blockchain_info.best_block_hash; + + let syncer_response = read_state_service + .ready() + .and_then(|service| service.call(ReadRequest::Tip)) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + // A freshly started validator answers RPC before it has committed + // its first block (getblockchaininfo reports a genesis-hash + // placeholder on an empty state), and the syncer has no tip until + // genesis arrives, which can take minutes of peer discovery. + // Everything below assumes a servable tip, so wait here instead + // of failing spawn and exit-looping the whole process. + let Some((syncer_height, syncer_tip_hash)) = + expected_read_response!(syncer_response, Tip) + else { + info!("Waiting for validator to serve its first block"); + tokio::time::sleep(std::time::Duration::from_secs(3)).await; + continue; + }; + + if server_height == syncer_height && server_tip_hash == syncer_tip_hash { + info!( + height = syncer_height.0, + tip_hash = %syncer_tip_hash, + "ReadStateService synced with Zebra" + ); + break; + } else { + info!( + syncer_height = syncer_height.0, + validator_height = server_height.0, + syncer_tip_hash = %syncer_tip_hash, + validator_tip_hash = %server_tip_hash, + "ReadStateService syncing with Zebra" + ); + tokio::time::sleep(std::time::Duration::from_millis(1000)).await; + continue; + } + } + + let source = ValidatorConnector::State(State { + read_state_service, + mempool_fetcher: rpc_client, + network: network.clone(), + chain_tip_change, + sync_task_handle: Some(Arc::new(sync_task_handle)), + }); + + Ok((source, info, network)) + } + + /// The backing [`ReadStateService`], when this connector is `State`-backed. + /// + /// Test-only escape hatch: live tests recompute expected chain data directly off + /// the `ReadStateService`. Production code goes through the `ChainIndex` API. + #[cfg(feature = "test_dependencies")] + pub(crate) fn read_state_service(&self) -> Option<&ReadStateService> { + match self { + ValidatorConnector::State(state) => Some(&state.read_state_service), + ValidatorConnector::Fetch(_) => None, + } + } +} + +/// Constructs the runtime network at first contact with the validator. +/// +/// The validator is the single source of truth for activation heights +/// (zaino#1076): the config carries only a network kind, and the runtime +/// network is constructed here before anything consumes one. On regtest the +/// schedule is adopted wholesale from the validator's report. On Mainnet and +/// The Public Testnet the compiled zebra parameters are used, but the +/// validator's reported schedule is verified against them first, so a +/// validator reporting custom activation heights under the `PubTestnet` +/// kind (a regtest net in configured-testnet clothing) fails loud here +/// instead of silently drifting from the index. There is no fallback: a +/// silently wrong schedule is the failure mode this removes. +/// +/// Adoption happens exactly once, at spawn. A validator restarted +/// mid-session with a different schedule is out of scope (the test harness +/// restarts validator and indexer together); it would invalidate the index +/// without being re-detected here. +async fn adopt_network( + common: &CommonBackendConfig, + rpc_client: &JsonRpSeeConnector, +) -> Result { + let blockchain_info = rpc_client.get_blockchain_info().await.map_err(|error| { + BlockchainSourceError::Unrecoverable(format!( + "cannot fetch activation heights from the validator at {}: {error}", + common.validator_rpc_address + )) + })?; + + // Shared by the Mainnet / The Public Testnet arms: the compiled network is only + // trusted after the validator's report agrees with it. + let verified = |network: zebra_chain::parameters::Network| { + verify_reported_upgrades(&network, &blockchain_info.upgrades).map_err(|reason| { + BlockchainSourceError::Unrecoverable(format!( + "the validator at {} disagrees with the compiled {network} parameters: {reason}", + common.validator_rpc_address + )) + })?; + Ok(network) + }; + + match common.network { + zaino_common::Network::Mainnet => verified(zebra_chain::parameters::Network::Mainnet), + zaino_common::Network::PubTestnet => { + verified(zebra_chain::parameters::Network::new_default_testnet()) + } + zaino_common::Network::Regtest => { + let heights = + activation_heights_from_upgrades(&blockchain_info.upgrades).map_err(|reason| { + BlockchainSourceError::Unrecoverable(format!( + "cannot adopt activation heights from the validator at {}: {reason}", + common.validator_rpc_address + )) + })?; + info!(?heights, "Adopted activation heights from the validator"); + Ok(heights.to_regtest_network()) + } + } +} + +/// Checks every `(upgrade, height)` the validator reports against `network`'s +/// compiled activation schedule, rejecting the first disagreement. +/// +/// Only reported entries are checked — the report's *completeness* is not: +/// the map is keyed by consensus branch ID, so upgrades zebra considers +/// unconfigured are simply absent, and an older validator may not know the +/// newest upgrades yet. +fn verify_reported_upgrades( + network: &zebra_chain::parameters::Network, + upgrades: &IndexMap, +) -> Result<(), String> { + for upgrade_info in upgrades.values() { + let (upgrade, height, _status) = upgrade_info.into_parts(); + let compiled = upgrade.activation_height(network); + if compiled != Some(height) { + return Err(format!( + "the validator reports {upgrade:?} activating at height {}, \ + but the compiled parameters say {:?}", + height.0, + compiled.map(|compiled_height| compiled_height.0) + )); + } + } + Ok(()) +} + +/// Builds the regtest activation heights from the validator's reported +/// upgrade schedule (`getblockchaininfo.upgrades`). +/// +/// The validator's configured activation heights are authoritative: the +/// config type is a payload-free kind, so both connector arms construct the +/// runtime network here at first contact, before anything consumes a +/// `Network` (zaino#1076). An upgrade absent from the validator's map is +/// never-activated — nothing is backfilled from defaults. Mainnet and +/// The Public Testnet use zebra's compiled parameters and never take this +/// path. +/// +/// `before_overwinter` is always `None` here: the map is keyed by consensus +/// branch ID, which `BeforeOverwinter` does not have, so it can never appear +/// in the report. Correctness relies on zebra's `new_regtest` supplying its +/// own `BeforeOverwinter` handling when `to_regtest_network` builds the +/// runtime network from these heights. +fn activation_heights_from_upgrades( + upgrades: &IndexMap, +) -> Result { + let mut heights = zaino_common::config::network::ActivationHeights::NEVER_ACTIVATED; + for upgrade_info in upgrades.values() { + let (upgrade, height, _status) = upgrade_info.into_parts(); + // Genesis is height 0 by definition; it has no configuration slot. + let Some(slot) = heights.slot_mut(upgrade) else { + continue; + }; + if slot.replace(height.0).is_some() { + return Err(format!("validator reported {upgrade:?} twice")); + } + } + Ok(heights) +} + /// Serialized empty Orchard-shaped commitment tree in the RPC encoding — what a pool /// reports when it has no treestate to serve, and the encoding both backend arms must /// agree on. @@ -91,6 +424,16 @@ fn fetch_pool_treestate_slot(treestate: zebra_rpc::client::Treestate) -> Option< } impl BlockchainSource for ValidatorConnector { + /// `Some` for the `State` variant, which drives its own Zebra syncer; the + /// JSON-RPC `Fetch` variant has no local tip-change stream and keeps the + /// trait's `None` default semantics. + fn chain_tip_change(&self) -> Option { + match self { + ValidatorConnector::State(state) => Some(state.chain_tip_change.clone()), + ValidatorConnector::Fetch(_) => None, + } + } + // ********** Block methods ********** async fn get_block( @@ -112,7 +455,7 @@ impl BlockchainSource for ValidatorConnector { { Ok(GetBlockResponse::Raw(raw_block)) => Ok(Some(Arc::new( zebra_chain::block::Block::zcash_deserialize(raw_block.as_ref()) - .map_err(|e| BlockchainSourceError::Unrecoverable(e.to_string()))?, + .map_err(BlockchainSourceError::unrecoverable)?, ))), Ok(_) => unreachable!(), Err(e) => match e { @@ -120,7 +463,7 @@ impl BlockchainSource for ValidatorConnector { // TODO/FIX: zcashd returns this transport error when a block is requested higher than current chain. is this correct? RpcRequestError::Transport(zaino_fetch::jsonrpsee::error::TransportError::ErrorStatusCode(500)) => Ok(None), RpcRequestError::ServerWorkQueueFull => Err(BlockchainSourceError::Unrecoverable("Work queue full. not yet implemented: handling of ephemeral network errors.".to_string())), - _ => Err(BlockchainSourceError::Unrecoverable(e.to_string())), + _ => Err(BlockchainSourceError::unrecoverable(e)), }, } } @@ -128,7 +471,7 @@ impl BlockchainSource for ValidatorConnector { "Read Request of Block returned Read Response of {otherwise:#?} \n\ This should be deterministically unreachable" ), - Err(e) => Err(BlockchainSourceError::Unrecoverable(e.to_string())), + Err(e) => Err(BlockchainSourceError::unrecoverable(e)), }, ValidatorConnector::Fetch(fetch) => { match fetch @@ -137,7 +480,7 @@ impl BlockchainSource for ValidatorConnector { { Ok(GetBlockResponse::Raw(raw_block)) => Ok(Some(Arc::new( zebra_chain::block::Block::zcash_deserialize(raw_block.as_ref()) - .map_err(|e| BlockchainSourceError::Unrecoverable(e.to_string()))?, + .map_err(BlockchainSourceError::unrecoverable)?, ))), Ok(_) => unreachable!(), Err(e) => match e { @@ -145,13 +488,201 @@ impl BlockchainSource for ValidatorConnector { // TODO/FIX: zcashd returns this transport error when a block is requested higher than current chain. is this correct? RpcRequestError::Transport(zaino_fetch::jsonrpsee::error::TransportError::ErrorStatusCode(500)) => Ok(None), RpcRequestError::ServerWorkQueueFull => Err(BlockchainSourceError::Unrecoverable("Work queue full. not yet implemented: handling of ephemeral network errors.".to_string())), - _ => Err(BlockchainSourceError::Unrecoverable(e.to_string())), + _ => Err(BlockchainSourceError::unrecoverable(e)), }, } } } } + async fn get_block_verbose( + &self, + hash_or_height: HashOrHeight, + verbosity: Option, + ) -> BlockchainSourceResult { + match self { + ValidatorConnector::State(state) => { + state.get_block_inner(hash_or_height, verbosity).await + } + ValidatorConnector::Fetch(fetch) => fetch + .get_block(hash_or_height.to_string(), verbosity) + .await + .map_err(BlockchainSourceError::unrecoverable)? + .try_into() + .map_err(|error: zebra_chain::serialization::SerializationError| { + BlockchainSourceError::unrecoverable(error) + }), + } + } + + async fn get_block_header( + &self, + hash: String, + verbose: bool, + ) -> BlockchainSourceResult { + match self { + ValidatorConnector::State(state) => { + let hash_or_height = + HashOrHeight::from_str(&hash).map_err(BlockchainSourceError::unrecoverable)?; + let header = state + .get_block_header_inner(hash_or_height, Some(verbose)) + .await?; + zebra_block_header_to_wire(header) + } + ValidatorConnector::Fetch(fetch) => fetch + .get_block_header(hash, verbose) + .await + .map_err(BlockchainSourceError::unrecoverable), + } + } + + async fn get_block_deltas(&self, hash: String) -> BlockchainSourceResult { + match self { + ValidatorConnector::State(state) => state.get_block_deltas(hash).await, + ValidatorConnector::Fetch(fetch) => fetch + .get_block_deltas(hash) + .await + .map_err(BlockchainSourceError::unrecoverable), + } + } + + // ********** Chain methods ********** + + async fn get_difficulty(&self) -> BlockchainSourceResult { + match self { + ValidatorConnector::State(state) => chain_tip_difficulty( + state.network.clone(), + state.read_state_service.clone(), + false, + ) + .await + .map_err(BlockchainSourceError::unrecoverable), + ValidatorConnector::Fetch(fetch) => Ok(fetch + .get_difficulty() + .await + .map_err(BlockchainSourceError::unrecoverable)? + .0), + } + } + + async fn get_blockchain_info(&self) -> BlockchainSourceResult { + match self { + ValidatorConnector::State(state) => state.get_blockchain_info().await, + ValidatorConnector::Fetch(fetch) => fetch + .get_blockchain_info() + .await + .map_err(BlockchainSourceError::unrecoverable)? + .try_into() + .map_err(|_error| { + BlockchainSourceError::Unrecoverable( + "getblockchaininfo: chainwork not hex-encoded integer".to_string(), + ) + }), + } + } + + // ********** Node-passthrough methods ********** + + async fn get_info(&self) -> BlockchainSourceResult { + Ok(self + .json_rpc_connector() + .get_info() + .await + .map_err(BlockchainSourceError::unrecoverable)? + .into()) + } + + async fn get_peer_info(&self) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_peer_info() + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_chain_tips( + &self, + ) -> BlockchainSourceResult + { + self.json_rpc_connector() + .get_chain_tips() + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_block_subsidy(&self, height: u32) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_block_subsidy(height) + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_mining_info(&self) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_mining_info() + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_tx_out( + &self, + txid: String, + n: u32, + include_mempool: Option, + ) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_tx_out(txid, n, include_mempool) + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_spent_info( + &self, + request: GetSpentInfoRequest, + ) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_spent_info(request) + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_network_sol_ps( + &self, + blocks: Option, + height: Option, + ) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_network_sol_ps(blocks, height) + .await + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn send_raw_transaction( + &self, + raw_transaction_hex: String, + ) -> BlockchainSourceResult { + // ReadStateService does not yet interface with the mempool, so both variants + // submit via the JSON-RPC connector. + self.json_rpc_connector() + .send_raw_transaction(raw_transaction_hex) + .await + .map(SentTransactionHash::from) + .map_err(BlockchainSourceError::unrecoverable) + } + + async fn get_treestate_by_id( + &self, + hash_or_height: String, + ) -> BlockchainSourceResult { + self.json_rpc_connector() + .get_treestate(hash_or_height) + .await + .map_err(BlockchainSourceError::unrecoverable)? + .try_into() + .map_err(|_error| { + BlockchainSourceError::Unrecoverable("failed to parse treestate".to_string()) + }) + } + // ********** Transaction methods ********** // Returns the transaction, and the height of the block that transaction is in if on the best chain @@ -168,7 +699,7 @@ impl BlockchainSource for ValidatorConnector { ValidatorConnector::State(State { read_state_service, mempool_fetcher, - network: _, + .. }) => { // Check state for transaction let mut read_state_service = read_state_service.clone(); @@ -184,7 +715,7 @@ impl BlockchainSource for ValidatorConnector { }) .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!("state read failed: {e}")) + BlockchainSourceError::unrecoverable_context("state read failed", e) })?; if let zebra_state::ReadResponse::AnyChainTransaction(opt) = response { @@ -221,9 +752,10 @@ impl BlockchainSource for ValidatorConnector { .get_raw_transaction(zebra_txid.to_string(), Some(0)) .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch transaction data: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch transaction data", + e, + ) })? { serialized_transaction } else { @@ -236,9 +768,10 @@ impl BlockchainSource for ValidatorConnector { std::io::Cursor::new(serialized_transaction.as_ref()), ) .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not deserialize transaction data: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not deserialize transaction data", + e, + ) })?; Ok(Some((transaction.into(), GetTransactionLocation::Mempool))) } else { @@ -251,9 +784,10 @@ impl BlockchainSource for ValidatorConnector { .get_raw_transaction(txid.to_rpc_hex(), Some(1)) .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch transaction data: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch transaction data", + e, + ) })? { transaction_object } else { @@ -266,9 +800,10 @@ impl BlockchainSource for ValidatorConnector { transaction_object.hex().as_ref(), )) .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not deserialize transaction data: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not deserialize transaction data", + e, + ) })?; let location = match transaction_object.height() { Some(-1) => GetTransactionLocation::NonbestChain, @@ -298,7 +833,7 @@ impl BlockchainSource for ValidatorConnector { .get_raw_mempool() .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!("could not fetch mempool data: {e}")) + BlockchainSourceError::unrecoverable_context("could not fetch mempool data", e) })? .transactions; @@ -306,9 +841,10 @@ impl BlockchainSource for ValidatorConnector { .into_iter() .map(|txid_str| { zebra_chain::transaction::Hash::from_str(&txid_str).map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "invalid transaction id '{txid_str}': {e}" - )) + BlockchainSourceError::unrecoverable_context( + format!("invalid transaction id '{txid_str}'"), + e, + ) }) }) .collect::>()?; @@ -325,7 +861,7 @@ impl BlockchainSource for ValidatorConnector { ValidatorConnector::State(State { read_state_service, mempool_fetcher, - network: _, + .. }) => { match read_state_service.best_tip() { Some((_height, hash)) => Ok(Some(hash)), @@ -336,9 +872,10 @@ impl BlockchainSource for ValidatorConnector { .get_best_blockhash() .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch best block hash from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch best block hash from validator", + e, + ) })? .0, )) @@ -350,9 +887,10 @@ impl BlockchainSource for ValidatorConnector { .get_best_blockhash() .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch best block hash from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch best block hash from validator", + e, + ) })? .0, )), @@ -367,7 +905,7 @@ impl BlockchainSource for ValidatorConnector { ValidatorConnector::State(State { read_state_service, mempool_fetcher, - network: _, + .. }) => { match read_state_service.best_tip() { Some((height, _hash)) => Ok(Some(height)), @@ -378,9 +916,10 @@ impl BlockchainSource for ValidatorConnector { .get_block_count() .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch best block hash from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch best block hash from validator", + e, + ) })? .into(), )) @@ -392,9 +931,10 @@ impl BlockchainSource for ValidatorConnector { .get_block_count() .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not fetch best block hash from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not fetch best block hash from validator", + e, + ) })? .into(), )), @@ -436,7 +976,7 @@ impl BlockchainSource for ValidatorConnector { let sapling = match ShieldedPool::Sapling .activation_upgrade() - .activation_height(&state.network.to_zebra_network()) + .activation_height(&state.network) { Some(activation_height) if height >= activation_height => Some( state @@ -467,7 +1007,7 @@ impl BlockchainSource for ValidatorConnector { let orchard = match ShieldedPool::Orchard .activation_upgrade() - .activation_height(&state.network.to_zebra_network()) + .activation_height(&state.network) { Some(activation_height) if height >= activation_height => Some( state @@ -498,7 +1038,7 @@ impl BlockchainSource for ValidatorConnector { let ironwood = match ShieldedPool::Ironwood .activation_upgrade() - .activation_height(&state.network.to_zebra_network()) + .activation_height(&state.network) { Some(activation_height) if height >= activation_height => Some( state @@ -613,9 +1153,10 @@ impl BlockchainSource for ValidatorConnector { } }) .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not get subtrees from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not get subtrees from validator", + e, + ) }) } @@ -624,9 +1165,10 @@ impl BlockchainSource for ValidatorConnector { .get_subtrees_by_index(pool.pool_string(), start_index, max_entries) .await .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not get subtrees from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not get subtrees from validator", + e, + ) })?; Ok(subtrees @@ -647,9 +1189,10 @@ impl BlockchainSource for ValidatorConnector { }) .collect::, _>>() .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!( - "could not get subtrees from validator: {e}" - )) + BlockchainSourceError::unrecoverable_context( + "could not get subtrees from validator", + e, + ) })?) } } @@ -681,12 +1224,9 @@ impl BlockchainSource for ValidatorConnector { .await; let (sapling_tree, orchard_tree, ironwood_tree) = match ( //TODO: Better readstateservice error handling - sapling_tree_response - .map_err(|e| BlockchainSourceError::Unrecoverable(e.to_string()))?, - orchard_tree_response - .map_err(|e| BlockchainSourceError::Unrecoverable(e.to_string()))?, - ironwood_tree_response - .map_err(|e| BlockchainSourceError::Unrecoverable(e.to_string()))?, + sapling_tree_response.map_err(BlockchainSourceError::unrecoverable)?, + orchard_tree_response.map_err(BlockchainSourceError::unrecoverable)?, + ironwood_tree_response.map_err(BlockchainSourceError::unrecoverable)?, ) { ( ReadResponse::SaplingTree(saptree), @@ -722,7 +1262,7 @@ impl BlockchainSource for ValidatorConnector { .to_string(), ) } - _ => BlockchainSourceError::Unrecoverable(e.to_string()), + _ => BlockchainSourceError::unrecoverable(e), })?; let GetTreestateResponse { sapling, @@ -742,7 +1282,7 @@ impl BlockchainSource for ValidatorConnector { }, ) .transpose() - .map_err(|e| BlockchainSourceError::Unrecoverable(format!("io error: {e}")))?; + .map_err(|e| BlockchainSourceError::unrecoverable_context("io error", e))?; let orchard_frontier = orchard .map_or_else( || Some(Ok(CommitmentTree::empty())), @@ -755,7 +1295,7 @@ impl BlockchainSource for ValidatorConnector { }, ) .transpose() - .map_err(|e| BlockchainSourceError::Unrecoverable(format!("io error: {e}")))?; + .map_err(|e| BlockchainSourceError::unrecoverable_context("io error", e))?; let ironwood_frontier = ironwood .map_or_else( || Some(Ok(CommitmentTree::empty())), @@ -768,7 +1308,7 @@ impl BlockchainSource for ValidatorConnector { }, ) .transpose() - .map_err(|e| BlockchainSourceError::Unrecoverable(format!("io error: {e}")))?; + .map_err(|e| BlockchainSourceError::unrecoverable_context("io error", e))?; let sapling_root = sapling_frontier .map(|tree| { zebra_chain::sapling::tree::Root::try_from(tree.root().to_bytes()) @@ -776,7 +1316,7 @@ impl BlockchainSource for ValidatorConnector { }) .transpose() .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!("could not deser: {e}")) + BlockchainSourceError::unrecoverable_context("could not deser", e) })?; let orchard_root = orchard_frontier .map(|tree| { @@ -785,7 +1325,7 @@ impl BlockchainSource for ValidatorConnector { }) .transpose() .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!("could not deser: {e}")) + BlockchainSourceError::unrecoverable_context("could not deser", e) })?; let ironwood_root = ironwood_frontier .map(|tree| { @@ -794,7 +1334,7 @@ impl BlockchainSource for ValidatorConnector { }) .transpose() .map_err(|e| { - BlockchainSourceError::Unrecoverable(format!("could not deser: {e}")) + BlockchainSourceError::unrecoverable_context("could not deser", e) })?; Ok((sapling_root, orchard_root, ironwood_root)) } @@ -821,15 +1361,11 @@ impl BlockchainSource for ValidatorConnector { GetAddressDeltasParams::Address(a) => (vec![a.clone()], 0, 0, false), }; - let tip = self.get_best_block_height().await?.unwrap().into(); - let mut start = Height(start_raw); - let mut end = Height(end_raw); - if end == Height(0) || end > tip { - end = tip; - } - if start > tip { - start = tip; - } + let (start, end) = clamp_deltas_range_to_tip( + self.get_best_block_height().await?, + start_raw, + end_raw, + )?; let transactions: Vec> = { let tx_ids_request = @@ -860,7 +1396,7 @@ impl BlockchainSource for ValidatorConnector { transaction.clone(), height, None, - &state.network.to_zebra_network(), + &state.network, None, None, Some(matches!( @@ -893,14 +1429,10 @@ impl BlockchainSource for ValidatorConnector { let response = read_state .ready() .await - .map_err(|error| { - BlockchainSourceError::Unrecoverable(error.to_string()) - })? + .map_err(BlockchainSourceError::unrecoverable)? .call(ReadRequest::BlockHeader(hash_or_height)) .await - .map_err(|error| { - BlockchainSourceError::Unrecoverable(error.to_string()) - })?; + .map_err(BlockchainSourceError::unrecoverable)?; match response { ReadResponse::BlockHeader { hash, .. } => Ok(BlockInfo::new( @@ -921,14 +1453,10 @@ impl BlockchainSource for ValidatorConnector { let response = read_state .ready() .await - .map_err(|error| { - BlockchainSourceError::Unrecoverable(error.to_string()) - })? + .map_err(BlockchainSourceError::unrecoverable)? .call(ReadRequest::BlockHeader(hash_or_height)) .await - .map_err(|error| { - BlockchainSourceError::Unrecoverable(error.to_string()) - })?; + .map_err(BlockchainSourceError::unrecoverable)?; match response { ReadResponse::BlockHeader { hash, .. } => Ok(BlockInfo::new( @@ -955,7 +1483,7 @@ impl BlockchainSource for ValidatorConnector { ValidatorConnector::Fetch(fetch) => fetch .get_address_deltas(params) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string())), + .map_err(BlockchainSourceError::unrecoverable), } } @@ -968,14 +1496,14 @@ impl BlockchainSource for ValidatorConnector { let mut state = state.read_state_service.clone(); let strings_set = address_strings.valid_addresses().map_err(|error| { - BlockchainSourceError::Unrecoverable(format!("invalid address: {error}")) + BlockchainSourceError::unrecoverable_context("invalid address", error) })?; let response = state .ready() .and_then(|service| service.call(ReadRequest::AddressBalance(strings_set))) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))?; + .map_err(BlockchainSourceError::unrecoverable)?; let (balance, received) = match response { ReadResponse::AddressBalance { balance, received } => (balance, received), @@ -1000,7 +1528,7 @@ impl BlockchainSource for ValidatorConnector { .collect(), ) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))? + .map_err(BlockchainSourceError::unrecoverable)? .into()), } } @@ -1018,7 +1546,7 @@ impl BlockchainSource for ValidatorConnector { .ready() .and_then(|service| service.call(ReadRequest::Tip)) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))?; + .map_err(BlockchainSourceError::unrecoverable)?; let (chain_height, _chain_hash) = expected_read_response!(response, Tip) .ok_or_else(|| { @@ -1045,9 +1573,7 @@ impl BlockchainSource for ValidatorConnector { addresses: GetAddressBalanceRequest::new(addresses) .valid_addresses() .map_err(|error| { - BlockchainSourceError::Unrecoverable(format!( - "invalid address: {error}" - )) + BlockchainSourceError::unrecoverable_context("invalid address", error) })?, height_range: zebra_chain::block::Height(start) @@ -1057,7 +1583,7 @@ impl BlockchainSource for ValidatorConnector { .ready() .and_then(|service| service.call(request)) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))?; + .map_err(BlockchainSourceError::unrecoverable)?; let hashes = expected_read_response!(response, AddressesTransactionIds); @@ -1086,14 +1612,15 @@ impl BlockchainSource for ValidatorConnector { fetch .get_address_txids(addresses, start, end) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))? + .map_err(BlockchainSourceError::unrecoverable)? .transactions .iter() .map(|txid_string| { TransactionHash::from_hex(txid_string.as_bytes()).map_err(|error| { - BlockchainSourceError::Unrecoverable(format!( - "invalid txid from getaddresstxids `{txid_string}`: {error}" - )) + BlockchainSourceError::unrecoverable_context( + format!("invalid txid from getaddresstxids `{txid_string}`"), + error, + ) }) }) .collect::, BlockchainSourceError>>() @@ -1110,7 +1637,7 @@ impl BlockchainSource for ValidatorConnector { let mut state = state.read_state_service.clone(); let valid_addresses = addresses.valid_addresses().map_err(|error| { - BlockchainSourceError::Unrecoverable(format!("invalid address: {error}")) + BlockchainSourceError::unrecoverable_context("invalid address", error) })?; let request = ReadRequest::UtxosByAddresses(valid_addresses); @@ -1118,7 +1645,7 @@ impl BlockchainSource for ValidatorConnector { .ready() .and_then(|service| service.call(request)) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))?; + .map_err(BlockchainSourceError::unrecoverable)?; let utxos = expected_read_response!(response, AddressUtxos); let mut last_output_location = @@ -1161,7 +1688,7 @@ impl BlockchainSource for ValidatorConnector { .collect(), ) .await - .map_err(|error| BlockchainSourceError::Unrecoverable(error.to_string()))? + .map_err(BlockchainSourceError::unrecoverable)? .into_iter() .map(|utxos| utxos.into()) .collect()), @@ -1180,9 +1707,7 @@ impl BlockchainSource for ValidatorConnector { > { match self { ValidatorConnector::State(State { - read_state_service, - mempool_fetcher: _, - network: _, + read_state_service, .. }) => { match read_state_service .clone() @@ -1204,6 +1729,860 @@ impl BlockchainSource for ValidatorConnector { ValidatorConnector::Fetch(_fetch) => Ok(None), } } + + fn shutdown(&self) { + // Only the `State` arm owns a long-lived resource: the Zebra chain-syncer + // task feeding the `ReadStateService`. Abort it so the backend drops cleanly. + // The `Fetch` arm holds only a stateless JSON-RPC client — nothing to tear down. + if let ValidatorConnector::State(State { + sync_task_handle: Some(handle), + .. + }) = self + { + handle.abort(); + } + } +} + +impl State { + /// Builds the `getblockheader`-shaped block header from the `ReadStateService`. + /// + /// Returns the zebra `GetBlockHeaderResponse` shape. This is the shared State-path + /// builder used by both [`ValidatorConnector::get_block_header`] (which converts it + /// to the JSON-RPC wire form) and [`State::get_block_inner`] (which reuses the + /// object fields to assemble a verbose block). + /// + /// Moved from the former `StateServiceSubscriber::get_block_header_inner`; the error + /// type is now [`BlockchainSourceError`] rather than the backend error, so the + /// zcashd-compatible `LegacyCode` on the not-in-best-chain case survives only in the + /// message string. + pub(super) async fn get_block_header_inner( + &self, + hash_or_height: HashOrHeight, + verbose: Option, + ) -> BlockchainSourceResult { + let mut state = self.read_state_service.clone(); + let verbose = verbose.unwrap_or(true); + let network = self.network.clone(); + + let ReadResponse::BlockHeader { + header, + hash, + height, + next_block_hash, + } = state + .ready() + .and_then(|service| service.call(ReadRequest::BlockHeader(hash_or_height))) + .await + .map_err(|_| { + BlockchainSourceError::Unrecoverable("block height not in best chain".to_string()) + })? + else { + return Err(BlockchainSourceError::Unrecoverable( + "Unexpected response to BlockHeader request".to_string(), + )); + }; + + let response = if !verbose { + GetBlockHeaderResponse::Raw(HexData( + header + .zcash_serialize_to_vec() + .map_err(BlockchainSourceError::unrecoverable)?, + )) + } else { + let ReadResponse::SaplingTree(sapling_tree) = state + .ready() + .and_then(|service| service.call(ReadRequest::SaplingTree(hash_or_height))) + .await + .map_err(BlockchainSourceError::unrecoverable)? + else { + return Err(BlockchainSourceError::Unrecoverable( + "Unexpected response to SaplingTree request".to_string(), + )); + }; + // This could be `None` if there's a chain reorg between state queries. + let sapling_tree = sapling_tree.ok_or_else(|| { + BlockchainSourceError::Unrecoverable("missing sapling tree for block".to_string()) + })?; + + let ReadResponse::Depth(depth) = state + .ready() + .and_then(|service| service.call(ReadRequest::Depth(hash))) + .await + .map_err(BlockchainSourceError::unrecoverable)? + else { + return Err(BlockchainSourceError::Unrecoverable( + "Unexpected response to Depth request".to_string(), + )); + }; + + // Confirmations are one more than the depth. Depth is limited by height, so + // it will never overflow an i64. + let confirmations = confirmations_from_depth(depth); + + let sapling_tree_size = sapling_tree.count(); + let final_sapling_root = + final_sapling_root(sapling_tree.root().into(), height, &network); + + let block_header = build_block_header_object( + &header, + hash, + height, + confirmations, + final_sapling_root, + sapling_tree_size, + next_block_hash, + &network, + )?; + + GetBlockHeaderResponse::Object(Box::new(block_header)) + }; + + Ok(response) + } + + /// Builds the `getblock`-shaped verbose block from the `ReadStateService`. + /// + /// Moved from the former `StateServiceSubscriber::get_block_inner`; the error type is + /// now [`BlockchainSourceError`]. + pub(super) async fn get_block_inner( + &self, + hash_or_height: HashOrHeight, + verbosity: Option, + ) -> BlockchainSourceResult { + let mut state_1 = self.read_state_service.clone(); + let verbosity = verbosity.unwrap_or(1); + match verbosity { + 0 => { + let request = ReadRequest::Block(hash_or_height); + let response = state_1 + .ready() + .and_then(|service| service.call(request)) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + let block = expected_read_response!(response, Block); + block + .map(SerializedBlock::from) + .map(GetBlock::Raw) + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable("block not found".to_string()) + }) + } + 1 | 2 => { + let network = self.network.clone(); + let state_2 = self.read_state_service.clone(); + let state_4 = self.read_state_service.clone(); + let state_5 = self.read_state_service.clone(); + + let blockandsize_future = { + let req = ReadRequest::BlockAndSize(hash_or_height); + async move { state_1.ready().and_then(|service| service.call(req)).await } + }; + let orchard_future = { + let req = ReadRequest::OrchardTree(hash_or_height); + async move { + state_2 + .clone() + .ready() + .and_then(|service| service.call(req)) + .await + } + }; + let ironwood_future = { + let req = ReadRequest::IronwoodTree(hash_or_height); + async move { + state_5 + .clone() + .ready() + .and_then(|service| service.call(req)) + .await + } + }; + let block_info_future = { + let req = ReadRequest::BlockInfo(hash_or_height); + async move { + state_4 + .clone() + .ready() + .and_then(|service| service.call(req)) + .await + } + }; + let (fullblock, orchard_tree_response, ironwood_tree_response, header, block_info) = futures::join!( + blockandsize_future, + orchard_future, + ironwood_future, + self.get_block_header_inner(hash_or_height, Some(true)), + block_info_future + ); + + let header_obj = match header? { + GetBlockHeaderResponse::Raw(_hex_data) => unreachable!( + "`true` was passed to get_block_header, an object should be returned" + ), + GetBlockHeaderResponse::Object(get_block_header_object) => { + get_block_header_object + } + }; + + let (block, size, block_info) = match (fullblock, block_info) { + ( + Ok(ReadResponse::BlockAndSize(Some((block, size)))), + Ok(ReadResponse::BlockInfo(Some(block_info))), + ) => (block, size, block_info), + (Ok(ReadResponse::Block(None)), Ok(ReadResponse::BlockInfo(None))) => { + return Err(BlockchainSourceError::Unrecoverable( + "block not found".to_string(), + )); + } + (Ok(unexpected), Ok(unexpected2)) => { + unreachable!("Unexpected responses from state service: {unexpected:?} {unexpected2:?}") + } + (Err(e), _) | (_, Err(e)) => { + return Err(BlockchainSourceError::unrecoverable(e)); + } + }; + + let orchard_tree_response = + orchard_tree_response.map_err(BlockchainSourceError::unrecoverable)?; + let orchard_tree = expected_read_response!(orchard_tree_response, OrchardTree) + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable("missing orchard tree".to_string()) + })?; + + let ironwood_tree_response = + ironwood_tree_response.map_err(BlockchainSourceError::unrecoverable)?; + let ironwood_tree = expected_read_response!(ironwood_tree_response, IronwoodTree) + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable("missing ironwood tree".to_string()) + })?; + + let final_orchard_root = + final_orchard_root(orchard_tree.root().into(), header_obj.height(), &network); + + let (chain_supply, value_pools) = ( + Some(GetBlockchainInfoBalance::chain_supply( + *block_info.value_pools(), + )), + Some(GetBlockchainInfoBalance::value_pools( + *block_info.value_pools(), + None, + )), + ); + + Ok(build_verbose_block( + &header_obj, + &block, + verbosity, + size as i64, + final_orchard_root, + orchard_tree.count(), + ironwood_tree.count(), + chain_supply, + value_pools, + &network, + )) + } + more_than_two => Err(BlockchainSourceError::Unrecoverable(format!( + "invalid verbosity of {more_than_two}" + ))), + } + } + + /// Builds the `getblockdeltas` response from the `ReadStateService`. + /// + /// Moved from the former `StateServiceSubscriber::get_block_deltas`; resolves each + /// spent prevout via a best-chain `ReadRequest::Transaction` (finalised + + /// non-finalised), then hands off to the shared [`assemble_block_deltas`]. + async fn get_block_deltas(&self, hash: String) -> BlockchainSourceResult { + let hash_or_height = + HashOrHeight::from_str(&hash).map_err(BlockchainSourceError::unrecoverable)?; + let GetBlock::Object(object) = self.get_block_inner(hash_or_height, Some(2)).await? else { + return Err(BlockchainSourceError::Unrecoverable( + "getblockdeltas: unexpected raw block".to_string(), + )); + }; + + // Per-call cache: many inputs may reference the same prevtxid, so each previous + // transaction is fetched at most once. + let mut prevtx_cache: HashMap< + zebra_chain::transaction::Hash, + Arc, + > = HashMap::new(); + for tx in object.tx() { + let GetBlockTransaction::Object(txo) = tx else { + continue; + }; + for input in txo.inputs() { + let Input::NonCoinbase { txid: prevtxid, .. } = input else { + continue; + }; + let prev_hash = zebra_chain::transaction::Hash::from_str(prevtxid) + .map_err(BlockchainSourceError::unrecoverable)?; + if prevtx_cache.contains_key(&prev_hash) { + continue; + } + let mut state = self.read_state_service.clone(); + let response = state + .ready() + .and_then(|service| service.call(ReadRequest::Transaction(prev_hash))) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + let mined_tx = expected_read_response!(response, Transaction).ok_or_else(|| { + BlockchainSourceError::Unrecoverable(format!( + "getblockdeltas: prevout tx {prevtxid} not in best chain" + )) + })?; + prevtx_cache.insert(prev_hash, mined_tx.tx); + } + } + + let median_time = self.median_time_past(&object).await?; + let network = self.network.clone(); + assemble_block_deltas(&object, &prevtx_cache, median_time, &network) + } + + /// Median time past over the 11-block window ending at `start`, walking backwards via + /// verbosity-1 `getblock` lookups against the `ReadStateService`. + // TODO(DRY): MockchainSource duplicates this walk; the only difference is the + // per-block fetch. A shared helper generic over an async block fetcher would unify them. + async fn median_time_past(&self, start: &BlockObject) -> BlockchainSourceResult { + const MEDIAN_TIME_PAST_WINDOW: usize = 11; + let mut times = Vec::with_capacity(MEDIAN_TIME_PAST_WINDOW); + let start_time = start.time().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("getblockdeltas: start block missing time".into()) + })?; + times.push(start_time); + + let mut prev = start.previous_block_hash(); + for _ in 0..(MEDIAN_TIME_PAST_WINDOW - 1) { + let Some(hash) = prev else { + break; // genesis + }; + match self + .get_block_inner(HashOrHeight::Hash(hash), Some(1)) + .await + { + Ok(GetBlock::Object(object)) => { + if let Some(time) = object.time() { + times.push(time); + } + prev = object.previous_block_hash(); + } + Ok(GetBlock::Raw(_)) => break, + Err(_) => break, // use values collected so far + } + } + + median_of_block_times(times) + } + + /// Builds the `getblockchaininfo` response from the `ReadStateService`. + /// + /// Moved from the former `StateServiceSubscriber::get_blockchain_info`; error type is + /// now [`BlockchainSourceError`], and the difficulty is propagated rather than + /// unwrapped. + async fn get_blockchain_info(&self) -> BlockchainSourceResult { + let mut state = self.read_state_service.clone(); + let network = self.network.clone(); + + let response = state + .ready() + .and_then(|service| service.call(ReadRequest::TipPoolValues)) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + let (height, hash, balance) = match response { + ReadResponse::TipPoolValues { + tip_height, + tip_hash, + value_balance, + } => (tip_height, tip_hash, value_balance), + unexpected => { + unreachable!("Unexpected response from state service: {unexpected:?}") + } + }; + + let usage_response = state + .ready() + .and_then(|service| service.call(ReadRequest::UsageInfo)) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + let size_on_disk = expected_read_response!(usage_response, UsageInfo); + + let response = state + .ready() + .and_then(|service| service.call(ReadRequest::BlockHeader(hash.into()))) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + let header = match response { + ReadResponse::BlockHeader { header, .. } => header, + unexpected => { + unreachable!("Unexpected response from state service: {unexpected:?}") + } + }; + + let now = Utc::now(); + let zebra_estimated_height = + NetworkChainTipHeightEstimator::new(header.time, height, &network) + .estimate_height_at(now); + let estimated_height = if header.time > now || zebra_estimated_height < height { + height + } else { + zebra_estimated_height + }; + + let upgrades = IndexMap::from_iter(network.full_activation_list().into_iter().filter_map( + |(activation_height, network_upgrade)| { + // Zebra defines network upgrades by consensus rule changes, zcashd by ZIPs; + // upgrades with a consensus branch ID are the same in both. + network_upgrade.branch_id().map(|branch_id| { + // zcashd's RPC ignores Disabled network upgrades, so Zebra does too. + let status = if height >= activation_height { + NetworkUpgradeStatus::Active + } else { + NetworkUpgradeStatus::Pending + }; + ( + ConsensusBranchIdHex::new(branch_id.into()), + NetworkUpgradeInfo::from_parts(network_upgrade, activation_height, status), + ) + }) + }, + )); + + let next_block_height = + (height + 1).expect("valid chain tips are a lot less than Height::MAX"); + let consensus = TipConsensusBranch::from_parts( + ConsensusBranchIdHex::new( + NetworkUpgrade::current(&network, height) + .branch_id() + .unwrap_or(ConsensusBranchId::RPC_MISSING_ID) + .into(), + ) + .inner(), + ConsensusBranchIdHex::new( + NetworkUpgrade::current(&network, next_block_height) + .branch_id() + .unwrap_or(ConsensusBranchId::RPC_MISSING_ID) + .into(), + ) + .inner(), + ); + + let difficulty = + chain_tip_difficulty(network.clone(), self.read_state_service.clone(), false) + .await + .map_err(BlockchainSourceError::unrecoverable)?; + + let verification_progress = f64::from(height.0) / f64::from(zebra_estimated_height.0); + + Ok(GetBlockchainInfoResponse::new( + network.bip70_network_name(), + height, + hash, + estimated_height, + GetBlockchainInfoBalance::chain_supply(balance), + // TODO: account for new delta_pools arg? + GetBlockchainInfoBalance::value_pools(balance, None), + upgrades, + consensus, + height, + difficulty, + verification_progress, + // TODO: store work in the finalized state for each height + // (see https://github.com/ZcashFoundation/zebra/issues/7109) + 0, + false, + size_on_disk, + // TODO (copied from zebra): investigate whether this needs implementing + // (it's sprout-only in zcashd) + 0, + )) + } +} + +/// Confirmations are one more than the depth, or -1 when the block is not on the best +/// chain. Depth is limited by height, so it never overflows an `i64`. +pub(crate) fn confirmations_from_depth(depth: Option) -> i64 { + const NOT_IN_BEST_CHAIN_CONFIRMATIONS: i64 = -1; + depth + .map(|depth| i64::from(depth) + 1) + .unwrap_or(NOT_IN_BEST_CHAIN_CONFIRMATIONS) +} + +/// The `finalsaplingroot` field: the sapling note-commitment tree root in display +/// (big-endian) byte order once Sapling has activated, else all-zero. +pub(crate) fn final_sapling_root( + root: [u8; 32], + height: zebra_chain::block::Height, + network: &zebra_chain::parameters::Network, +) -> [u8; 32] { + match NetworkUpgrade::Sapling.activation_height(network) { + Some(activation) if height >= activation => { + let mut root = root; + root.reverse(); + root + } + _ => [0; 32], + } +} + +/// The `finalorchardroot` field: `Some(root)` once NU5 has activated, else `None`. +pub(crate) fn final_orchard_root( + root: [u8; 32], + height: zebra_chain::block::Height, + network: &zebra_chain::parameters::Network, +) -> Option<[u8; 32]> { + match NetworkUpgrade::Nu5.activation_height(network) { + Some(activation) if height >= activation => Some(root), + _ => None, + } +} + +/// Assembles a `getblockheader` object from its already-resolved primitive pieces. +/// +/// Shared by the [`ValidatorConnector`] State path (which reads the pieces from the +/// `ReadStateService`) and by `MockchainSource` (which reads them from test vectors), +/// so the two never drift. +#[allow(clippy::too_many_arguments)] +pub(crate) fn build_block_header_object( + header: &Header, + hash: zebra_chain::block::Hash, + height: zebra_chain::block::Height, + confirmations: i64, + final_sapling_root: [u8; 32], + sapling_tree_size: u64, + next_block_hash: Option, + network: &zebra_chain::parameters::Network, +) -> BlockchainSourceResult { + let mut nonce = *header.nonce; + nonce.reverse(); + let difficulty = header.difficulty_threshold.relative_to_network(network); + let block_commitments = + header_to_block_commitments(header, network, height, final_sapling_root)?; + + Ok(GetBlockHeaderObject::new( + hash, + confirmations, + height, + header.version, + header.merkle_root, + block_commitments, + final_sapling_root, + sapling_tree_size, + header.time.timestamp(), + nonce, + header.solution, + header.difficulty_threshold, + difficulty, + header.previous_block_hash, + next_block_hash, + )) +} + +/// Assembles a verbose (`verbosity` 1 or 2) `getblock` object from a resolved header +/// object plus the block's transactions and pool aggregates. +/// +/// Shared by the [`ValidatorConnector`] State path and `MockchainSource`. `chain_supply` +/// / `value_pools` are the cumulative pool balances (present for the validator, `None` +/// for the mock, whose vectors don't carry them). +#[allow(clippy::too_many_arguments)] +pub(crate) fn build_verbose_block( + header_obj: &GetBlockHeaderObject, + block: &zebra_chain::block::Block, + verbosity: u8, + size: i64, + final_orchard_root: Option<[u8; 32]>, + orchard_tree_size: u64, + ironwood_tree_size: u64, + chain_supply: Option, + value_pools: Option<[GetBlockchainInfoBalance; 6]>, + network: &zebra_chain::parameters::Network, +) -> GetBlock { + let transactions_response: Vec = block + .transactions + .iter() + .map(|transaction| match verbosity { + 1 => GetBlockTransaction::Hash(transaction.hash()), + 2 => GetBlockTransaction::Object(Box::new(TransactionObject::from_transaction( + transaction.clone(), + Some(header_obj.height()), + Some(header_obj.confirmations()), + network, + DateTime::::from_timestamp(header_obj.time(), 0), + Some(header_obj.hash()), + // A non-optional header height indicates a mainchain block; sidechain + // data is out of scope for now. TODO: return Some(true/false) once resolved. + None, + transaction.hash(), + ))), + _ => unreachable!("verbosity known to be 1 or 2"), + }) + .collect(); + + let trees = GetBlockTrees::new( + header_obj.sapling_tree_size(), + orchard_tree_size, + ironwood_tree_size, + ); + let transaction_count = transactions_response.len(); + + GetBlock::Object(Box::new(BlockObject::new( + header_obj.hash(), + header_obj.confirmations(), + Some(size), + Some(header_obj.height()), + Some(header_obj.version()), + Some(header_obj.merkle_root()), + Some(header_obj.block_commitments()), + Some(header_obj.final_sapling_root()), + final_orchard_root, + transaction_count, + transactions_response, + Some(header_obj.time()), + Some(header_obj.nonce()), + Some(header_obj.solution()), + Some(header_obj.bits()), + Some(header_obj.difficulty()), + chain_supply, + value_pools, + trees, + Some(header_obj.previous_block_hash()), + header_obj.next_block_hash(), + ))) +} + +/// Computes the `blockcommitments` field for a block header. +/// +/// Moved verbatim (bar the error type) from the former `state.rs` +/// `header_to_block_commitments`. +pub(crate) fn header_to_block_commitments( + header: &Header, + network: &zebra_chain::parameters::Network, + height: zebra_chain::block::Height, + final_sapling_root: [u8; 32], +) -> BlockchainSourceResult<[u8; 32]> { + let hash = match header.commitment(network, height).map_err(|error| { + BlockchainSourceError::unrecoverable_context("invalid block commitment", error) + })? { + zebra_chain::block::Commitment::PreSaplingReserved(bytes) => bytes, + zebra_chain::block::Commitment::FinalSaplingRoot(_root) => final_sapling_root, + zebra_chain::block::Commitment::ChainHistoryActivationReserved => [0; 32], + zebra_chain::block::Commitment::ChainHistoryRoot(root) => root.bytes_in_display_order(), + zebra_chain::block::Commitment::ChainHistoryBlockTxAuthCommitment(hash) => { + hash.bytes_in_display_order() + } + }; + Ok(hash) +} + +/// Converts the zebra `GetBlockHeaderResponse` (Raw/Object) into the JSON-RPC wire +/// `getblockheader` response (Compact/Verbose) expected by the backends. +/// +/// The two shapes serialize to the same JSON, so a serde round-trip is the least-error- +/// prone conversion. +pub(crate) fn zebra_block_header_to_wire( + header: GetBlockHeaderResponse, +) -> BlockchainSourceResult { + let value = serde_json::to_value(&header).map_err(BlockchainSourceError::unrecoverable)?; + serde_json::from_value(value).map_err(BlockchainSourceError::unrecoverable) +} + +/// Returns the median of a non-empty set of block times. +pub(crate) fn median_of_block_times(mut times: Vec) -> BlockchainSourceResult { + if times.is_empty() { + return Err(BlockchainSourceError::Unrecoverable( + "getblockdeltas: no block times collected for median".to_string(), + )); + } + times.sort_unstable(); + Ok(times[times.len() / 2]) +} + +/// Maps a verbose-transaction output to its `getblockdeltas` output delta, +/// or `None` when the output does not participate in address deltas: only +/// outputs with exactly one derivable address are attributed. Zero means no +/// address to credit (nonstandard script); more than one (e.g. bare multisig) +/// has no single owning address, and zcashd's getblockdeltas omits such +/// outputs rather than crediting the first address. +fn output_delta_from_verbose(vout: &zebra_rpc::client::Output) -> Option { + let address = + vout.script_pub_key() + .addresses() + .as_ref() + .and_then(|addresses| match addresses.as_slice() { + [address] => Some(address.clone()), + _ => None, + })?; + let satoshis: Amount = vout.value_zat().try_into().ok()?; + Some(OutputDelta { + address, + satoshis, + index: vout.n(), + }) +} + +/// Assembles a `getblockdeltas` response from a verbosity-2 `getblock` object plus a +/// cache of previous transactions (keyed by txid) needed to resolve each spend's address +/// and value, its median time, and the running network. +/// +/// Shared by the [`ValidatorConnector`] State path and `MockchainSource`: both resolve +/// the prevout transactions their own way, then hand the assembled inputs here so the +/// delta-shaping logic lives in one place. +pub(crate) fn assemble_block_deltas( + object: &BlockObject, + prevtx_cache: &HashMap< + zebra_chain::transaction::Hash, + Arc, + >, + median_time: i64, + network: &zebra_chain::parameters::Network, +) -> BlockchainSourceResult { + let mut deltas = Vec::with_capacity(object.tx().len()); + for (tx_index, tx) in object.tx().iter().enumerate() { + let GetBlockTransaction::Object(txo) = tx else { + return Err(BlockchainSourceError::Unrecoverable( + "getblockdeltas: unexpected hash when expecting object".to_string(), + )); + }; + let txid = txo.txid().to_string(); + + let mut inputs: Vec = Vec::new(); + for (i, vin) in txo.inputs().iter().enumerate() { + let (prevtxid, prevout) = match vin { + Input::Coinbase { .. } => continue, + Input::NonCoinbase { + txid: prevtxid, + vout: prevout, + .. + } => (prevtxid, *prevout), + }; + + let prev_hash = zebra_chain::transaction::Hash::from_str(prevtxid) + .map_err(BlockchainSourceError::unrecoverable)?; + let prev_tx = prevtx_cache.get(&prev_hash).ok_or_else(|| { + BlockchainSourceError::Unrecoverable(format!( + "getblockdeltas: prevout tx {prevtxid} not resolved" + )) + })?; + + let output = prev_tx.outputs().get(prevout as usize).ok_or_else(|| { + BlockchainSourceError::Unrecoverable(format!( + "getblockdeltas: prevout index {prevout} out of range for {prevtxid}" + )) + })?; + + // Nonstandard script ⇒ no derivable address ⇒ skip (matches the outputs branch). + let address = match output.address(network) { + Some(address) => address.to_string(), + None => continue, + }; + + // Inputs are debits, so the amount leaves the address. + let satoshis: Amount = (-output.value().zatoshis()).try_into().map_err(|error| { + BlockchainSourceError::unrecoverable_context( + format!("getblockdeltas: input amount out of range for {prevtxid}:{prevout}"), + error, + ) + })?; + + inputs.push(InputDelta { + address, + satoshis, + index: i as u32, + prevtxid: prevtxid.clone(), + prevout, + }); + } + + let outputs: Vec = txo + .outputs() + .iter() + .filter_map(output_delta_from_verbose) + .collect(); + + deltas.push(BlockDelta { + txid, + index: tx_index as u32, + inputs, + outputs, + }); + } + + Ok(BlockDeltas { + hash: object.hash().to_string(), + confirmations: object.confirmations(), + size: object.size().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("getblockdeltas: block size missing".to_string()) + })?, + height: object + .height() + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getblockdeltas: block height missing".to_string(), + ) + })? + .0, + version: object.version().ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getblockdeltas: block version missing".to_string(), + ) + })?, + merkle_root: object + .merkle_root() + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getblockdeltas: block merkle root missing".to_string(), + ) + })? + .encode_hex::(), + deltas, + time: object.time().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("getblockdeltas: block time missing".to_string()) + })?, + median_time, + nonce: hex::encode(object.nonce().ok_or_else(|| { + BlockchainSourceError::Unrecoverable("getblockdeltas: block nonce missing".to_string()) + })?), + bits: object + .bits() + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getblockdeltas: block bits missing".to_string(), + ) + })? + .to_string(), + difficulty: object.difficulty().ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getblockdeltas: block difficulty missing".to_string(), + ) + })?, + previous_block_hash: object.previous_block_hash().map(|hash| hash.to_string()), + next_block_hash: object.next_block_hash().map(|hash| hash.to_string()), + }) +} + +#[cfg(test)] +mod zebra_block_header_to_wire { + use super::*; + + /// The Direct connection's non-verbose `getblockheader` builds a + /// `GetBlockHeaderResponse::Raw` and crosses to the wire type through the + /// serde round-trip under test: the raw hex must land in the untagged + /// `Compact` variant with the same bytes, or every non-verbose + /// `getblockheader` on a Direct connection errors at runtime. The + /// mockchain `get_block_header` test covers the same conversion end to + /// end; this pins it at its definition. + #[test] + fn raw_header_round_trips_to_compact_hex() { + let header_bytes = vec![0x01, 0x02, 0xab, 0xcd]; + let raw = GetBlockHeaderResponse::Raw(HexData(header_bytes.clone())); + + let wire = zebra_block_header_to_wire(raw) + .expect("the raw header shape must convert to the wire type"); + + assert_eq!(wire, GetBlockHeader::Compact(hex::encode(header_bytes))); + } } #[cfg(test)] @@ -1242,6 +2621,172 @@ mod fetch_pool_treestate_slot { } } +/// Clamps a `getaddressdeltas` height range to the current best tip: +/// `end == 0` means "to the tip", and both bounds clamp down to it. A source +/// that reports no best height (nothing indexed yet) is a typed error. +fn clamp_deltas_range_to_tip( + tip: Option, + start_raw: u32, + end_raw: u32, +) -> BlockchainSourceResult<(Height, Height)> { + let tip: Height = tip + .ok_or_else(|| { + BlockchainSourceError::Unrecoverable( + "getaddressdeltas: the source reports no best block height".to_string(), + ) + })? + .into(); + let mut start = Height(start_raw); + let mut end = Height(end_raw); + if end == Height(0) || end > tip { + end = tip; + } + if start > tip { + start = tip; + } + Ok((start, end)) +} + +#[cfg(test)] +mod output_delta_from_verbose { + use super::*; + + fn output_with_addresses(addresses: Option>) -> zebra_rpc::client::Output { + zebra_rpc::client::Output::new( + 0.00000001, + 1, + 0, + zebra_rpc::client::ScriptPubKey::new( + "asm".to_string(), + zebra_chain::transparent::Script::new(&[0u8]), + addresses.as_ref().map(|a| a.len() as u32), + "pubkeyhash".to_string(), + addresses, + ), + ) + } + + /// An output with exactly one derivable address is attributed to it. + #[test] + fn single_address_output_is_attributed() { + let delta = + output_delta_from_verbose(&output_with_addresses(Some(vec!["t1address".to_string()]))) + .expect("a single-address output participates in deltas"); + assert_eq!(delta.address, "t1address"); + } + + /// An output with multiple derivable addresses (e.g. bare multisig) has no + /// single owning address: zcashd's getblockdeltas omits it, and crediting + /// the first address would fabricate balance for it. The pre-fix code took + /// `addresses.first()`. + #[test] + fn multi_address_output_is_omitted() { + assert_eq!( + output_delta_from_verbose(&output_with_addresses(Some(vec![ + "t1first".to_string(), + "t1second".to_string(), + ]))), + None, + "a multi-address output must be omitted, not attributed to the first address" + ); + } + + /// No derivable address (nonstandard script) means no delta. + #[test] + fn addressless_output_is_omitted() { + assert_eq!( + output_delta_from_verbose(&output_with_addresses(None)), + None + ); + } +} + +#[cfg(test)] +mod clamp_deltas_range_to_tip { + use super::*; + + /// A source that reports no best height (nothing indexed yet, or the + /// RPC fallback found no tip) must yield a typed error, not a panic: + /// this range clamp previously lived inline in `get_address_deltas` + /// behind `get_best_block_height().await?.unwrap()`. + #[test] + fn absent_tip_is_a_typed_error() { + assert!( + clamp_deltas_range_to_tip(None, 0, 0).is_err(), + "an absent tip must surface as a BlockchainSourceError" + ); + } + + /// `end == 0` means "to the tip", and both bounds clamp down to the tip. + #[test] + fn bounds_clamp_to_tip() { + let tip = Some(zebra_chain::block::Height(100)); + + let (start, end) = + clamp_deltas_range_to_tip(tip, 5, 0).expect("a present tip clamps successfully"); + assert_eq!((start, end), (Height(5), Height(100))); + + let (start, end) = + clamp_deltas_range_to_tip(tip, 5, 400).expect("a present tip clamps successfully"); + assert_eq!((start, end), (Height(5), Height(100))); + + let (start, end) = + clamp_deltas_range_to_tip(tip, 300, 50).expect("a present tip clamps successfully"); + assert_eq!((start, end), (Height(100), Height(50))); + } +} + +#[cfg(test)] +mod get_block_verbose { + use super::*; + + /// zaino-serve recovers zcashd-compatible RPC error codes by downcast-walking + /// [`std::error::Error::source`] chains (see + /// `getblock_error_object_from_indexer_error` and + /// `sendrawtransaction_error_object_from_indexer_error` in + /// `zaino-serve/src/rpc/jsonrpc/service.rs`). The connector boundary must + /// therefore preserve the typed cause instead of flattening it to a string, + /// or `getblock` failures surface as generic internal errors rather than the + /// legacy `-8` code lightwalletd-family clients key on. + #[tokio::test] + async fn fetch_error_keeps_transport_error_in_source_chain() { + // Port 1 refuses connections, so the request fails at the transport + // layer without contacting any validator. + let connector = zaino_fetch::jsonrpsee::connector::JsonRpSeeConnector::new_with_basic_auth( + "http://127.0.0.1:1/" + .parse() + .expect("static url literal parses"), + "user".to_string(), + "password".to_string(), + ) + .expect("connector construction is network-free"); + let source = ValidatorConnector::Fetch(connector); + + let error = source + .get_block_verbose(HashOrHeight::Height(zebra_chain::block::Height(1)), Some(1)) + .await + .expect_err("a request to a closed port must fail"); + + let mut current: Option<&(dyn std::error::Error + 'static)> = Some(&error); + let mut transport_error_reachable = false; + while let Some(source_error) = current { + if source_error + .downcast_ref::() + .is_some() + { + transport_error_reachable = true; + break; + } + current = source_error.source(); + } + assert!( + transport_error_reachable, + "the typed TransportError must stay reachable via the source() chain; \ + stringifying it strips the RPC error code the serve layer recovers" + ); + } +} + #[cfg(test)] mod ironwood_treestate_slot { /// Regression test: when the validator reported no ironwood treestate (below NU6.3 @@ -1272,3 +2817,171 @@ mod ironwood_treestate_slot { ); } } + +#[cfg(test)] +mod activation_heights_from_upgrades { + use zaino_common::config::network::ActivationHeights; + + /// All-`None` heights: the starting point adoption fills from the + /// validator's map, and the expected value for every absent upgrade. + const NEVER_ACTIVATED: ActivationHeights = ActivationHeights::NEVER_ACTIVATED; + + fn upgrades_map( + json: &str, + ) -> indexmap::IndexMap< + zebra_rpc::methods::ConsensusBranchIdHex, + zebra_rpc::methods::NetworkUpgradeInfo, + > { + serde_json::from_str(json).expect("upgrades fixture parses") + } + + fn adopted_heights( + upgrades: &indexmap::IndexMap< + zebra_rpc::methods::ConsensusBranchIdHex, + zebra_rpc::methods::NetworkUpgradeInfo, + >, + ) -> ActivationHeights { + super::activation_heights_from_upgrades(upgrades).expect("valid schedule") + } + + /// An upgrade absent from the validator's map is never-activated — + /// nothing is backfilled from any default schedule. + #[test] + fn regtest_network_from_upgrades_leaves_absent_upgrades_never_activated() { + let upgrades = upgrades_map( + r#"{ + "c2d6d0b4": { "name": "NU5", "activationheight": 2, "status": "active" }, + "c8e71055": { "name": "NU6", "activationheight": 2, "status": "active" } + }"#, + ); + + assert_eq!( + adopted_heights(&upgrades), + ActivationHeights { + nu5: Some(2), + nu6: Some(2), + ..NEVER_ACTIVATED + } + ); + } + + /// The ORCHARD_THEN_IRONWOOD transition shape: everything through NU6.2 + /// at 1–2, NU6.3 at 6 — the schedule the ironwood_activation fixtures + /// launch validators with. + #[test] + fn regtest_network_from_upgrades_reads_a_transition_schedule() { + let upgrades = upgrades_map( + r#"{ + "5ba81b19": { "name": "Overwinter", "activationheight": 1, "status": "active" }, + "76b809bb": { "name": "Sapling", "activationheight": 1, "status": "active" }, + "2bb40e60": { "name": "Blossom", "activationheight": 1, "status": "active" }, + "f5b9230b": { "name": "Heartwood", "activationheight": 1, "status": "active" }, + "e9ff75a6": { "name": "Canopy", "activationheight": 1, "status": "active" }, + "c2d6d0b4": { "name": "NU5", "activationheight": 2, "status": "active" }, + "c8e71055": { "name": "NU6", "activationheight": 2, "status": "active" }, + "4dec4df0": { "name": "NU6.1", "activationheight": 2, "status": "active" }, + "5437f330": { "name": "NU6.2", "activationheight": 2, "status": "active" }, + "37a5165b": { "name": "NU6.3", "activationheight": 6, "status": "pending" } + }"#, + ); + + assert_eq!( + adopted_heights(&upgrades), + ActivationHeights { + overwinter: Some(1), + sapling: Some(1), + blossom: Some(1), + heartwood: Some(1), + canopy: Some(1), + nu5: Some(2), + nu6: Some(2), + nu6_1: Some(2), + nu6_2: Some(2), + nu6_3: Some(6), + ..NEVER_ACTIVATED + } + ); + } + + /// A validator reporting the same upgrade twice is nonsense; adoption + /// must fail loudly rather than pick a height. + #[test] + fn regtest_network_from_upgrades_rejects_a_duplicate_upgrade() { + let upgrades = upgrades_map( + r#"{ + "c2d6d0b4": { "name": "NU5", "activationheight": 2, "status": "active" }, + "c8e71055": { "name": "NU5", "activationheight": 3, "status": "pending" } + }"#, + ); + + let reason = + super::activation_heights_from_upgrades(&upgrades).expect_err("duplicate must fail"); + assert!( + reason.contains("twice"), + "error should name the duplication, got: {reason}" + ); + } +} + +#[cfg(test)] +mod verify_reported_upgrades { + fn upgrades_map( + json: &str, + ) -> indexmap::IndexMap< + zebra_rpc::methods::ConsensusBranchIdHex, + zebra_rpc::methods::NetworkUpgradeInfo, + > { + serde_json::from_str(json).expect("upgrades fixture parses") + } + + /// A report agreeing with the compiled schedule passes. + #[test] + fn accepts_a_report_matching_the_compiled_schedule() { + let upgrades = upgrades_map( + r#"{ + "76b809bb": { "name": "Sapling", "activationheight": 419200, "status": "active" } + }"#, + ); + + super::verify_reported_upgrades(&zebra_chain::parameters::Network::Mainnet, &upgrades) + .expect("mainnet Sapling at 419200 matches the compiled parameters"); + } + + /// A reported height disagreeing with the compiled schedule fails loud — + /// the wrong-schedule / wrong-network drift class. + #[test] + fn rejects_a_mismatched_height() { + let upgrades = upgrades_map( + r#"{ + "76b809bb": { "name": "Sapling", "activationheight": 419201, "status": "active" } + }"#, + ); + + let reason = + super::verify_reported_upgrades(&zebra_chain::parameters::Network::Mainnet, &upgrades) + .expect_err("a wrong Sapling height must be rejected"); + assert!( + reason.contains("419201"), + "error should name the reported height, got: {reason}" + ); + } + + /// An upgrade the compiled schedule never activates must also be + /// rejected when the validator reports a height for it. + #[test] + fn rejects_an_upgrade_the_compiled_schedule_lacks() { + let upgrades = upgrades_map( + r#"{ + "77190ad8": { "name": "NU7", "activationheight": 123, "status": "pending" } + }"#, + ); + + let reason = + super::verify_reported_upgrades(&zebra_chain::parameters::Network::Mainnet, &upgrades) + .expect_err("an unscheduled upgrade with a reported height must be rejected"); + assert!( + reason.contains("None"), + "error should show the compiled schedule has no height, got: {reason}" + ); + } +} diff --git a/packages/zaino-state/src/chain_index/tests.rs b/packages/zaino-state/src/chain_index/tests.rs index d2cd1d58e..47e954214 100644 --- a/packages/zaino-state/src/chain_index/tests.rs +++ b/packages/zaino-state/src/chain_index/tests.rs @@ -26,7 +26,7 @@ use std::path::{Path, PathBuf}; use tempfile::TempDir; use tokio::sync::OnceCell; use tokio::time::Duration; -use zaino_common::{network::ActivationHeights, DatabaseConfig, Network, StorageConfig}; +use zaino_common::{network::ActivationHeights, DatabaseConfig, StorageConfig}; use crate::{ chain_index::{ @@ -134,7 +134,7 @@ async fn load_with_settings( }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let indexer = NodeBackedChainIndex::new_with_sync_timings(source.clone(), config, sync_timings) @@ -222,7 +222,7 @@ async fn v1_finalised_seed_dir(mode: MockchainMode) -> &'static Path { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let zaino_db = FinalisedState::spawn(config, source).await.unwrap(); diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/ephemeral.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/ephemeral.rs index d2b455b22..34475c4ef 100644 --- a/packages/zaino-state/src/chain_index/tests/finalised_state/ephemeral.rs +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/ephemeral.rs @@ -15,7 +15,7 @@ use std::sync::Arc; use tempfile::TempDir; use zaino_common::network::ActivationHeights; -use zaino_common::{DatabaseConfig, Network, StorageConfig}; +use zaino_common::{DatabaseConfig, StorageConfig}; use zaino_proto::proto::utils::{compact_block_with_pool_types, PoolTypeFilter}; use crate::chain_index::finalised_state::FinalisedState; @@ -45,7 +45,7 @@ pub(crate) async fn spawn_ephemeral_finalised_state( }, ephemeral: true, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let finalised_state = FinalisedState::spawn(config, source).await?; diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations.rs index e0058faf6..11b399a6e 100644 --- a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations.rs +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations.rs @@ -10,3 +10,5 @@ mod v1_0_to_v1_1; #[cfg(not(feature = "transparent_address_history_experimental"))] mod v1_1_to_v1_2; +#[cfg(not(feature = "transparent_address_history_experimental"))] +mod v1_2_to_v1_3; diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_0_to_v1_1.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_0_to_v1_1.rs index e2db7587b..3f70ca3aa 100644 --- a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_0_to_v1_1.rs +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_0_to_v1_1.rs @@ -3,7 +3,7 @@ use std::path::PathBuf; use tempfile::TempDir; use zaino_common::network::ActivationHeights; -use zaino_common::{DatabaseConfig, Network, StorageConfig}; +use zaino_common::{DatabaseConfig, StorageConfig}; use crate::chain_index::finalised_state::capability::{ DbCore as _, DbRead as _, DbVersion, MigrationStatus, @@ -34,7 +34,7 @@ async fn v1_0_to_v1_1_metadata_migration() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_active_mockchain_source(150, blocks.clone()); @@ -100,7 +100,7 @@ async fn v1_0_to_v1_1_mixed_blockheaderdata_formats() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_active_mockchain_source(initial_active_height.0, blocks.clone()); diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_1_to_v1_2.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_1_to_v1_2.rs index 882763cdc..bfa0a32ef 100644 --- a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_1_to_v1_2.rs +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_1_to_v1_2.rs @@ -4,7 +4,7 @@ use lmdb::{Cursor as _, Transaction as _, WriteFlags}; use std::path::PathBuf; use tempfile::TempDir; use zaino_common::network::ActivationHeights; -use zaino_common::{DatabaseConfig, Network, StorageConfig}; +use zaino_common::{DatabaseConfig, StorageConfig}; use crate::chain_index::finalised_state::capability::{ BlockCoreExt as _, CapabilityRequest, DbRead as _, DbVersion, MigrationStatus, @@ -426,7 +426,7 @@ async fn v1_1_to_v1_2_spent_index_backfill_from_old_version() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_active_mockchain_source(initial_active_height.0, blocks.clone()); @@ -498,7 +498,7 @@ async fn v1_1_to_v1_2_spent_index_migration_resumes_after_crash() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_active_mockchain_source(initial_active_height.0, blocks.clone()); @@ -597,7 +597,7 @@ async fn v1_2_0_cache_missing_txid_location_index_is_rebuilt() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_active_mockchain_source(initial_active_height.0, blocks.clone()); diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_2_to_v1_3.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_2_to_v1_3.rs new file mode 100644 index 000000000..b7ac41d76 --- /dev/null +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/migrations/v1_2_to_v1_3.rs @@ -0,0 +1,102 @@ +//! V1.2.1 to V1.3.0 migration tests. +//! +//! Coverage note: these tests use `ActivationHeights::default()`, whose NU6.3 activation is `None`, +//! so the migration takes the below-activation branch (rebuild each commitment row in place from the +//! legacy fixed-length table, no ironwood). The at/above-activation *ironwood backfill* branch — +//! which refetches block data from the validator — cannot be exercised yet: `MockchainSource` +//! serves no ironwood commitment roots (see its `get_commitment_tree_roots` TODO) and the test +//! vectors carry no ironwood actions, so building a post-NU6.3 block would fail resolving the +//! (required) ironwood root. That path needs ironwood-capable test vectors before it can be tested. + +use std::path::PathBuf; +use tempfile::TempDir; +use zaino_common::network::ActivationHeights; +use zaino_common::{DatabaseConfig, StorageConfig}; + +use crate::chain_index::finalised_state::capability::{DbVersion, MigrationStatus}; +use crate::chain_index::finalised_state::finalised_source::v1::DB_VERSION_V1; +use crate::chain_index::finalised_state::FinalisedState; +use crate::chain_index::tests::init_tracing; +use crate::chain_index::tests::vectors::{ + build_active_mockchain_source, load_test_vectors, TestVectorData, +}; +use crate::{ChainIndexConfig, Height, StatusType}; + +fn v1_2_1() -> DbVersion { + DbVersion { + major: 1, + minor: 2, + patch: 1, + } +} + +/// Regression test for the startup ordering bug: opening a pre-1.3.0 cache used to start the +/// background validator concurrently with the v1.2.1 → v1.3.0 migration. The validator's +/// `initial_block_scan` read the freshly-created-empty `commitment_tree_data_1_3_0` table before the +/// migration rebuilt it, failing with `MDB_NOTFOUND` ("block scan") and latching `CriticalError`. +/// +/// The fix defers the validator until every migration finishes. This test builds an on-disk v1.2.1 +/// database, then reopens it through the production `FinalisedState::spawn` path (which migrates to +/// the current schema and only then starts the validator) and asserts it reaches `Ready` — i.e. the +/// migration completed *before* validation, and validation then passed. +// multi_thread required: the background validator runs blocking LMDB validation +// (`validate_block_blocking`) inline on its task, which would starve this test's status polling on +// a current-thread runtime. +#[tokio::test(flavor = "multi_thread")] +async fn v1_2_1_cache_migrates_to_current_then_validates() { + init_tracing(); + + let TestVectorData { blocks, .. } = load_test_vectors().unwrap(); + let active_height = Height(150); + + let temporary_directory: TempDir = tempfile::tempdir().unwrap(); + let database_path: PathBuf = temporary_directory.path().to_path_buf(); + + let database_config = ChainIndexConfig { + storage: StorageConfig { + database: DatabaseConfig { + path: database_path, + ..Default::default() + }, + ..Default::default() + }, + ephemeral: false, + db_version: 1, + network: ActivationHeights::default().to_regtest_network(), + }; + + let source = build_active_mockchain_source(active_height.0, blocks.clone()); + + // Build an on-disk v1.2.1 database (the pre-1.3.0 cache shape), then release it. + let old_database = + FinalisedState::build_db_to_version(database_config.clone(), source.clone(), v1_2_1()) + .await + .unwrap(); + old_database.wait_until_synced().await; + assert_eq!(old_database.get_metadata().await.unwrap().version, v1_2_1()); + old_database.shutdown().await.unwrap(); + drop(old_database); + + // Reopen through the production spawn path: it runs the v1.2.1 → v1.3.0 migration and only then + // starts the validator. Before the ordering fix this raced the migration and latched + // `CriticalError`; now it must migrate first and validate cleanly. + let migrated_database = FinalisedState::spawn(database_config.clone(), source.clone()) + .await + .unwrap(); + migrated_database.wait_until_synced().await; + + assert_eq!( + migrated_database.status(), + StatusType::Ready, + "the validator must run only after the migration completes, and then reach Ready" + ); + + let migrated_metadata = migrated_database.get_metadata().await.unwrap(); + assert_eq!(migrated_metadata.version, DB_VERSION_V1); + assert_eq!(migrated_metadata.migration_status, MigrationStatus::Empty); + + let migrated_height = migrated_database.db_height().await.unwrap().unwrap(); + assert_eq!(migrated_height, active_height); + + migrated_database.shutdown().await.unwrap(); +} diff --git a/packages/zaino-state/src/chain_index/tests/finalised_state/v1.rs b/packages/zaino-state/src/chain_index/tests/finalised_state/v1.rs index 1ef3c02ce..d3febab5b 100644 --- a/packages/zaino-state/src/chain_index/tests/finalised_state/v1.rs +++ b/packages/zaino-state/src/chain_index/tests/finalised_state/v1.rs @@ -4,7 +4,7 @@ use hex::ToHex; use std::path::PathBuf; use tempfile::TempDir; use zaino_common::network::ActivationHeights; -use zaino_common::{DatabaseConfig, Network, StorageConfig}; +use zaino_common::{DatabaseConfig, StorageConfig}; use zaino_proto::proto::utils::{compact_block_with_pool_types, PoolTypeFilter}; use crate::chain_index::finalised_state::finalised_source::FinalisedSource; @@ -44,7 +44,7 @@ pub(crate) async fn spawn_v1_zaino_db( }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let zaino_db = FinalisedState::spawn(config, source).await.unwrap(); @@ -193,7 +193,7 @@ async fn save_db_to_file_and_reload() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let source = build_mockchain_source(blocks.clone()); @@ -272,7 +272,7 @@ async fn load_db_backend_from_file() { }, ephemeral: false, db_version: 1, - network: Network::Regtest(ActivationHeights::default()), + network: ActivationHeights::default().to_regtest_network(), }; let finalized_state_backend: FinalisedSource = FinalisedSource::spawn_v1(&config).await.unwrap(); @@ -338,7 +338,7 @@ async fn try_write_invalid_block() { orchard_tree_size as u32, None, None, // no parent chainwork for this test - zaino_common::Network::Regtest(ActivationHeights::default()).to_zebra_network(), + ActivationHeights::default().to_regtest_network(), ); let mut chain_block = diff --git a/packages/zaino-state/src/chain_index/tests/mockchain_tests.rs b/packages/zaino-state/src/chain_index/tests/mockchain_tests.rs index 0d6baf05e..2ee2de790 100644 --- a/packages/zaino-state/src/chain_index/tests/mockchain_tests.rs +++ b/packages/zaino-state/src/chain_index/tests/mockchain_tests.rs @@ -7,7 +7,7 @@ use crate::{ vectors::{indexed_block_chain, load_test_vectors, TestVectorBlockData}, }, types::{BestChainLocation, ChainScope, TransactionHash}, - ChainIndex, NodeBackedChainIndexSubscriber, + ChainIndex, ChainIndexRpcExt, NodeBackedChainIndexSubscriber, }, BlockchainSource as _, Outpoint, }; @@ -16,8 +16,11 @@ use tokio_stream::StreamExt as _; use zaino_fetch::jsonrpsee::response::address_deltas::{ GetAddressDeltasParams, GetAddressDeltasResponse, }; -use zebra_chain::serialization::ZcashDeserializeInto; +use zaino_fetch::jsonrpsee::response::block_header::GetBlockHeader; +use zebra_chain::serialization::{ZcashDeserializeInto, ZcashSerialize as _}; use zebra_rpc::client::{GetAddressBalanceRequest, GetAddressTxIdsRequest}; +use zebra_rpc::methods::GetBlock; +use zebra_state::HashOrHeight; /// Polls the indexer's nonfinalized-state snapshot until its best-tip height /// equals `expected`, or panics after a 10 s budget. @@ -904,3 +907,397 @@ async fn get_outpoint_spenders_empty_and_single() { vec![None], ); } + +/// `z_get_block` served through the ChainIndex from the mock vectors: verbosity 0 +/// round-trips to the stored block, and verbosity 1 matches the source and reports the +/// block's hash / height / txids. +#[tokio::test(flavor = "multi_thread")] +async fn z_get_block() { + let (_blocks, _indexer, index_reader, mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + let active_height = mockchain.active_height(); + + for height in [1u32, active_height / 2, active_height] { + let id = HashOrHeight::Height(zebra_chain::block::Height(height)); + let expected_block = mockchain.get_block(id).await.unwrap().unwrap(); + + // Verbosity 0: the raw serialized block round-trips to the stored block. + let GetBlock::Raw(serialized) = index_reader + .z_get_block(height.to_string(), Some(0)) + .await + .unwrap() + else { + panic!("expected a raw block at verbosity 0"); + }; + let decoded: zebra_chain::block::Block = + serialized.as_ref().zcash_deserialize_into().unwrap(); + assert_eq!(decoded, *expected_block); + + // Verbosity 1: the ChainIndex delegates to the source and the object reports the + // block's hash, height, and every txid. + let via_index = index_reader + .z_get_block(height.to_string(), Some(1)) + .await + .unwrap(); + let via_source = mockchain.get_block_verbose(id, Some(1)).await.unwrap(); + let value = serde_json::to_value(&via_index).unwrap(); + assert_eq!(value, serde_json::to_value(&via_source).unwrap()); + assert_eq!( + value["hash"].as_str().unwrap(), + expected_block.hash().to_string() + ); + assert_eq!(value["height"].as_u64().unwrap(), u64::from(height)); + assert_eq!( + value["tx"].as_array().unwrap().len(), + expected_block.transactions.len() + ); + } +} + +/// `get_block_header` served through the ChainIndex from the mock vectors: the compact +/// (non-verbose) form round-trips to the stored header bytes, and the verbose form +/// matches the source and reports the right hash / height. +#[tokio::test(flavor = "multi_thread")] +async fn get_block_header() { + let (_blocks, _indexer, index_reader, mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + let active_height = mockchain.active_height(); + + for height in [1u32, active_height / 2, active_height] { + let id = HashOrHeight::Height(zebra_chain::block::Height(height)); + let block = mockchain.get_block(id).await.unwrap().unwrap(); + let hash = block.hash().to_string(); + + // Non-verbose: the compact hex decodes to the block header's serialization. + let GetBlockHeader::Compact(compact) = index_reader + .get_block_header(hash.clone(), false) + .await + .unwrap() + else { + panic!("expected a compact header when verbose = false"); + }; + assert_eq!( + hex::decode(compact).unwrap(), + block.header.zcash_serialize_to_vec().unwrap() + ); + + // Verbose: the ChainIndex delegates to the source and reports hash / height. + let via_index = index_reader + .get_block_header(hash.clone(), true) + .await + .unwrap(); + let via_source = mockchain + .get_block_header(hash.clone(), true) + .await + .unwrap(); + let value = serde_json::to_value(&via_index).unwrap(); + assert_eq!(value, serde_json::to_value(&via_source).unwrap()); + assert_eq!(value["hash"].as_str().unwrap(), hash); + assert_eq!(value["height"].as_u64().unwrap(), u64::from(height)); + } +} + +/// `get_block_deltas` served through the ChainIndex from the mock vectors: it matches the +/// source, reports the right block hash / height, and surfaces transparent deltas. +#[tokio::test(flavor = "multi_thread")] +async fn get_block_deltas() { + let (_blocks, _indexer, index_reader, mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + let active_height = mockchain.active_height(); + + let mut saw_delta_entries = false; + for height in [1u32, active_height / 2, active_height] { + let id = HashOrHeight::Height(zebra_chain::block::Height(height)); + let block = mockchain.get_block(id).await.unwrap().unwrap(); + let hash = block.hash().to_string(); + + let via_index = index_reader.get_block_deltas(hash.clone()).await.unwrap(); + let via_source = mockchain.get_block_deltas(hash.clone()).await.unwrap(); + assert_eq!( + serde_json::to_value(&via_index).unwrap(), + serde_json::to_value(&via_source).unwrap() + ); + assert_eq!(via_index.hash, hash); + assert_eq!(via_index.height, height); + if via_index + .deltas + .iter() + .any(|delta| !delta.inputs.is_empty() || !delta.outputs.is_empty()) + { + saw_delta_entries = true; + } + } + assert!( + saw_delta_entries, + "expected transparent deltas in at least one sampled block" + ); +} + +/// `get_difficulty` served through the ChainIndex from the mock vectors matches the +/// source and is a positive difficulty value. +#[tokio::test(flavor = "multi_thread")] +async fn get_difficulty() { + let (_blocks, _indexer, index_reader, mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let via_index = index_reader.get_difficulty().await.unwrap(); + let via_source = mockchain.get_difficulty().await.unwrap(); + assert_eq!(via_index, via_source); + assert!( + via_index > 0.0, + "difficulty should be positive, got {via_index}" + ); +} + +/// Drives the merged [`NodeBackedIndexerServiceSubscriber`] RPC layer over a +/// `MockchainSource`, confirming the service delegates to its chain index: the +/// service's `get_latest_block` reports the same tip the mockchain was synced to. +#[tokio::test(flavor = "multi_thread")] +async fn node_backed_indexer_service_serves_latest_block() { + use crate::indexer::node_backed_indexer::NodeBackedIndexerServiceSubscriber; + use crate::LightWalletIndexer as _; + use zaino_common::network::ActivationHeights; + + let (blocks, _indexer, index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let expected_tip = (blocks.len() as u32) - 1; + wait_for_indexer_tip(&index_reader, expected_tip).await; + + let service = NodeBackedIndexerServiceSubscriber::new_for_test( + index_reader, + ActivationHeights::default().to_regtest_network(), + ); + + let latest = service.get_latest_block().await.unwrap(); + assert_eq!(latest.height, expected_tip as u64); +} + +/// Dropping the chain index without an explicit `shutdown()` call must still +/// release source-owned background work: `Drop` previously only cancelled the +/// sync worker, and the async `shutdown()`'s `?` on the DB teardown skipped +/// the source release when the DB shutdown errored — either way the Direct +/// connection's Zebra syncer task could outlive its index. +#[tokio::test(flavor = "multi_thread")] +async fn dropping_the_chain_index_releases_the_source() { + let (_blocks, indexer, _index_reader, mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + assert!( + !mockchain.shutdown_called(), + "the source must not be shut down while the index is live" + ); + drop(indexer); + assert!( + mockchain.shutdown_called(), + "dropping the index must release source-owned background work" + ); +} + +/// zebra's `getblock` resolves negative heights against the tip (`-1` is the +/// tip block). The old Rpc backend forwarded the raw identifier string to the +/// validator, so `getblock "-1"` worked; the merged pre-parse used +/// `HashOrHeight::from_str`, which rejects negative heights. +#[tokio::test(flavor = "multi_thread")] +async fn z_get_block_resolves_negative_heights_against_the_tip() { + let (blocks, _indexer, index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + let tip_height = (blocks.len() as u32) - 1; + wait_for_indexer_tip(&index_reader, tip_height).await; + + let by_negative_height = index_reader + .z_get_block("-1".to_string(), Some(1)) + .await + .expect("height -1 must resolve to the tip block"); + let by_tip_height = index_reader + .z_get_block(tip_height.to_string(), Some(1)) + .await + .expect("the tip height resolves"); + assert_eq!(by_negative_height, by_tip_height); +} + +/// An unparsable `getblock` identifier must carry zcashd's legacy +/// InvalidParameter code (-8) as a typed `RpcError` in the `source()` chain, +/// not be flattened into an internal-error string: the serve layer recovers +/// legacy codes by downcast-walking the chain. +#[tokio::test(flavor = "multi_thread")] +async fn z_get_block_invalid_identifier_keeps_legacy_error_code() { + let (blocks, _indexer, index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + wait_for_indexer_tip(&index_reader, (blocks.len() as u32) - 1).await; + + let error = index_reader + .z_get_block("notablockid".to_string(), Some(1)) + .await + .expect_err("an unparsable identifier must be rejected"); + + let mut current: Option<&(dyn std::error::Error + 'static)> = Some(&error); + let mut rpc_error_code = None; + while let Some(source_error) = current { + if let Some(rpc_error) = + source_error.downcast_ref::() + { + rpc_error_code = Some(rpc_error.code); + break; + } + current = source_error.source(); + } + assert_eq!( + rpc_error_code, + Some(zebra_rpc::server::error::LegacyCode::InvalidParameter as i64), + "the typed RpcError (legacy code -8) must stay reachable via the source() chain" + ); +} + +/// During the initial finalised-state build there is no non-finalised +/// snapshot. Both pre-merge backends answered `getchaintips` in that window by +/// proxying the validator's own response; the merged service must fall back to +/// the source the same way, rather than serving UnavailableNotSyncedEnough for +/// the whole build (hours on mainnet). +#[tokio::test] +async fn get_chain_tips_falls_back_to_source_while_syncing() { + use crate::chain_index::non_finalised_state::ChainIndexSnapshot; + use crate::chain_index::tests::vectors::build_mockchain_source; + use crate::indexer::node_backed_indexer::chain_tips_for_snapshot; + + let blocks = load_test_vectors().unwrap().blocks; + let tip_height = (blocks.len() as u32) - 1; + let expected_tip_hash = blocks[tip_height as usize].zebra_block.hash().to_string(); + let mock = build_mockchain_source(blocks); + + let syncing_snapshot = ChainIndexSnapshot::StillSyncingFinalizedState { + validator_finalized_height: crate::Height(tip_height), + }; + + let tips = chain_tips_for_snapshot(&syncing_snapshot, &mock) + .await + .expect("the syncing window must proxy the source's chain tips"); + + assert_eq!( + tips, + vec![zaino_fetch::jsonrpsee::response::chain_tips::ChainTip::new( + tip_height, + expected_tip_hash, + 0, + zaino_fetch::jsonrpsee::response::chain_tips::ChainTipStatus::Active, + )] + ); +} + +/// Dropping the service must not panic on a thread with no Tokio runtime. +/// `Drop` runs the blocking teardown, which previously called +/// `Handle::current()` unconditionally; outside a runtime that panics, and a +/// panic inside `Drop` during unwind aborts the process. +/// +/// multi_thread required: the harness's finalised-state validation uses +/// `block_in_place`. The drop under test happens on a separate plain thread. +#[tokio::test(flavor = "multi_thread")] +async fn service_drop_survives_thread_without_runtime() { + use crate::indexer::node_backed_indexer::NodeBackedIndexerService; + use zaino_common::network::ActivationHeights; + + let (_blocks, indexer, _index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let service = NodeBackedIndexerService::new_for_test( + indexer, + ActivationHeights::default().to_regtest_network(), + ); + + std::thread::spawn(move || drop(service)) + .join() + .expect("dropping the service off-runtime must not panic"); +} + +/// Dropping the service must not panic on a current-thread Tokio runtime, +/// where `block_in_place` (the old unconditional teardown entry) aborts. +/// +/// multi_thread required: the harness's finalised-state validation uses +/// `block_in_place`. The drop under test happens inside a current-thread +/// runtime built on a separate thread. +#[tokio::test(flavor = "multi_thread")] +async fn service_drop_survives_current_thread_runtime() { + use crate::indexer::node_backed_indexer::NodeBackedIndexerService; + use zaino_common::network::ActivationHeights; + + let (_blocks, indexer, _index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let service = NodeBackedIndexerService::new_for_test( + indexer, + ActivationHeights::default().to_regtest_network(), + ); + + std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("current-thread runtime builds") + .block_on(async move { drop(service) }); + }) + .join() + .expect("dropping the service on a current-thread runtime must not panic"); +} + +/// The `Rpc` connection has no local chain-tip-change stream, so requesting a +/// chain-tip subscriber over such a source must yield `None` rather than +/// panic. Before this method returned `Option`, it existed only in a +/// panicking form (`.expect("chaintip_update_subscriber requires the Direct +/// connection")`) reachable by any embedder configured with `backend = "rpc"`; +/// pre-merge the misuse was a compile error because only the State-backed +/// subscriber type had the method. +#[tokio::test(flavor = "multi_thread")] +async fn chaintip_update_subscriber_absent_without_tip_stream() { + use crate::indexer::node_backed_indexer::NodeBackedIndexerServiceSubscriber; + use zaino_common::network::ActivationHeights; + + let (_blocks, _indexer, index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let service = NodeBackedIndexerServiceSubscriber::new_for_test( + index_reader, + ActivationHeights::default().to_regtest_network(), + ); + + assert!( + service.chaintip_update_subscriber().is_none(), + "a source with no local tip-change stream must yield no subscriber, not panic" + ); +} + +/// `sendrawtransaction` rejections must carry zcashd's legacy error code: +/// zaino-serve forwards the code by downcast-walking the `source()` chain for +/// the typed `RpcError` (`sendrawtransaction_error_object_from_indexer_error`), +/// so stringifying it downgrades the legacy `-8` "invalid hex" rejection to a +/// generic `-32603` internal error. +/// +/// Invalid hex fails in local validation before the source is consulted, so +/// this also pins that the rejection happens without a validator round-trip +/// (the mock's `send_raw_transaction` would panic if reached). +#[tokio::test(flavor = "multi_thread")] +async fn send_raw_transaction_invalid_hex_keeps_legacy_error_code() { + let (_blocks, _indexer, index_reader, _mockchain) = + load_test_vectors_and_sync_chain_index(MockchainMode::Static).await; + + let error = index_reader + .send_raw_transaction("notahexstring".to_string()) + .await + .expect_err("invalid hex must be rejected"); + + let mut current: Option<&(dyn std::error::Error + 'static)> = Some(&error); + let mut rpc_error_code = None; + while let Some(source_error) = current { + if let Some(rpc_error) = + source_error.downcast_ref::() + { + rpc_error_code = Some(rpc_error.code); + break; + } + current = source_error.source(); + } + assert_eq!( + rpc_error_code, + Some(zebra_rpc::server::error::LegacyCode::InvalidParameter as i64), + "the typed RpcError (legacy code -8) must stay reachable via the source() chain" + ); +} diff --git a/packages/zaino-state/src/chain_index/tests/proptest_blockgen.rs b/packages/zaino-state/src/chain_index/tests/proptest_blockgen.rs index ddb17407a..22c0da968 100644 --- a/packages/zaino-state/src/chain_index/tests/proptest_blockgen.rs +++ b/packages/zaino-state/src/chain_index/tests/proptest_blockgen.rs @@ -7,7 +7,7 @@ use proptest::{ }; use rand::seq::IndexedRandom; use tokio_stream::StreamExt as _; -use zaino_common::{network::ActivationHeights, DatabaseConfig, Network, StorageConfig}; +use zaino_common::{network::ActivationHeights, DatabaseConfig, StorageConfig}; use zaino_fetch::jsonrpsee::response::address_deltas::{ GetAddressDeltasParams, GetAddressDeltasResponse, }; @@ -33,8 +33,8 @@ use crate::{ types::BestChainLocation, NonFinalizedSnapshot, OPERATIONAL_NFS_DEPTH, }, - BlockHash, BlockchainSource, ChainIndex, ChainIndexConfig, Height, NodeBackedChainIndex, - NodeBackedChainIndexSubscriber, TransactionHash, + BlockHash, BlockchainSource, ChainIndex, ChainIndexConfig, ChainIndexRpcExt, Height, + NodeBackedChainIndex, NodeBackedChainIndexSubscriber, TransactionHash, }; use zaino_proto::proto::utils::PoolTypeFilter; @@ -58,7 +58,7 @@ fn passthrough_test( ), ) { passthrough_test_on( - Network::Regtest(ActivationHeights::default()), + ActivationHeights::default().to_regtest_network(), // Slow the source enough to hold the indexer in passthrough while the // assertions run, without slowing passthrough more than necessary. Some(Duration::from_millis(100)), @@ -76,7 +76,7 @@ fn passthrough_test( /// header's merkle root is already arbitrary — the passthrough path tolerates that by /// construction. fn passthrough_test_on( - network: Network, + network: zebra_chain::parameters::Network, source_delay: Option, mutate_segment: impl Fn(&mut Vec>), test: impl AsyncFn( @@ -92,7 +92,7 @@ fn passthrough_test_on( // from this line to `runtime.block_on(async {` are all // copy-pasted. Could a macro get rid of some of this boilerplate? - proptest::proptest!(proptest::test_runner::Config::with_cases(1), |(segments in make_branching_chain(branch_count, segment_length, network))| { + proptest::proptest!(proptest::test_runner::Config::with_cases(1), |(segments in make_branching_chain(branch_count, segment_length, network.clone()))| { let runtime = tokio::runtime::Builder::new_multi_thread().worker_threads(2).enable_time().build().unwrap(); runtime.block_on(async { let (mut genesis_segment, mut branching_segments) = segments; @@ -120,7 +120,7 @@ fn passthrough_test_on( }, ephemeral: true, db_version: 1, - network, + network: network.clone(), }; @@ -451,7 +451,7 @@ fn zebra_arbitrary_generates_v6_transactions_for_nu6_3() { /// NU6.3 active from height 2, so post-activation generated blocks carry V6 /// transactions whose shielded data lands in the Ironwood pool. -const NU6_3_ACTIVE_HEIGHTS: ActivationHeights = ActivationHeights { +const IRONWOOD_ONLY_HEIGHTS: ActivationHeights = ActivationHeights { before_overwinter: Some(1), overwinter: Some(1), sapling: Some(1), @@ -478,7 +478,7 @@ const NU6_3_ACTIVE_HEIGHTS: ActivationHeights = ActivationHeights { /// source of truth. #[test] fn passthrough_metadata_consistency_ironwood_only() { - metadata_consistency_for_era(NU6_3_ACTIVE_HEIGHTS, Some(2), false) + metadata_consistency_for_era(IRONWOOD_ONLY_HEIGHTS, Some(2), false) } /// Orchard-only heights: every upgrade through NU6.2 at height 2, NU6.3 never @@ -516,7 +516,7 @@ fn passthrough_metadata_consistency_orchard_to_ironwood_transition() { metadata_consistency_for_era( ActivationHeights { nu6_3: Some(boundary), - ..NU6_3_ACTIVE_HEIGHTS + ..IRONWOOD_ONLY_HEIGHTS }, Some(boundary), true, @@ -602,7 +602,7 @@ fn metadata_consistency_for_era( }; passthrough_test_on( - Network::Regtest(heights), + heights.to_regtest_network(), // No artificial source delay: this test waits for the indexer to finish // syncing, because compact blocks are not served while the finalised state // is still syncing (get_compact_block's StillSyncingFinalizedState arm). @@ -791,14 +791,14 @@ fn metadata_consistency_for_era( #[test] fn make_chain() { init_tracing(); - let network = Network::Regtest(ActivationHeights::default()); + let network = ActivationHeights::default().to_regtest_network(); let segment_length = 12; let branch_count = 2; // default is 256. As each case takes multiple seconds, this seems too many. // TODO: this should be higher than 1. Currently set to 1 for ease of iteration - proptest::proptest!(proptest::test_runner::Config::with_cases(1), |(segments in make_branching_chain(branch_count, segment_length, network))| { + proptest::proptest!(proptest::test_runner::Config::with_cases(1), |(segments in make_branching_chain(branch_count, segment_length, network.clone()))| { let runtime = tokio::runtime::Builder::new_multi_thread().worker_threads(2).enable_time().build().unwrap(); runtime.block_on(async { let (genesis_segment, branching_segments) = segments; @@ -822,7 +822,7 @@ fn make_chain() { }, ephemeral: true, db_version: 1, - network, + network: network.clone(), }; @@ -1031,6 +1031,115 @@ impl BlockchainSource for ProptestMockchain { } } + async fn get_block_verbose( + &self, + _hash_or_height: HashOrHeight, + _verbosity: Option, + ) -> BlockchainSourceResult { + // ProptestMockchain exercises sync/reorg, not the verbose getblock RPC. + unimplemented!() + } + + async fn get_block_header( + &self, + _hash: String, + _verbose: bool, + ) -> BlockchainSourceResult + { + // ProptestMockchain exercises sync/reorg, not the getblockheader RPC. + unimplemented!() + } + + async fn get_block_deltas( + &self, + _hash: String, + ) -> BlockchainSourceResult { + // ProptestMockchain exercises sync/reorg, not the getblockdeltas RPC. + unimplemented!() + } + + async fn get_difficulty(&self) -> BlockchainSourceResult { + // ProptestMockchain exercises sync/reorg, not the getdifficulty RPC. + unimplemented!() + } + + async fn get_blockchain_info( + &self, + ) -> BlockchainSourceResult { + // ProptestMockchain exercises sync/reorg, not the getblockchaininfo RPC. + unimplemented!() + } + + async fn get_info(&self) -> BlockchainSourceResult { + unimplemented!() + } + + async fn get_peer_info( + &self, + ) -> BlockchainSourceResult { + unimplemented!() + } + + async fn get_chain_tips( + &self, + ) -> BlockchainSourceResult + { + unimplemented!() + } + + async fn get_block_subsidy( + &self, + _height: u32, + ) -> BlockchainSourceResult + { + unimplemented!() + } + + async fn get_mining_info( + &self, + ) -> BlockchainSourceResult + { + unimplemented!() + } + + async fn get_tx_out( + &self, + _txid: String, + _n: u32, + _include_mempool: Option, + ) -> BlockchainSourceResult { + unimplemented!() + } + + async fn get_spent_info( + &self, + _request: zaino_fetch::jsonrpsee::response::GetSpentInfoRequest, + ) -> BlockchainSourceResult { + unimplemented!() + } + + async fn get_network_sol_ps( + &self, + _blocks: Option, + _height: Option, + ) -> BlockchainSourceResult { + unimplemented!() + } + + async fn send_raw_transaction( + &self, + _raw_transaction_hex: String, + ) -> BlockchainSourceResult { + unimplemented!() + } + + async fn get_treestate_by_id( + &self, + _hash_or_height: String, + ) -> BlockchainSourceResult { + unimplemented!() + } + /// Returns the block commitment tree data by hash async fn get_commitment_tree_roots( &self, @@ -1257,9 +1366,9 @@ fn make_branching_chain( // The length of the initial segment, and of the branches // TODO: it would be useful to allow branches of different lengths. chain_size: usize, - network_override: Network, + network_override: zebra_chain::parameters::Network, ) -> BoxedStrategy<(ChainSegment, Vec)> { - let network_override = Some(network_override.to_zebra_network()); + let network_override = Some(network_override); add_segment( SummaryDebug(Vec::new()), network_override.clone(), diff --git a/packages/zaino-state/src/chain_index/types/db/block.rs b/packages/zaino-state/src/chain_index/types/db/block.rs index 1a224b5f2..cb55b4ba1 100644 --- a/packages/zaino-state/src/chain_index/types/db/block.rs +++ b/packages/zaino-state/src/chain_index/types/db/block.rs @@ -321,7 +321,7 @@ mod tests { impl ChainWork { /// Returns ChainWork as a U256. - pub(super) fn to_u256(&self) -> primitive_types::U256 { + pub(super) fn to_u256(self) -> primitive_types::U256 { primitive_types::U256::from_big_endian(&self.0) } diff --git a/packages/zaino-state/src/config.rs b/packages/zaino-state/src/config.rs index 0083dbcd4..3cfff0d76 100644 --- a/packages/zaino-state/src/config.rs +++ b/packages/zaino-state/src/config.rs @@ -42,31 +42,37 @@ impl std::fmt::Display for DonationAddress { } } -/// Type of backend to be used. +/// How the [`NodeBackedIndexerService`](crate::NodeBackedIndexerService) connects to its +/// validator to source blockchain data. /// -/// Determines how Zaino fetches blockchain data from the validator. -#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, serde::Deserialize, serde::Serialize)] -#[serde(rename_all = "lowercase")] -pub enum BackendType { - /// Uses Zebra's ReadStateService for direct state access. - /// - /// More efficient but requires running on the same machine as Zebra. - State, - /// Uses JSON-RPC client to fetch data. +/// Carries the connection-specific configuration inline: the `Direct` variant owns the +/// Zebra `ReadStateService` settings, while `Rpc` needs only the JSON-RPC connection bits +/// already held in [`CommonBackendConfig`]. +#[derive(Debug, Clone)] +pub enum ValidatorConnectionType { + /// JSON-RPC connection (formerly `Fetch`). /// /// Compatible with Zcashd, Zebra, or another Zaino instance. - #[default] - Fetch, + Rpc, + /// Direct Zebra `ReadStateService` connection (formerly `State`). + /// + /// More efficient but requires running alongside a Zebra whose state DB and gRPC + /// sync endpoint we own. + Direct(DirectConnectionConfig), } -/// Unified backend configuration enum. +/// Connection parameters for the [`ValidatorConnectionType::Direct`] backend. +/// +/// Bundles the Zebra `ReadStateService` DB config with the gRPC sync endpoint used to +/// drive it, so the whole "direct connection" description travels as one value. #[derive(Debug, Clone)] -#[allow(deprecated)] -pub enum BackendConfig { - /// StateService config. - State(StateServiceConfig), - /// Fetchservice config. - Fetch(FetchServiceConfig), +pub struct DirectConnectionConfig { + /// Zebra [`zebra_state::ReadStateService`] config data (DB cache dir etc.). + pub validator_state_config: zebra_state::Config, + /// Validator gRPC address (requires ip:port format for Zebra state sync). + pub validator_grpc_address: std::net::SocketAddr, + /// Whether validator cookie authentication is enabled. + pub validator_cookie_auth: bool, } /// Configuration shared by every backend variant. @@ -109,29 +115,55 @@ pub struct CommonBackendConfig { pub indexer_version: String, } -/// Holds config data for [crate::StateService]. +impl CommonBackendConfig { + /// Builds a [`CommonBackendConfig`], applying the default RPC user/password + /// placeholder (`"xxxxxx"`) and this crate's `CARGO_PKG_VERSION` as the + /// indexer version. Shared by the [`NodeBackedIndexerServiceConfig`] constructors. + #[allow(clippy::too_many_arguments)] + pub fn new( + validator_rpc_address: String, + validator_cookie_path: Option, + validator_rpc_user: Option, + validator_rpc_password: Option, + service: ServiceConfig, + storage: StorageConfig, + ephemeral_finalised_state: bool, + network: Network, + donation_address: Option, + ) -> Self { + CommonBackendConfig { + validator_rpc_address, + validator_cookie_path, + validator_rpc_user: validator_rpc_user.unwrap_or_else(|| "xxxxxx".to_string()), + validator_rpc_password: validator_rpc_password.unwrap_or_else(|| "xxxxxx".to_string()), + service, + storage, + ephemeral_finalised_state, + network, + donation_address, + indexer_version: env!("CARGO_PKG_VERSION").to_string(), + } + } +} + +/// Holds config data for [`crate::NodeBackedIndexerService`]. +/// +/// Replaces the former per-backend `FetchServiceConfig` / `StateServiceConfig`: the +/// shared bits live in [`CommonBackendConfig`] and the backend-specific bits (only the +/// `Direct` backend has any) live in [`ValidatorConnectionType`]. #[derive(Debug, Clone)] -// #[deprecated] -pub struct StateServiceConfig { - /// Settings shared with [`FetchServiceConfig`]. +pub struct NodeBackedIndexerServiceConfig { + /// Connection-independent settings (validator RPC, storage, network, ...). pub common: CommonBackendConfig, - /// Zebra [`zebra_state::ReadStateService`] config data - pub validator_state_config: zebra_state::Config, - /// Validator gRPC address (requires ip:port format for Zebra state sync). - pub validator_grpc_address: std::net::SocketAddr, - /// Validator cookie auth. - pub validator_cookie_auth: bool, + /// Which validator connection to use, and its connection-specific config. + pub connection: ValidatorConnectionType, } -#[allow(deprecated)] -impl StateServiceConfig { - /// Returns a new instance of [`StateServiceConfig`]. +impl NodeBackedIndexerServiceConfig { + /// Returns a JSON-RPC (`Rpc`) service config (formerly `FetchServiceConfig::new`). #[allow(clippy::too_many_arguments)] - pub fn new( - validator_state_config: zebra_state::Config, + pub fn new_rpc( validator_rpc_address: String, - validator_grpc_address: std::net::SocketAddr, - validator_cookie_auth: bool, validator_cookie_path: Option, validator_rpc_user: Option, validator_rpc_password: Option, @@ -141,44 +173,30 @@ impl StateServiceConfig { network: Network, donation_address: Option, ) -> Self { - tracing::trace!( - activations = ?network.to_zebra_network().full_activation_list(), - "state service expecting NU activations" - ); - StateServiceConfig { - common: CommonBackendConfig { + NodeBackedIndexerServiceConfig { + common: CommonBackendConfig::new( validator_rpc_address, validator_cookie_path, - validator_rpc_user: validator_rpc_user.unwrap_or("xxxxxx".to_string()), - validator_rpc_password: validator_rpc_password.unwrap_or("xxxxxx".to_string()), + validator_rpc_user, + validator_rpc_password, service, storage, ephemeral_finalised_state, network, donation_address, - indexer_version: env!("CARGO_PKG_VERSION").to_string(), - }, - validator_state_config, - validator_grpc_address, - validator_cookie_auth, + ), + connection: ValidatorConnectionType::Rpc, } } -} -/// Holds config data for [crate::FetchService]. -#[derive(Debug, Clone)] -#[deprecated] -pub struct FetchServiceConfig { - /// Settings shared with [`StateServiceConfig`]. - pub common: CommonBackendConfig, -} - -#[allow(deprecated)] -impl FetchServiceConfig { - /// Returns a new instance of [`FetchServiceConfig`]. + /// Returns a direct-`ReadStateService` (`Direct`) service config + /// (formerly `StateServiceConfig::new`). #[allow(clippy::too_many_arguments)] - pub fn new( + pub fn new_direct( + validator_state_config: zebra_state::Config, validator_rpc_address: String, + validator_grpc_address: std::net::SocketAddr, + validator_cookie_auth: bool, validator_cookie_path: Option, validator_rpc_user: Option, validator_rpc_password: Option, @@ -188,19 +206,25 @@ impl FetchServiceConfig { network: Network, donation_address: Option, ) -> Self { - FetchServiceConfig { - common: CommonBackendConfig { + // The config carries only the network kind; the activation schedule + // is adopted from the validator at spawn and logged there (#1076). + NodeBackedIndexerServiceConfig { + common: CommonBackendConfig::new( validator_rpc_address, validator_cookie_path, - validator_rpc_user: validator_rpc_user.unwrap_or("xxxxxx".to_string()), - validator_rpc_password: validator_rpc_password.unwrap_or("xxxxxx".to_string()), + validator_rpc_user, + validator_rpc_password, service, storage, ephemeral_finalised_state, network, donation_address, - indexer_version: env!("CARGO_PKG_VERSION").to_string(), - }, + ), + connection: ValidatorConnectionType::Direct(DirectConnectionConfig { + validator_state_config, + validator_grpc_address, + validator_cookie_auth, + }), } } } @@ -212,8 +236,10 @@ pub struct ChainIndexConfig { pub storage: StorageConfig, /// Database version selected to be run. pub db_version: u32, - /// Network type. - pub network: Network, + /// The runtime network, carrying the activation schedule adopted from + /// the validator (zaino#1076) — or, in fixtures that are their own + /// chain, the fixture's schedule. + pub network: zebra_chain::parameters::Network, /// Ephemeral finalised state: /// /// If true, FinalisedState does not write data to disk, @@ -225,44 +251,24 @@ pub struct ChainIndexConfig { } impl ChainIndexConfig { - /// Returns a new instance of [`BlockCacheConfig`]. - #[allow(dead_code)] - pub fn new(storage: StorageConfig, db_version: u32, network: Network, ephemeral: bool) -> Self { - ChainIndexConfig { - storage, - db_version, - network, - ephemeral, - } - } -} - -impl From for ChainIndexConfig { - fn from(value: CommonBackendConfig) -> Self { + /// Builds the chain-index config from the backend config plus the + /// runtime network. The backend config carries only a network kind, so + /// the adopted runtime network arrives as its own argument — there is + /// no conversion from a service config alone. + pub fn from_backend_config( + common: &CommonBackendConfig, + network: zebra_chain::parameters::Network, + ) -> Self { Self { - storage: value.storage, + storage: common.storage.clone(), // TODO: update zaino configs to include db version. db_version: 1, - network: value.network, - ephemeral: value.ephemeral_finalised_state, + network, + ephemeral: common.ephemeral_finalised_state, } } } -#[allow(deprecated)] -impl From for ChainIndexConfig { - fn from(value: StateServiceConfig) -> Self { - value.common.into() - } -} - -#[allow(deprecated)] -impl From for ChainIndexConfig { - fn from(value: FetchServiceConfig) -> Self { - value.common.into() - } -} - #[cfg(test)] mod tests { use super::*; diff --git a/packages/zaino-state/src/error.rs b/packages/zaino-state/src/error.rs index 839c17045..b0eb87b31 100644 --- a/packages/zaino-state/src/error.rs +++ b/packages/zaino-state/src/error.rs @@ -10,11 +10,11 @@ use std::{any::type_name, fmt::Display}; use zaino_fetch::jsonrpsee::connector::RpcRequestError; use zaino_proto::proto::utils::GetBlockRangeError; -/// Errors related to the `StateService`. -// #[deprecated] +/// Errors returned by the [`NodeBackedIndexerService`](crate::NodeBackedIndexerService) +/// subscriber's `ZcashIndexer` / `LightWalletIndexer` methods. #[derive(Debug, thiserror::Error)] #[allow(clippy::result_large_err)] -pub enum StateServiceError { +pub enum NodeBackedIndexerServiceError { /// Critical Errors, Restart Zaino. #[error("Critical error: {0}")] Critical(String), @@ -88,7 +88,7 @@ pub enum StateServiceError { UnavailableNotSyncedEnough, } -impl From for StateServiceError { +impl From for NodeBackedIndexerServiceError { fn from(value: GetBlockRangeError) -> Self { match value { GetBlockRangeError::StartHeightOutOfRange => { @@ -115,55 +115,59 @@ impl From for StateServiceError { } #[allow(deprecated)] -impl From for tonic::Status { - fn from(error: StateServiceError) -> Self { +impl From for tonic::Status { + fn from(error: NodeBackedIndexerServiceError) -> Self { match error { - StateServiceError::Critical(message) => tonic::Status::internal(message), - StateServiceError::Custom(message) => tonic::Status::internal(message), - StateServiceError::JoinError(err) => { + NodeBackedIndexerServiceError::Critical(message) => tonic::Status::internal(message), + NodeBackedIndexerServiceError::Custom(message) => tonic::Status::internal(message), + NodeBackedIndexerServiceError::JoinError(err) => { tonic::Status::internal(format!("Join error: {err}")) } - StateServiceError::JsonRpcConnectorError(err) => { + NodeBackedIndexerServiceError::JsonRpcConnectorError(err) => { tonic::Status::internal(format!("JsonRpcConnector error: {err}")) } - StateServiceError::RpcError(err) => { + NodeBackedIndexerServiceError::RpcError(err) => { tonic::Status::internal(format!("RPC error: {err:?}")) } - StateServiceError::ChainIndexError(err) => match err.kind { + NodeBackedIndexerServiceError::ChainIndexError(err) => match err.kind { ChainIndexErrorKind::InternalServerError => tonic::Status::internal(err.message), ChainIndexErrorKind::InvalidSnapshot => { tonic::Status::failed_precondition(err.message) } }, - StateServiceError::BlockCacheError(err) => { + NodeBackedIndexerServiceError::BlockCacheError(err) => { tonic::Status::internal(format!("BlockCache error: {err:?}")) } - StateServiceError::MempoolError(err) => { + NodeBackedIndexerServiceError::MempoolError(err) => { tonic::Status::internal(format!("Mempool error: {err:?}")) } - StateServiceError::TonicStatusError(err) => err, - StateServiceError::SerializationError(err) => { + NodeBackedIndexerServiceError::TonicStatusError(err) => err, + NodeBackedIndexerServiceError::SerializationError(err) => { tonic::Status::internal(format!("Serialization error: {err}")) } - StateServiceError::TryFromIntError(err) => { + NodeBackedIndexerServiceError::TryFromIntError(err) => { tonic::Status::internal(format!("Integer conversion error: {err}")) } - StateServiceError::IoError(err) => tonic::Status::internal(format!("IO error: {err}")), - StateServiceError::Generic(err) => { + NodeBackedIndexerServiceError::IoError(err) => { + tonic::Status::internal(format!("IO error: {err}")) + } + NodeBackedIndexerServiceError::Generic(err) => { tonic::Status::internal(format!("Generic error: {err}")) } - ref err @ StateServiceError::ZebradVersionMismatch { .. } => { + ref err @ NodeBackedIndexerServiceError::ZebradVersionMismatch { .. } => { tonic::Status::internal(err.to_string()) } - StateServiceError::UnhandledRpcError(e) => tonic::Status::internal(e.to_string()), - StateServiceError::UnavailableNotSyncedEnough => { + NodeBackedIndexerServiceError::UnhandledRpcError(e) => { + tonic::Status::internal(e.to_string()) + } + NodeBackedIndexerServiceError::UnavailableNotSyncedEnough => { tonic::Status::failed_precondition("zaino not yet synced".to_string()) } } } } -impl From> for StateServiceError { +impl From> for NodeBackedIndexerServiceError { fn from(value: RpcRequestError) -> Self { match value { RpcRequestError::Transport(transport_error) => { @@ -184,122 +188,6 @@ impl From> for StateServiceError { } } -/// Errors related to the `FetchService`. -#[deprecated] -#[derive(Debug, thiserror::Error)] -pub enum FetchServiceError { - /// Critical Errors, Restart Zaino. - #[error("Critical error: {0}")] - Critical(String), - - /// Error from JsonRpcConnector. - #[error("JsonRpcConnector error: {0}")] - JsonRpcConnectorError(#[from] zaino_fetch::jsonrpsee::error::TransportError), - - /// Chain index error. - #[error("Chain index error: {0}")] - ChainIndexError(#[from] ChainIndexError), - - /// RPC error in compatibility with zcashd. - #[error("RPC error: {0:?}")] - RpcError(#[from] zaino_fetch::jsonrpsee::connector::RpcError), - - /// Tonic gRPC error. - #[error("Tonic status error: {0}")] - TonicStatusError(#[from] tonic::Status), - - /// Serialization error. - #[error("Serialization error: {0}")] - SerializationError(#[from] zebra_chain::serialization::SerializationError), - #[error("Zaino has not synced high enough to serve this data")] - /// Zaino has not yet synced. - UnavailableNotSyncedEnough, -} - -impl From for tonic::Status { - fn from(error: FetchServiceError) -> Self { - match error { - FetchServiceError::Critical(message) => tonic::Status::internal(message), - FetchServiceError::JsonRpcConnectorError(err) => { - tonic::Status::internal(format!("JsonRpcConnector error: {err}")) - } - FetchServiceError::ChainIndexError(err) => match err.kind { - ChainIndexErrorKind::InternalServerError => tonic::Status::internal(err.message), - ChainIndexErrorKind::InvalidSnapshot => { - tonic::Status::failed_precondition(err.message) - } - }, - FetchServiceError::RpcError(err) => { - tonic::Status::internal(format!("RPC error: {err:?}")) - } - FetchServiceError::TonicStatusError(err) => err, - FetchServiceError::SerializationError(err) => { - tonic::Status::internal(format!("Serialization error: {err}")) - } - FetchServiceError::UnavailableNotSyncedEnough => { - tonic::Status::failed_precondition("zaino not yet synced".to_string()) - } - } - } -} - -impl From> for FetchServiceError { - fn from(value: RpcRequestError) -> Self { - match value { - RpcRequestError::Transport(transport_error) => { - FetchServiceError::JsonRpcConnectorError(transport_error) - } - RpcRequestError::JsonRpc(error) => { - FetchServiceError::Critical(format!("argument failed to serialze: {error}")) - } - RpcRequestError::InternalUnrecoverable(e) => { - FetchServiceError::Critical(format!("Internal unrecoverable error: {e}")) - } - RpcRequestError::ServerWorkQueueFull => FetchServiceError::Critical( - "Server queue full. Handling for this not yet implemented".to_string(), - ), - RpcRequestError::Method(e) => FetchServiceError::Critical(format!( - "unhandled rpc-specific {} error: {}", - type_name::(), - e.to_string() - )), - RpcRequestError::UnexpectedErrorResponse(error) => { - FetchServiceError::Critical(format!( - "unhandled rpc-specific {} error: {}", - type_name::(), - error - )) - } - } - } -} - -impl From for FetchServiceError { - fn from(value: GetBlockRangeError) -> Self { - match value { - GetBlockRangeError::StartHeightOutOfRange => { - FetchServiceError::TonicStatusError(tonic::Status::out_of_range( - "Error: Start height out of range. Failed to convert to u32.", - )) - } - GetBlockRangeError::NoStartHeightProvided => FetchServiceError::TonicStatusError( - tonic::Status::out_of_range("Error: No start height given"), - ), - GetBlockRangeError::EndHeightOutOfRange => { - FetchServiceError::TonicStatusError(tonic::Status::out_of_range( - "Error: End height out of range. Failed to convert to u32.", - )) - } - GetBlockRangeError::NoEndHeightProvided => FetchServiceError::TonicStatusError( - tonic::Status::out_of_range("Error: No end height given."), - ), - GetBlockRangeError::PoolTypeArgumentError(_) => FetchServiceError::TonicStatusError( - tonic::Status::invalid_argument("Error: invalid pool type"), - ), - } - } -} - /// These aren't the best conversions, but the MempoolError should go away /// in favor of a new type with the new chain cache is complete impl From> for MempoolError { @@ -622,6 +510,18 @@ impl ChainIndexError { } } + /// Constructs an `InternalServerError`-kind error from a typed error, + /// preserving it as `source` so zaino-serve's RPC-error-code recovery + /// walks can downcast to it (e.g. the legacy `-8` code carried by an + /// [`RpcError`](zaino_fetch::jsonrpsee::connector::RpcError)). + pub(crate) fn internal_from(error: impl std::error::Error + Send + Sync + 'static) -> Self { + Self { + kind: ChainIndexErrorKind::InternalServerError, + message: error.to_string(), + source: Some(Box::new(error)), + } + } + pub(crate) fn backing_validator(value: impl std::error::Error + Send + Sync + 'static) -> Self { Self { kind: ChainIndexErrorKind::InternalServerError, diff --git a/packages/zaino-state/src/indexer.rs b/packages/zaino-state/src/indexer.rs index acdf57a43..42199f193 100644 --- a/packages/zaino-state/src/indexer.rs +++ b/packages/zaino-state/src/indexer.rs @@ -1,5 +1,10 @@ -//! Holds the Indexer trait containing the zcash RPC definitions served by zaino -//! and generic wrapper structs for the various backend options available. +//! Zaino's indexer frontend: the `ZcashIndexer` / `LightWalletIndexer` RPC trait +//! definitions served by zaino, the generic [`IndexerService`] / [`IndexerSubscriber`] +//! wrappers, and the concrete [`node_backed_indexer::NodeBackedIndexerService`] — the +//! single validator-backed service (JSON-RPC `Rpc` or direct `ReadStateService` +//! connection, selected at runtime). + +pub(crate) mod node_backed_indexer; use crate::SendFut; use tokio::{sync::mpsc, time::timeout}; @@ -12,9 +17,12 @@ use zaino_fetch::jsonrpsee::response::{ chain_tips::GetChainTipsResponse, mining_info::GetMiningInfoWire, peer_info::GetPeerInfo, - z_validate_address::ZValidateAddressResponse, + z_validate_address::{ + InvalidZValidateAddress, KnownZValidateAddress, ZValidateAddressResponse, + DEPRECATION_NOTICE as Z_VALIDATE_DEPRECATION, + }, GetMempoolInfoResponse, GetNetworkSolPsResponse, GetSpentInfoRequest, GetSpentInfoResponse, - GetTxOutSetInfoResponse, + GetSubtreesResponse, GetTxOutSetInfoResponse, }; use zaino_proto::proto::{ compact_formats::CompactBlock, @@ -29,7 +37,9 @@ use zebra_chain::{ block::Height, serialization::BytesInDisplayOrder as _, subtree::NoteCommitmentSubtreeIndex, }; use zebra_rpc::{ - client::{GetSubtreesByIndexResponse, GetTreestateResponse, ValidateAddressResponse}, + client::{ + GetSubtreesByIndexResponse, GetTreestateResponse, SubtreeRpcData, ValidateAddressResponse, + }, methods::{ AddressBalance, GetAddressBalanceRequest, GetAddressTxIdsRequest, GetAddressUtxos, GetBlock, GetBlockHash, GetBlockchainInfoResponse, GetInfo, GetRawTransaction, @@ -43,10 +53,10 @@ use crate::{ AddressStream, CompactBlockStream, CompactTransactionStream, RawTransactionStream, SubtreeRootReplyStream, UtxoReplyStream, }, - BackendType, }; -/// Wrapper Struct for a ZainoState chain-fetch service (StateService, FetchService) +/// Wrapper struct for a ZainoState chain-fetch service (currently the single +/// [`node_backed_indexer::NodeBackedIndexerService`]). /// /// The future plan is to also add a TonicService and DarksideService to this to enable /// wallets to use a single unified chain fetch service. @@ -87,9 +97,6 @@ where /// Implementors automatically gain [`Liveness`](zaino_common::probing::Liveness) and /// [`Readiness`](zaino_common::probing::Readiness) via the [`Status`] supertrait. pub trait ZcashService: Sized + Status { - /// Backend type. Read state or fetch service. - const BACKEND_TYPE: BackendType; - /// A subscriber to the service, used to fetch chain data. type Subscriber: Clone + ZcashIndexer + LightWalletIndexer + Status; @@ -108,7 +115,8 @@ pub trait ZcashService: Sized + Status { fn close(&mut self); } -/// Wrapper Struct for a ZainoState chain-fetch service subscriber (StateServiceSubscriber, FetchServiceSubscriber) +/// Wrapper struct for a ZainoState chain-fetch service subscriber (currently the single +/// [`node_backed_indexer::NodeBackedIndexerServiceSubscriber`]). /// /// The future plan is to also add a TonicServiceSubscriber and DarksideServiceSubscriber to this to enable wallets to use a single unified chain fetch service. #[derive(Clone)] @@ -983,3 +991,523 @@ pub(crate) async fn handle_raw_transaction( } } } + +/// Maps a Zebra network to the `zcash_protocol` network type used for address decoding. +fn address_network_type( + network: &zebra_chain::parameters::Network, +) -> zcash_protocol::consensus::NetworkType { + use zcash_protocol::consensus::NetworkType; + use zebra_chain::parameters::NetworkKind; + match network.kind() { + NetworkKind::Mainnet => NetworkType::Main, + NetworkKind::Testnet => NetworkType::Test, + NetworkKind::Regtest => NetworkType::Regtest, + } +} + +/// Validates a Zcash address for the `validateaddress` RPC. +/// +/// Pure address parsing over `network`; no chain data required, so both backends share +/// this implementation. +pub(crate) fn validate_address( + raw_address: String, + network: &zebra_chain::parameters::Network, +) -> ValidateAddressResponse { + use zcash_keys::address::Address; + use zcash_transparent::address::TransparentAddress; + + let Ok(address) = raw_address.parse::() else { + return ValidateAddressResponse::invalid(); + }; + + let address = match address.convert_if_network::
(address_network_type(network)) { + Ok(address) => address, + Err(err) => { + tracing::debug!(?err, "conversion error"); + return ValidateAddressResponse::invalid(); + } + }; + + match address { + Address::Transparent(taddr) => ValidateAddressResponse::new( + true, + Some(raw_address), + Some(matches!(taddr, TransparentAddress::ScriptHash(_))), + ), + _ => ValidateAddressResponse::invalid(), + } +} + +/// Validates a Zcash address for the deprecated `z_validateaddress` RPC. +/// +/// Pure address parsing over `network`; shared by both backends. +pub(crate) fn z_validate_address( + address: String, + network: &zebra_chain::parameters::Network, +) -> ZValidateAddressResponse { + use zcash_keys::address::Address; + use zcash_keys::encoding::AddressCodec as _; + use zcash_transparent::address::TransparentAddress; + + tracing::warn!("{}", Z_VALIDATE_DEPRECATION); + + let invalid = || { + ZValidateAddressResponse::Known(KnownZValidateAddress::Invalid( + InvalidZValidateAddress::new(), + )) + }; + + let Ok(parsed_address) = address.parse::() else { + return invalid(); + }; + + let converted_address = + match parsed_address.convert_if_network::
(address_network_type(network)) { + Ok(address) => address, + Err(err) => { + tracing::debug!(?err, "conversion error"); + return invalid(); + } + }; + + // Note: It could be the case that Zaino needs to support Sprout. For now, it's been disabled. + match converted_address { + Address::Transparent(TransparentAddress::PublicKeyHash(_)) => { + ZValidateAddressResponse::p2pkh(address) + } + Address::Transparent(TransparentAddress::ScriptHash(_)) => { + ZValidateAddressResponse::p2sh(address) + } + Address::Unified(u) => ZValidateAddressResponse::unified(u.encode(network)), + Address::Sapling(s) => { + let (diversifier, pk_d) = sapling_key_bytes(&s); + ZValidateAddressResponse::sapling( + s.encode(network), + Some(hex::encode(diversifier)), + Some(hex::encode(pk_d)), + ) + } + _ => invalid(), + } +} + +/// Extracts the diversifier and pk_d bytes from a validated Sapling +/// [`sapling_crypto::PaymentAddress`], returning pk_d in zcashd's big-endian byte order. +/// +/// # Deprecation +/// +/// See [`Z_VALIDATE_DEPRECATION`]. This function exists to support the `z_validateaddress` +/// RPC endpoint, which itself exists solely for zcashd compatibility. The pk_d bytes are +/// reversed from `sapling-crypto`'s native little-endian representation to match zcashd's +/// big-endian hex output. +/// +/// # Precondition +/// +/// The caller must have obtained `s` through `PaymentAddress::from_bytes` or equivalent +/// (e.g. `ZcashAddress::convert_if_network`), which guarantees the diversifier has a valid +/// `g_d()` and pk_d is a non-identity Jubjub subgroup point. No additional validation is +/// performed here. +/// +/// # Layout +/// +/// `PaymentAddress::to_bytes()` returns 43 bytes: `diversifier (11) || pk_d (32)`. +pub(crate) fn sapling_key_bytes(s: &sapling_crypto::PaymentAddress) -> ([u8; 11], [u8; 32]) { + let bytes = s.to_bytes(); + let diversifier: [u8; 11] = bytes[..11] + .try_into() + .expect("PaymentAddress::to_bytes always returns 43 bytes: diversifier is the first 11"); + let mut pk_d: [u8; 32] = bytes[11..] + .try_into() + .expect("PaymentAddress::to_bytes always returns 43 bytes: pk_d is the last 32"); + pk_d.reverse(); + (diversifier, pk_d) +} + +/// Shapes a `z_getsubtreesbyindex` JSON-RPC response from raw subtree roots. +/// +/// The `(root, end_height)` pairs from [`ChainIndex::get_subtree_roots`] are already in +/// the byte order the JSON-RPC uses (sapling `to_bytes`, orchard `to_repr` — zcashd's +/// `z_getsubtreesbyindex` does not reverse orchard subtree roots), so they are hex-encoded +/// as-is. Shared by both backends so the shaping lives in one place. +/// +/// [`ChainIndex::get_subtree_roots`]: crate::ChainIndex::get_subtree_roots +pub(crate) fn build_subtrees_by_index_response( + pool: String, + start_index: NoteCommitmentSubtreeIndex, + roots: Vec<([u8; 32], u32)>, +) -> GetSubtreesByIndexResponse { + use hex::ToHex as _; + + let subtrees = roots + .into_iter() + .map(|(root, end_height)| { + SubtreeRpcData { + root: root.encode_hex(), + end_height: Height(end_height), + } + .into() + }) + .collect(); + + GetSubtreesResponse { + pool, + start_index, + subtrees, + } + .into() +} + +/// Builds the gRPC [`TreeState`](zaino_proto::proto::service::TreeState) from a +/// `z_gettreestate` response: hex-encoded per-pool final states (the ironwood field is +/// the empty string below NU6.3 activation, matching lightwalletd behaviour). +fn tree_state_from_treestate_response( + network: String, + treestate_response: zebra_rpc::client::GetTreestateResponse, +) -> zaino_proto::proto::service::TreeState { + let sapling_tree = hex::encode( + treestate_response + .sapling() + .commitments() + .final_state() + .clone() + .unwrap_or_default(), + ); + let orchard_tree = hex::encode( + treestate_response + .orchard() + .commitments() + .final_state() + .clone() + .unwrap_or_default(), + ); + let ironwood_tree = treestate_response + .ironwood() + .clone() + .and_then(|treestate| treestate.commitments().final_state().clone()) + .map(hex::encode) + .unwrap_or_default(); + + zaino_proto::proto::service::TreeState { + network, + height: treestate_response.height().0 as u64, + hash: treestate_response.hash().to_string(), + time: treestate_response.time(), + sapling_tree, + orchard_tree, + ironwood_tree, + } +} + +/// Builds the `z_gettreestate` response from the per-pool treestates the chain index +/// reported. +/// +/// `Commitments::new(final_root, final_state)`: the note-commitment tree is the +/// `final_state`. The ironwood treestate is `Some` only from NU6.3 activation, so +/// pre-NU6.3 responses omit the field exactly as zebrad does. +fn build_treestate_response( + hash: zebra_chain::block::Hash, + height: zebra_chain::block::Height, + time: u32, + (sapling, orchard, ironwood): ( + Option, + Option, + Option, + ), +) -> zebra_rpc::client::GetTreestateResponse { + fn treestate( + pool: Option, + ) -> zebra_rpc::client::Treestate { + let (final_root, final_state) = match pool { + Some(pool) => (pool.final_root, Some(pool.final_state)), + None => (None, None), + }; + zebra_rpc::client::Treestate::new(zebra_rpc::client::Commitments::new( + final_root, + final_state, + )) + } + + let sprout_treestate = None; + let ironwood_treestate = ironwood.map(|pool| treestate(Some(pool))); + zebra_rpc::client::GetTreestateResponse::new( + hash, + height, + time, + sprout_treestate, + treestate(sapling), + treestate(orchard), + ironwood_treestate, + ) +} + +fn latest_network_upgrade( + upgrades: &indexmap::IndexMap< + zebra_rpc::methods::ConsensusBranchIdHex, + zebra_rpc::methods::NetworkUpgradeInfo, + >, +) -> Result<&zebra_rpc::methods::NetworkUpgradeInfo, tonic::Status> { + upgrades.last().map(|(_, upgrade)| upgrade).ok_or_else(|| { + tonic::Status::failed_precondition("validator returned no network upgrade metadata") + }) +} + +/// Maximum number of addresses a single `get_address_utxos` / `get_address_utxos_stream` +/// request may carry. +/// +/// The service resolves the full backend UTXO set before applying `max_entries` / +/// `start_height` (issue #974). A complete pushdown fix needs upstream interface changes +/// the caller-supplied entry cap cannot reach today, so until then this bounds the one +/// input the service controls locally: the address fan-out. It stops an unauthenticated +/// caller forcing an unbounded number of backend address lookups in a single request, and +/// is set well above realistic wallet usage. +/// +/// TODO: make this deployment-configurable rather than a fixed constant. +const UTXO_MAX_ADDRESSES: usize = 1000; + +/// Reject a `get_address_utxos` request whose address list exceeds [`UTXO_MAX_ADDRESSES`]. +/// +/// `max_entries` bounds the response size, not the backend work; this guard bounds the +/// address fan-out, the part the service can cap without upstream changes. +fn validate_utxo_address_count(count: usize) -> Result<(), tonic::Status> { + if count > UTXO_MAX_ADDRESSES { + return Err(tonic::Status::invalid_argument(format!( + "Error: too many addresses in request: {count} exceeds the maximum of {UTXO_MAX_ADDRESSES}." + ))); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn latest_network_upgrade_rejects_empty_metadata() { + let upgrades = indexmap::IndexMap::new(); + let err = super::latest_network_upgrade(&upgrades).expect_err("empty upgrades must fail"); + + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert_eq!( + err.message(), + "validator returned no network upgrade metadata" + ); + } + + #[test] + fn utxo_address_count_within_limit_is_accepted() { + assert!(super::validate_utxo_address_count(0).is_ok()); + assert!(super::validate_utxo_address_count(super::UTXO_MAX_ADDRESSES).is_ok()); + } + + #[test] + fn utxo_address_count_over_limit_is_rejected() { + let err = super::validate_utxo_address_count(super::UTXO_MAX_ADDRESSES + 1) + .expect_err("over-limit address count must fail"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + } + + #[test] + fn build_subtrees_by_index_response_hex_encodes_roots() { + let roots = vec![([0xabu8; 32], 100u32), ([0xcdu8; 32], 200u32)]; + let response = build_subtrees_by_index_response( + "orchard".to_string(), + NoteCommitmentSubtreeIndex(5), + roots, + ); + + assert_eq!(response.pool().as_str(), "orchard"); + assert_eq!(response.start_index(), NoteCommitmentSubtreeIndex(5)); + + let subtrees = response.subtrees(); + assert_eq!(subtrees.len(), 2); + assert_eq!(subtrees[0].root, hex::encode([0xabu8; 32])); + assert_eq!(subtrees[0].end_height, Height(100)); + assert_eq!(subtrees[1].root, hex::encode([0xcdu8; 32])); + assert_eq!(subtrees[1].end_height, Height(200)); + } + + /// Classifies the byte-level relationship between two slices. + #[derive(Debug, PartialEq)] + enum ByteRelation { + /// The slices are identical. + Equal, + /// `actual` fully byte-reversed equals `expected` (endian swap). + FullByteReversal, + /// Each byte's bits reversed maps `actual` to `expected`. + PerByteBitReversal, + /// Reversing bytes within 16-bit chunks maps `actual` to `expected`. + ChunkSwap16, + /// Reversing bytes within 32-bit chunks maps `actual` to `expected`. + ChunkSwap32, + /// Reversing bytes within 64-bit chunks maps `actual` to `expected`. + ChunkSwap64, + /// No recognized transformation. + Unrecognized, + } + + impl std::fmt::Display for ByteRelation { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Equal => write!(f, "equal"), + Self::FullByteReversal => write!(f, "full byte-reversal (endian swap)"), + Self::PerByteBitReversal => write!(f, "per-byte bit-reversal"), + Self::ChunkSwap16 => write!(f, "16-bit pairwise byte-swap"), + Self::ChunkSwap32 => write!(f, "32-bit chunk byte-reversal"), + Self::ChunkSwap64 => write!(f, "64-bit chunk byte-reversal"), + Self::Unrecognized => write!(f, "unrecognized mismatch"), + } + } + } + + /// Applies each candidate byte transformation to `actual` and returns + /// the first that produces `expected`, or [`ByteRelation::Unrecognized`]. + // `u32::is_multiple_of` is only stable from Rust 1.87; keep `% n == 0` for our older MSRV. + #[allow(clippy::manual_is_multiple_of)] + fn classify_byte_relation(actual: &[u8], expected: &[u8]) -> ByteRelation { + if actual == expected { + return ByteRelation::Equal; + } + + let chunk_swap = |size: usize| -> Vec { + actual + .chunks(size) + .flat_map(|c| c.iter().rev()) + .copied() + .collect() + }; + + let mut reversed = actual.to_vec(); + reversed.reverse(); + if reversed == expected { + return ByteRelation::FullByteReversal; + } + + let bit_reversed: Vec = actual.iter().map(|b| b.reverse_bits()).collect(); + if bit_reversed == expected { + return ByteRelation::PerByteBitReversal; + } + + if actual.len() % 2 == 0 && chunk_swap(2) == expected { + return ByteRelation::ChunkSwap16; + } + if actual.len() % 4 == 0 && chunk_swap(4) == expected { + return ByteRelation::ChunkSwap32; + } + if actual.len() % 8 == 0 && chunk_swap(8) == expected { + return ByteRelation::ChunkSwap64; + } + + ByteRelation::Unrecognized + } + + /// Verifies that our Sapling address parsing logic produces the same + /// diversifier and diversified transmission key (pk_d) hex strings as + /// zcashd's `z_validateaddress` RPC. + /// + /// # Guarantees + /// + /// - Exercises the production `sapling_key_bytes` function directly. + /// - The 11-byte diversifier matches the zcashd-derived test vector. + /// - The 32-byte pk_d (after the endian reversal inside `sapling_key_bytes`) + /// matches the zcashd-derived test vector. + /// - If the upstream serialization changes, the failure message + /// classifies the mismatch (endian swap, bit-reversal, chunk swap, + /// or unrecognized) to aid diagnosis. + /// + /// # Non-guarantees + /// + /// - Does not prove the test vector constants themselves are correct; + /// they were captured from zcashd and are trusted as ground truth. + /// - Does not exercise the full `z_validate_address` RPC path through + /// `StateService` — only the `sapling_key_bytes` extraction function. + /// - Does not verify behavior for malformed Sapling addresses or + /// addresses on other networks (mainnet, testnet). + #[test] + fn sapling_pk_d_byte_order_matches_test_vector() { + use crate::indexer::sapling_key_bytes; + use zcash_keys::address::Address; + use zcash_protocol::consensus::NetworkType; + + // Canonical source: live-tests/clientless/src/lib.rs::rpc::json_rpc + // Tracked for DRY consolidation: https://github.com/zingolabs/zaino/issues/988 + const SAPLING_ADDRESS: &str = "zregtestsapling1jalqhycwumq3unfxlzyzcktq3n478n82k2wacvl8gwfxk6ahshkxmtp2034qj28n7gl92ka5wca"; + const EXPECTED_DIVERSIFIER: &str = "977e0b930ee6c11e4d26f8"; + const EXPECTED_PK_D: &str = + "553ef2f328096a7c2aac6dec85b76b6b9243e733dc9db2eacce3eb8c60592c88"; + + let parsed: zcash_address::ZcashAddress = SAPLING_ADDRESS.parse().unwrap(); + let converted = parsed + .convert_if_network::
(NetworkType::Regtest) + .unwrap(); + + let Address::Sapling(s) = converted else { + panic!("expected Sapling address"); + }; + + let (diversifier, pk_d) = sapling_key_bytes(&s); + + let expected_diversifier = hex::decode(EXPECTED_DIVERSIFIER).unwrap(); + let expected_pk_d = hex::decode(EXPECTED_PK_D).unwrap(); + + // Diversifier + match classify_byte_relation(&diversifier, &expected_diversifier) { + ByteRelation::Equal => {} + relation => panic!( + "diversifier mismatch.\n relation: {relation}\n actual: {}\n expected: {}", + hex::encode(diversifier), + hex::encode(expected_diversifier), + ), + } + + // pk_d (sapling_key_bytes already applies the endian reversal) + match classify_byte_relation(&pk_d, &expected_pk_d) { + ByteRelation::Equal => {} + relation => panic!( + "pk_d mismatch — upstream serialization may have changed.\ + \n relation: {relation}\n actual: {}\n expected: {}", + hex::encode(pk_d), + hex::encode(expected_pk_d), + ), + } + } + + #[test] + fn classify_byte_relation_detects_known_transforms() { + let original = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08]; + + assert_eq!( + classify_byte_relation(&original, &original), + ByteRelation::Equal, + ); + + let mut reversed = original.to_vec(); + reversed.reverse(); + assert_eq!( + classify_byte_relation(&original, &reversed), + ByteRelation::FullByteReversal, + ); + + let bit_rev: Vec = original.iter().map(|b| b.reverse_bits()).collect(); + assert_eq!( + classify_byte_relation(&original, &bit_rev), + ByteRelation::PerByteBitReversal, + ); + + let swapped_16: Vec = original + .chunks(2) + .flat_map(|c| c.iter().rev()) + .copied() + .collect(); + assert_eq!( + classify_byte_relation(&original, &swapped_16), + ByteRelation::ChunkSwap16, + ); + + let garbage = [0xFF; 8]; + assert_eq!( + classify_byte_relation(&original, &garbage), + ByteRelation::Unrecognized, + ); + } +} diff --git a/packages/zaino-state/src/backends/fetch.rs b/packages/zaino-state/src/indexer/node_backed_indexer.rs similarity index 77% rename from packages/zaino-state/src/backends/fetch.rs rename to packages/zaino-state/src/indexer/node_backed_indexer.rs index aefbb97fb..0aeaa446d 100644 --- a/packages/zaino-state/src/backends/fetch.rs +++ b/packages/zaino-state/src/indexer/node_backed_indexer.rs @@ -24,8 +24,7 @@ use zebra_rpc::{ use zaino_fetch::{ chain::{transaction::FullTransaction, utils::ParseFromSlice}, jsonrpsee::{ - connector::{JsonRpSeeConnector, RpcError}, - raw_transaction::validate_raw_transaction_hex, + connector::RpcError, response::{ address_deltas::{GetAddressDeltasParams, GetAddressDeltasResponse}, block_deltas::BlockDeltas, @@ -34,9 +33,7 @@ use zaino_fetch::{ chain_tips::GetChainTipsResponse, mining_info::GetMiningInfoWire, peer_info::GetPeerInfo, - z_validate_address::{ - ZValidateAddressResponse, DEPRECATION_NOTICE as Z_VALIDATE_DEPRECATION, - }, + z_validate_address::ZValidateAddressResponse, GetMempoolInfoResponse, GetNetworkSolPsResponse, GetSpentInfoRequest, GetSpentInfoResponse, GetTxOutResponse, GetTxOutSetInfoResponse, }, @@ -60,9 +57,15 @@ use zaino_proto::proto::{ #[allow(deprecated)] use crate::{ chain_index::chain_tips_from_nonfinalized_snapshot, - chain_index::{source::ValidatorConnector, types}, - config::{DonationAddress, FetchServiceConfig}, - error::FetchServiceError, + chain_index::{ + source::{BlockchainSource, ValidatorConnector}, + types, + }, + config::{ + ChainIndexConfig, CommonBackendConfig, DonationAddress, NodeBackedIndexerServiceConfig, + ValidatorConnectionType, + }, + error::NodeBackedIndexerServiceError, indexer::{ handle_raw_transaction, IndexerSubscriber, LightWalletIndexer, ZcashIndexer, ZcashService, }, @@ -72,96 +75,123 @@ use crate::{ UtxoReplyStream, }, utils::{get_build_info, ServiceMetadata}, - BackendType, }; use crate::{ chain_index::{non_finalised_state::ChainIndexSnapshot, NonFinalizedSnapshot}, - ChainIndex, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, + ChainIndex, ChainIndexRpcExt, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, }; -/// Chain fetch service backed by Zcashd's JsonRPC engine. +/// A single node-backed chain-fetch + tx-submission service, generic over its +/// [`BlockchainSource`]. /// -/// This service is a central service, [`FetchServiceSubscriber`] should be created to fetch data. -/// This is done to enable large numbers of concurrent subscribers without significant slowdowns. +/// Replaces the former `FetchService` / `StateService` split: the JSON-RPC (`Rpc`) and +/// direct-`ReadStateService` (`Direct`) backends are now one type selected at runtime +/// via [`NodeBackedIndexerServiceConfig`]. Production instantiates the default +/// `Source = ValidatorConnector` (which itself carries the `Rpc`/`Direct` arm); tests +/// may instantiate over a mock source. /// -/// NOTE: We currently do not implement clone for chain fetch services as this service is responsible for maintaining and closing its child processes. -/// ServiceSubscribers are used to create separate chain fetch processes while allowing central state processes to be managed in a single place. -/// If we want the ability to clone Service all JoinHandle's should be converted to Arc\. +/// This service is a central service — create a [`NodeBackedIndexerServiceSubscriber`] +/// (via [`ZcashService::get_subscriber`]) to fetch data. This lets large numbers of +/// concurrent subscribers run without contending on the single central process. +/// +/// NOTE: We do not implement `Clone` for the central service: it owns and closes its +/// child processes. Subscribers are the clone-safe read handles. #[derive(Debug)] -#[deprecated = "Will be eventually replaced by `BlockchainSource`"] -pub struct FetchService { - /// JsonRPC Client. - /// - /// NOTE: DEPRCATED, USE INDEXER OR VALIDATOR_CONNECTOR. - fetcher: JsonRpSeeConnector, +pub struct NodeBackedIndexerService { /// Core indexer. - indexer: NodeBackedChainIndex, + indexer: NodeBackedChainIndex, /// Service metadata. data: ServiceMetadata, - - /// StateService config data. - #[allow(deprecated)] - config: FetchServiceConfig, + /// Connection-independent config (validator RPC, storage, network, ...). + config: CommonBackendConfig, } -#[allow(deprecated)] -impl Status for FetchService { +impl Status for NodeBackedIndexerService { fn status(&self) -> StatusType { self.indexer.status() } } -#[allow(deprecated)] -impl ZcashService for FetchService { - const BACKEND_TYPE: BackendType = BackendType::Fetch; +impl NodeBackedIndexerService { + /// Tears down the indexer (sync loop, finalised DB, mempool, and any source-owned + /// syncer task) from a synchronous context. Shared by [`ZcashService::close`] and + /// [`Drop`]. + /// + /// `block_in_place` is only legal on a multi-thread runtime; on a + /// current-thread runtime or a thread with no runtime it panics, and a + /// panic inside `Drop` during unwind aborts the process. Those contexts + /// fall back to the synchronous best-effort teardown, which skips only + /// the finalised DB's async shutdown step (the DB's own `Drop` still + /// releases its resources). + fn shutdown_blocking(&self) { + use tokio::runtime::{Handle, RuntimeFlavor}; + + match Handle::try_current() { + Ok(handle) if handle.runtime_flavor() == RuntimeFlavor::MultiThread => { + tokio::task::block_in_place(|| { + handle.block_on(async { + let _ = self.indexer.shutdown().await; + }); + }); + } + _ => self.indexer.shutdown_sync_best_effort(), + } + } +} - type Subscriber = FetchServiceSubscriber; - type Config = FetchServiceConfig; +impl ZcashService for NodeBackedIndexerService { + type Subscriber = NodeBackedIndexerServiceSubscriber; + type Config = NodeBackedIndexerServiceConfig; - /// Initializes a new FetchService instance and starts sync process. - #[instrument(name = "FetchService::spawn", skip(config), fields(network = %config.common.network))] - async fn spawn(config: FetchServiceConfig) -> Result { + /// Initializes a new [`NodeBackedIndexerService`] and starts its sync process. + #[instrument(name = "NodeBackedIndexerService::spawn", skip(config), fields(network = %config.common.network))] + async fn spawn( + config: NodeBackedIndexerServiceConfig, + ) -> Result { info!( rpc_address = %config.common.validator_rpc_address, network = %config.common.network, - "Launching Fetch Service" + "Launching NodeBackedIndexerService" ); - let fetcher = JsonRpSeeConnector::new_from_config_parts( - &config.common.validator_rpc_address, - config.common.validator_rpc_user.clone(), - config.common.validator_rpc_password.clone(), - config.common.validator_cookie_path.clone(), - ) - .await?; + // Select the validator connection from config; both arms return the built source, + // the validator `getinfo` used for service metadata, and the runtime network + // (activation schedule adopted from the validator at first contact, zaino#1076). + let (source, zebra_build_data, network) = match &config.connection { + ValidatorConnectionType::Rpc => ValidatorConnector::spawn_fetch(&config.common).await, + ValidatorConnectionType::Direct(direct) => { + ValidatorConnector::spawn_state(&config.common, direct).await + } + } + .map_err(|error| NodeBackedIndexerServiceError::Critical(error.to_string()))?; - let zebra_build_data = fetcher.get_info().await?; let data = ServiceMetadata::new( get_build_info(config.common.indexer_version.clone()), - config.common.network.to_zebra_network(), + network.clone(), zebra_build_data.build, zebra_build_data.subversion, ); info!(build = %data.zebra_build(), subversion = %data.zebra_subversion(), "Connected to Zcash node"); - let source = ValidatorConnector::Fetch(fetcher.clone()); - let indexer = NodeBackedChainIndex::new(source, config.clone().into()) - .await - .unwrap(); + let indexer = NodeBackedChainIndex::new( + source, + ChainIndexConfig::from_backend_config(&config.common, network), + ) + .await + .map_err(|error| NodeBackedIndexerServiceError::Critical(error.to_string()))?; - let fetch_service = Self { - fetcher, + let service = Self { indexer, data, - config, + config: config.common, }; // wait for sync to complete, return error on sync fail. loop { - match fetch_service.status() { + match service.status() { StatusType::Ready | StatusType::Closing => break, StatusType::CriticalError => { - return Err(FetchServiceError::Critical( + return Err(NodeBackedIndexerServiceError::Critical( "ChainIndex initial sync failed, check full log for details.".to_string(), )); } @@ -171,78 +201,210 @@ impl ZcashService for FetchService { } } - Ok(fetch_service) + Ok(service) } - /// Returns a [`FetchServiceSubscriber`]. - fn get_subscriber(&self) -> IndexerSubscriber { - IndexerSubscriber::new(FetchServiceSubscriber { - fetcher: self.fetcher.clone(), + /// Returns a [`NodeBackedIndexerServiceSubscriber`]. + fn get_subscriber( + &self, + ) -> IndexerSubscriber> { + IndexerSubscriber::new(NodeBackedIndexerServiceSubscriber { indexer: self.indexer.subscriber(), data: self.data.clone(), config: self.config.clone(), }) } - /// Shuts down the StateService. + /// Shuts down the service, tearing down its indexer (sync loop, finalised DB, + /// mempool, and any source-owned syncer task). fn close(&mut self) { - tokio::task::block_in_place(|| { - tokio::runtime::Handle::current().block_on(async { - let _ = self.indexer.shutdown().await; - }); - }); + self.shutdown_blocking(); } } -#[allow(deprecated)] -impl Drop for FetchService { +impl Drop for NodeBackedIndexerService { fn drop(&mut self) { - self.close() + self.shutdown_blocking(); } } -/// A fetch service subscriber. -/// -/// Subscribers should be +/// A clone-safe, read-only subscriber to a [`NodeBackedIndexerService`]. #[derive(Debug, Clone)] -#[allow(deprecated)] -pub struct FetchServiceSubscriber { - /// JsonRPC Client. - /// - /// NOTE: DEPRCATED, USE INDEXER OR VALIDATOR_CONNECTOR. - fetcher: JsonRpSeeConnector, +pub struct NodeBackedIndexerServiceSubscriber { /// Core indexer. - pub indexer: NodeBackedChainIndexSubscriber, + pub indexer: NodeBackedChainIndexSubscriber, /// Service metadata. pub data: ServiceMetadata, - /// StateService config data. - #[allow(deprecated)] - config: FetchServiceConfig, + /// Connection-independent config (validator RPC, storage, network, ...). + config: CommonBackendConfig, } -impl Status for FetchServiceSubscriber { +impl Status for NodeBackedIndexerServiceSubscriber { fn status(&self) -> StatusType { self.indexer.status() } } -impl FetchServiceSubscriber { +impl NodeBackedIndexerServiceSubscriber { /// Fetches the current status #[deprecated(note = "Use the Status trait method instead")] pub fn get_status(&self) -> StatusType { self.indexer.status() } - /// Returns the network type running. - #[allow(deprecated)] - pub fn network(&self) -> zaino_common::Network { - self.config.common.network + /// Returns the runtime network, carrying the activation schedule + /// adopted from the validator (zaino#1076). + pub fn network(&self) -> zebra_chain::parameters::Network { + self.data.network() } } -impl ZcashIndexer for FetchServiceSubscriber { - #[allow(deprecated)] - type Error = FetchServiceError; +/// `getchaintips` served from the non-finalised snapshot when it exists, +/// falling back to the validator's own response during the initial +/// finalised-state build — matching both pre-merge backends, which proxied +/// the validator for that window instead of erroring. +pub(crate) async fn chain_tips_for_snapshot( + snapshot: &ChainIndexSnapshot, + source: &Source, +) -> Result { + match snapshot.get_nfs_snapshot() { + Some(non_finalized_snapshot) => Ok(chain_tips_from_nonfinalized_snapshot( + non_finalized_snapshot, + )), + None => Ok(source + .get_chain_tips() + .await + .map_err(crate::error::ChainIndexError::backing_validator)?), + } +} + +/// Placeholder metadata and config for test-only service construction. Takes the +/// runtime (zebra) network directly — tests are their own chain, so they stand in +/// for the validator the production path adopts the schedule from. +#[cfg(test)] +fn test_service_parts( + network: zebra_chain::parameters::Network, +) -> (ServiceMetadata, CommonBackendConfig) { + let network_kind = zaino_common::Network::from(network.clone()); + ( + ServiceMetadata::new( + get_build_info("test".to_string()), + network, + "test-build".to_string(), + "test-subversion".to_string(), + ), + CommonBackendConfig::new( + "127.0.0.1:0".to_string(), + None, + None, + None, + zaino_common::ServiceConfig::default(), + zaino_common::StorageConfig::default(), + true, + network_kind, + None, + ), + ) +} + +#[cfg(test)] +impl NodeBackedIndexerService { + /// Wraps a chain index in a service for tests, with placeholder + /// metadata/config. Lets unit tests exercise the service lifecycle over a + /// mock source (no real validator). Production builds go through + /// [`ZcashService::spawn`]. + pub(crate) fn new_for_test( + indexer: NodeBackedChainIndex, + network: zebra_chain::parameters::Network, + ) -> Self { + let (data, config) = test_service_parts(network); + Self { + data, + config, + indexer, + } + } +} + +#[cfg(test)] +impl NodeBackedIndexerServiceSubscriber { + /// Wraps a chain-index subscriber in a service subscriber for tests, with placeholder + /// metadata/config. Lets unit tests drive the service RPC layer over a mock source + /// (no real validator). Production builds go through [`ZcashService::get_subscriber`]. + pub(crate) fn new_for_test( + indexer: NodeBackedChainIndexSubscriber, + network: zebra_chain::parameters::Network, + ) -> Self { + let (data, config) = test_service_parts(network); + Self { + data, + config, + indexer, + } + } +} + +/// A subscriber to any chaintip updates. +#[derive(Clone)] +pub struct ChainTipSubscriber { + monitor: zebra_state::ChainTipChange, +} + +impl ChainTipSubscriber { + /// Waits until the tip hash has changed (relative to the last time this method + /// was called), then returns the best tip's block hash. + pub async fn next_tip_hash( + &mut self, + ) -> Result { + self.monitor + .wait_for_tip_change() + .await + .map(|tip| tip.best_tip_hash()) + } +} + +impl NodeBackedIndexerServiceSubscriber { + /// A subscriber to chain-tip updates, when the backing source exposes a + /// local tip-change stream. `Some` only on the `Direct` connection; the + /// `Rpc` connection (and any other stream-less source) observes tips by + /// polling the validator and yields `None`. + pub fn chaintip_update_subscriber(&self) -> Option { + Some(ChainTipSubscriber { + monitor: self.indexer.source().chain_tip_change()?, + }) + } +} + +/// Methods available only on the production (`ValidatorConnector`-backed) subscriber: +/// the `Direct` connection owns a `ReadStateService` that the generic source +/// abstraction does not expose. +impl NodeBackedIndexerServiceSubscriber { + /// The backing Zebra [`zebra_state::ReadStateService`] (`Direct` connection only). + /// + /// Test-only escape hatch: live tests recompute expected chain data directly off the + /// `ReadStateService`. Production code goes through the `ChainIndex` API. + #[cfg(feature = "test_dependencies")] + pub fn read_state_service(&self) -> zebra_state::ReadStateService { + self.indexer + .source() + .read_state_service() + .expect("read_state_service requires the Direct connection") + .clone() + } + + /// The indexer's mempool subscriber. + /// + /// Test-only escape hatch: live tests recompute expected `getmempoolinfo` values + /// directly off the mempool's entries. Production code goes through the `ChainIndex` + /// mempool API. + #[cfg(feature = "test_dependencies")] + pub fn mempool(&self) -> &crate::chain_index::mempool::MempoolSubscriber { + self.indexer.mempool_subscriber() + } +} + +impl ZcashIndexer for NodeBackedIndexerServiceSubscriber { + type Error = NodeBackedIndexerServiceError; /// Returns information about all changes to the given transparent addresses within the given inclusive block-height range. /// @@ -282,7 +444,7 @@ impl ZcashIndexer for FetchServiceSubscriber { /// in Zebra's [`GetInfo`]. Zebra uses the field names and formats from the /// [zcashd code](https://github.com/zcash/zcash/blob/v4.6.0-1/src/rpc/misc.cpp#L86-L87). async fn get_info(&self) -> Result { - Ok(self.fetcher.get_info().await?.into()) + Ok(self.indexer.get_info().await?) } /// Returns blockchain state information, as a [`GetBlockchainInfoResponse`] JSON struct. @@ -296,18 +458,7 @@ impl ZcashIndexer for FetchServiceSubscriber { /// Some fields from the zcashd reference are missing from Zebra's [`GetBlockchainInfoResponse`]. It only contains the fields /// [required for lightwalletd support.](https://github.com/zcash/lightwalletd/blob/v0.4.9/common/common.go#L72-L89) async fn get_blockchain_info(&self) -> Result { - self.fetcher - .get_blockchain_info() - .await? - .try_into() - .map_err(|_e| { - #[allow(deprecated)] - FetchServiceError::SerializationError( - zebra_chain::serialization::SerializationError::Parse( - "chainwork not hex-encoded integer", - ), - ) - }) + Ok(self.indexer.get_blockchain_info().await?) } /// Returns details on the active state of the TX memory pool. @@ -324,7 +475,7 @@ impl ZcashIndexer for FetchServiceSubscriber { } async fn get_peer_info(&self) -> Result { - Ok(self.fetcher.get_peer_info().await?) + Ok(self.indexer.get_peer_info().await?) } /// Returns the proof-of-work difficulty as a multiple of the minimum difficulty. @@ -333,11 +484,11 @@ impl ZcashIndexer for FetchServiceSubscriber { /// method: post /// tags: blockchain async fn get_difficulty(&self) -> Result { - Ok(self.fetcher.get_difficulty().await?.0) + Ok(self.indexer.get_difficulty().await?) } async fn get_block_subsidy(&self, height: u32) -> Result { - Ok(self.fetcher.get_block_subsidy(height).await?) + Ok(self.indexer.get_block_subsidy(height).await?) } /// Returns the total balance of a provided `addresses` in an [`AddressBalance`] instance. @@ -388,12 +539,10 @@ impl ZcashIndexer for FetchServiceSubscriber { &self, raw_transaction_hex: String, ) -> Result { - validate_raw_transaction_hex(&raw_transaction_hex)?; Ok(self - .fetcher + .indexer .send_raw_transaction(raw_transaction_hex) - .await? - .into()) + .await?) } /// Returns the requested block by hash or height, as a [`GetBlock`] JSON string. @@ -425,11 +574,7 @@ impl ZcashIndexer for FetchServiceSubscriber { hash_or_height: String, verbosity: Option, ) -> Result { - Ok(self - .fetcher - .get_block(hash_or_height, verbosity) - .await? - .try_into()?) + Ok(self.indexer.z_get_block(hash_or_height, verbosity).await?) } /// Returns information about the given block and its transactions. @@ -440,7 +585,7 @@ impl ZcashIndexer for FetchServiceSubscriber { /// /// Note: This method has only been implemented in `zcashd`. Zebra has no intention of supporting it. async fn get_block_deltas(&self, hash: String) -> Result { - Ok(self.fetcher.get_block_deltas(hash).await?) + Ok(self.indexer.get_block_deltas(hash).await?) } async fn get_block_header( @@ -448,11 +593,11 @@ impl ZcashIndexer for FetchServiceSubscriber { hash: String, verbose: bool, ) -> Result { - Ok(self.fetcher.get_block_header(hash, verbose).await?) + Ok(self.indexer.get_block_header(hash, verbose).await?) } async fn get_mining_info(&self) -> Result { - Ok(self.fetcher.get_mining_info().await?) + Ok(self.indexer.get_mining_info().await?) } /// Returns statistics about the unspent transaction output set. @@ -484,7 +629,9 @@ impl ZcashIndexer for FetchServiceSubscriber { /// [The function in rpc/blockchain.cpp](https://github.com/zcash/zcash/blob/654a8be2274aa98144c80c1ac459400eaf0eacbe/src/rpc/blockchain.cpp#L325) /// where `return chainActive.Tip()->GetBlockHash().GetHex();` is the [return expression](https://github.com/zcash/zcash/blob/654a8be2274aa98144c80c1ac459400eaf0eacbe/src/rpc/blockchain.cpp#L339)returning a `std::string` async fn get_best_blockhash(&self) -> Result { - Ok(self.fetcher.get_best_blockhash().await?.into()) + let snapshot = self.indexer.snapshot_nonfinalized_state().await?; + let tip = self.indexer.best_chaintip(&snapshot).await?; + Ok(GetBlockHashResponse::new(tip.hash.into())) } /// Returns the current block count in the best valid block chain. @@ -493,18 +640,15 @@ impl ZcashIndexer for FetchServiceSubscriber { /// method: post /// tags: blockchain async fn get_block_count(&self) -> Result { - Ok(self.fetcher.get_block_count().await?.into()) + let snapshot = self.indexer.snapshot_nonfinalized_state().await?; + let tip = self.indexer.best_chaintip(&snapshot).await?; + Ok(tip.height.into()) } + #[allow(deprecated)] async fn get_chain_tips(&self) -> Result { let snapshot = self.indexer.snapshot_nonfinalized_state().await?; - let Some(non_finalized_snapshot) = snapshot.get_nfs_snapshot() else { - return Ok(self.fetcher.get_chain_tips().await?); - }; - - Ok(chain_tips_from_nonfinalized_snapshot( - non_finalized_snapshot, - )) + chain_tips_for_snapshot(&snapshot, self.indexer.source()).await } /// Return information about the given Zcash address. @@ -519,7 +663,9 @@ impl ZcashIndexer for FetchServiceSubscriber { &self, address: String, ) -> Result { - Ok(self.fetcher.validate_address(address).await?) + #[allow(deprecated)] + let network = self.data.network(); + Ok(crate::indexer::validate_address(address, &network)) } #[allow(deprecated)] @@ -527,8 +673,9 @@ impl ZcashIndexer for FetchServiceSubscriber { &self, address: String, ) -> Result { - tracing::warn!("{}", Z_VALIDATE_DEPRECATION); - Ok(self.fetcher.z_validate_address(address).await?) + #[allow(deprecated)] + let network = self.data.network(); + Ok(crate::indexer::z_validate_address(address, &network)) } /// Returns all transaction ids in the memory pool, as a JSON array. @@ -567,7 +714,7 @@ impl ZcashIndexer for FetchServiceSubscriber { /// NOTE: This method currently has to fetch data from 2 places (get_treestate and get_indexed_block_by_*), /// If `ValidatorConnector::GetTreeState` was updated to return the additional information /// required, this second call could be removed, improving the performance of this method. - // Pre-existing lint: `FetchServiceError` is a large error type; returning it by value here is + // Pre-existing lint: `NodeBackedIndexerServiceError` is a large error type; returning it by value here is // flagged by `result_large_err`. Suppressed to satisfy `-D warnings` without an invasive // boxing refactor of the shared error enum. #[allow(clippy::result_large_err)] @@ -587,7 +734,7 @@ impl ZcashIndexer for FetchServiceSubscriber { .await? .ok_or( #[allow(deprecated)] - FetchServiceError::RpcError(RpcError::new_from_legacycode( + NodeBackedIndexerServiceError::RpcError(RpcError::new_from_legacycode( zebra_rpc::server::error::LegacyCode::InvalidParameter, "Failed to fetch block data.", )), @@ -598,7 +745,7 @@ impl ZcashIndexer for FetchServiceSubscriber { .await? .ok_or( #[allow(deprecated)] - FetchServiceError::RpcError(RpcError::new_from_legacycode( + NodeBackedIndexerServiceError::RpcError(RpcError::new_from_legacycode( zebra_rpc::server::error::LegacyCode::InvalidParameter, "Failed to fetch block data.", )), @@ -608,7 +755,7 @@ impl ZcashIndexer for FetchServiceSubscriber { let treestates = self.indexer.get_treestate(block_data.hash()).await?; let time: u32 = block_data.data().time().try_into().map_err(|_error| { #[allow(deprecated)] - FetchServiceError::RpcError(RpcError::new_from_legacycode( + NodeBackedIndexerServiceError::RpcError(RpcError::new_from_legacycode( zebra_rpc::server::error::LegacyCode::InvalidParameter, "Block time is out of range for u32.", )) @@ -636,25 +783,10 @@ impl ZcashIndexer for FetchServiceSubscriber { return local_result; } - self.fetcher - .get_treestate(fallback_hash_or_height) - .await - .map_err(|_error| { - #[allow(deprecated)] - FetchServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to fetch treestate.", - )) - }) - .and_then(|treestate| { - treestate.try_into().map_err(|_error| { - #[allow(deprecated)] - FetchServiceError::RpcError(RpcError::new_from_legacycode( - zebra_rpc::server::error::LegacyCode::InvalidParameter, - "Failed to parse treestate.", - )) - }) - }) + Ok(self + .indexer + .get_treestate_by_id(fallback_hash_or_height) + .await?) } /// Returns information about a range of Sapling or Orchard subtrees. @@ -675,17 +807,36 @@ impl ZcashIndexer for FetchServiceSubscriber { /// starting at the chain tip. This RPC will return an empty list if the `start_index` subtree /// exists, but has not been rebuilt yet. This matches `zcashd`'s behaviour when subtrees aren't /// available yet. (But `zcashd` does its rebuild before syncing any blocks.) + #[allow(deprecated)] async fn z_get_subtrees_by_index( &self, pool: String, start_index: NoteCommitmentSubtreeIndex, limit: Option, ) -> Result { - Ok(self - .fetcher - .get_subtrees_by_index(pool, start_index.0, limit.map(|limit_index| limit_index.0)) - .await? - .into()) + let shielded_pool = match pool.as_str() { + "sapling" => crate::chain_index::ShieldedPool::Sapling, + "orchard" => crate::chain_index::ShieldedPool::Orchard, + otherwise => { + return Err(NodeBackedIndexerServiceError::RpcError( + RpcError::new_from_legacycode( + zebra_rpc::server::error::LegacyCode::Misc, + format!( + "invalid pool name \"{otherwise}\", must be \"sapling\" or \"orchard\"" + ), + ), + )) + } + }; + let roots = self + .indexer + .get_subtree_roots(shielded_pool, start_index.0, limit.map(|index| index.0)) + .await?; + Ok(crate::indexer::build_subtrees_by_index_response( + pool, + start_index, + roots, + )) } /// Returns the raw transaction data, as a [`GetRawTransaction`] JSON string or structure. @@ -714,7 +865,7 @@ impl ZcashIndexer for FetchServiceSubscriber { ) -> Result { #[allow(deprecated)] let txid = types::TransactionHash::from_hex(&txid_hex).map_err(|error| { - FetchServiceError::RpcError(RpcError::new_from_legacycode( + NodeBackedIndexerServiceError::RpcError(RpcError::new_from_legacycode( zebra_rpc::server::error::LegacyCode::InvalidAddressOrKey, error.to_string(), )) @@ -722,7 +873,7 @@ impl ZcashIndexer for FetchServiceSubscriber { #[allow(deprecated)] let not_found_error = || { - FetchServiceError::RpcError(RpcError::new_from_legacycode( + NodeBackedIndexerServiceError::RpcError(RpcError::new_from_legacycode( zebra_rpc::server::error::LegacyCode::InvalidAddressOrKey, "No such mempool or main chain transaction", )) @@ -778,7 +929,7 @@ impl ZcashIndexer for FetchServiceSubscriber { height, confirmations, #[allow(deprecated)] - &self.config.common.network.to_zebra_network(), + &self.data.network(), None, block_hash, in_best_chain, @@ -798,24 +949,19 @@ impl ZcashIndexer for FetchServiceSubscriber { n: u32, include_mempool: Option, ) -> Result { - Ok(self.fetcher.get_tx_out(txid, n, include_mempool).await?) + Ok(self.indexer.get_tx_out(txid, n, include_mempool).await?) } async fn get_spent_info( &self, request: GetSpentInfoRequest, ) -> Result { - Ok(self.fetcher.get_spent_info(request).await?) + Ok(self.indexer.get_spent_info(request).await?) } async fn chain_height(&self) -> Result { - Ok(Height( - self.indexer - .snapshot_nonfinalized_state() - .await? - .max_serviceable_height() - .0, - )) + let snapshot = self.indexer.snapshot_nonfinalized_state().await?; + Ok(self.indexer.best_chaintip(&snapshot).await?.height.into()) } /// Returns the transaction ids made by the provided transparent addresses. /// @@ -886,12 +1032,12 @@ impl ZcashIndexer for FetchServiceSubscriber { blocks: Option, height: Option, ) -> Result { - Ok(self.fetcher.get_network_sol_ps(blocks, height).await?) + Ok(self.indexer.get_network_sol_ps(blocks, height).await?) } } #[allow(deprecated)] -impl LightWalletIndexer for FetchServiceSubscriber { +impl LightWalletIndexer for NodeBackedIndexerServiceSubscriber { /// Return the height of the tip of the best chain async fn get_latest_block(&self) -> Result { match self.indexer.snapshot_nonfinalized_state().await? { @@ -902,7 +1048,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // TODO: This probably shouldn't be an error. // this is an improvement over previous behaviour of reporting // the genesis block - Err(FetchServiceError::UnavailableNotSyncedEnough) + Err(NodeBackedIndexerServiceError::UnavailableNotSyncedEnough) } } // dbg!(&tip); @@ -911,7 +1057,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { /// Return the compact block corresponding to the given block identifier async fn get_block(&self, request: BlockId) -> Result { let hash_or_height = blockid_to_hashorheight(request).ok_or( - FetchServiceError::TonicStatusError(tonic::Status::invalid_argument( + NodeBackedIndexerServiceError::TonicStatusError(tonic::Status::invalid_argument( "Error: Invalid hash and/or height out of range. Failed to convert to u32.", )), )?; @@ -923,12 +1069,12 @@ impl LightWalletIndexer for FetchServiceSubscriber { match self.indexer.get_block_height(&snapshot, hash.into()).await { Ok(Some(height)) => height.0, Ok(None) => { - return Err(FetchServiceError::TonicStatusError(tonic::Status::invalid_argument( + return Err(NodeBackedIndexerServiceError::TonicStatusError(tonic::Status::invalid_argument( "Error: Invalid hash and/or height out of range. Hash not founf in chain", ))); } Err(_e) => { - return Err(FetchServiceError::TonicStatusError( + return Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::internal("Error: Internal db error."), )); } @@ -940,7 +1086,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // TODO: This probably shouldn't be an error. // this is an improvement over previous behaviour of // acting as if we are only synced to the genesis block - return Err(FetchServiceError::UnavailableNotSyncedEnough); + return Err(NodeBackedIndexerServiceError::UnavailableNotSyncedEnough); }; match self @@ -956,32 +1102,38 @@ impl LightWalletIndexer for FetchServiceSubscriber { Ok(None) => { let chain_height = non_finalized_snapshot.best_tip.height.0; match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - FetchServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ + HashOrHeight::Height(Height(height)) if height >= chain_height => { + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::out_of_range(format!( + "Error: Height out of range [{hash_or_height}]. Height requested \ is greater than the best chain tip [{chain_height}].", - ))), - ), - _otherwise => Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Failed to retrieve block from state.", - ))), + )), + )) + } + _otherwise => Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown("Error: Failed to retrieve block from state."), + )), } } Err(e) => { let chain_height = non_finalized_snapshot.best_tip.height.0; match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - FetchServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ + HashOrHeight::Height(Height(height)) if height >= chain_height => { + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::out_of_range(format!( + "Error: Height out of range [{hash_or_height}]. Height requested \ is greater than the best chain tip [{chain_height}].", - ))), - ), + )), + )) + } _otherwise => // TODO: Hide server error from clients before release. Currently useful for dev purposes. { - Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - format!("Error: Failed to retrieve block from node. Server Error: {e}",), - ))) + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown(format!( + "Error: Failed to retrieve block from node. Server Error: {e}", + )), + )) } } } @@ -993,7 +1145,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { /// NOTE: Currently this only returns Orchard nullifiers to follow Lightwalletd functionality but Sapling could be added if required by wallets. async fn get_block_nullifiers(&self, request: BlockId) -> Result { let hash_or_height = blockid_to_hashorheight(request).ok_or( - FetchServiceError::TonicStatusError(tonic::Status::invalid_argument( + NodeBackedIndexerServiceError::TonicStatusError(tonic::Status::invalid_argument( "Error: Invalid hash and/or height out of range. Failed to convert to u32.", )), )?; @@ -1004,12 +1156,12 @@ impl LightWalletIndexer for FetchServiceSubscriber { match self.indexer.get_block_height(&snapshot, hash.into()).await { Ok(Some(height)) => height.0, Ok(None) => { - return Err(FetchServiceError::TonicStatusError(tonic::Status::invalid_argument( + return Err(NodeBackedIndexerServiceError::TonicStatusError(tonic::Status::invalid_argument( "Error: Invalid hash and/or height out of range. Hash not founf in chain", ))); } Err(_e) => { - return Err(FetchServiceError::TonicStatusError( + return Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::internal("Error: Internal db error."), )); } @@ -1020,7 +1172,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // TODO: This probably shouldn't be an error. // this is an improvement over previous behaviour of // acting as if we are only synced to the genesis block - return Err(FetchServiceError::UnavailableNotSyncedEnough); + return Err(NodeBackedIndexerServiceError::UnavailableNotSyncedEnough); }; match self .indexer @@ -1035,40 +1187,44 @@ impl LightWalletIndexer for FetchServiceSubscriber { Ok(None) => { let chain_height = non_finalized_snapshot.best_tip.height.0; match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - FetchServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ + HashOrHeight::Height(Height(height)) if height >= chain_height => { + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::out_of_range(format!( + "Error: Height out of range [{hash_or_height}]. Height requested \ is greater than the best chain tip [{chain_height}].", - ))), - ), + )), + )) + } HashOrHeight::Height(height) if height > self.data.network().sapling_activation_height() => { - Err(FetchServiceError::TonicStatusError( + Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::out_of_range(format!( "Error: Height out of range [{hash_or_height}]. Height requested \ is below sapling activation height [{chain_height}].", )), )) } - _otherwise => Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Failed to retrieve block from state.", - ))), + _otherwise => Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown("Error: Failed to retrieve block from state."), + )), } } Err(e) => { let chain_height = non_finalized_snapshot.best_tip.height.0; match hash_or_height { - HashOrHeight::Height(Height(height)) if height >= chain_height => Err( - FetchServiceError::TonicStatusError(tonic::Status::out_of_range(format!( - "Error: Height out of range [{hash_or_height}]. Height requested \ + HashOrHeight::Height(Height(height)) if height >= chain_height => { + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::out_of_range(format!( + "Error: Height out of range [{hash_or_height}]. Height requested \ is greater than the best chain tip [{chain_height}].", - ))), - ), + )), + )) + } HashOrHeight::Height(height) if height > self.data.network().sapling_activation_height() => { - Err(FetchServiceError::TonicStatusError( + Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::out_of_range(format!( "Error: Height out of range [{hash_or_height}]. Height requested \ is below sapling activation height [{chain_height}].", @@ -1078,9 +1234,11 @@ impl LightWalletIndexer for FetchServiceSubscriber { _otherwise => // TODO: Hide server error from clients before release. Currently useful for dev purposes. { - Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - format!("Error: Failed to retrieve block from node. Server Error: {e}",), - ))) + Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown(format!( + "Error: Failed to retrieve block from node. Server Error: {e}", + )), + )) } } } @@ -1094,24 +1252,20 @@ impl LightWalletIndexer for FetchServiceSubscriber { request: BlockRange, ) -> Result { let validated_request = ValidatedBlockRangeRequest::new_from_block_range(&request) - .map_err(FetchServiceError::from)?; + .map_err(NodeBackedIndexerServiceError::from)?; let pool_type_filter = PoolTypeFilter::new_from_pool_types(&validated_request.pool_types()) .map_err(GetBlockRangeError::PoolTypeArgumentError) - .map_err(FetchServiceError::from)?; + .map_err(NodeBackedIndexerServiceError::from)?; // Note conversion here is safe due to the use of [`ValidatedBlockRangeRequest::new_from_block_range`] let start = validated_request.start() as u32; let end = validated_request.end() as u32; - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - let snapshot = fetch_service_clone - .indexer - .snapshot_nonfinalized_state() - .await?; + let service_clone = self.clone(); + let service_timeout = self.config.service.timeout; + let (channel_tx, channel_rx) = mpsc::channel(self.config.service.channel_size as usize); + let snapshot = service_clone.indexer.snapshot_nonfinalized_state().await?; tokio::spawn(async move { let timeout_result = timeout( @@ -1134,7 +1288,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // Use the snapshot tip directly, as this function doesn't support passthrough let chain_height = non_finalized_snapshot.best_tip.height.0; - match fetch_service_clone + match service_clone .indexer .get_compact_block_stream( &snapshot, @@ -1227,24 +1381,20 @@ impl LightWalletIndexer for FetchServiceSubscriber { request: BlockRange, ) -> Result { let validated_request = ValidatedBlockRangeRequest::new_from_block_range(&request) - .map_err(FetchServiceError::from)?; + .map_err(NodeBackedIndexerServiceError::from)?; let pool_type_filter = PoolTypeFilter::new_from_pool_types(&validated_request.pool_types()) .map_err(GetBlockRangeError::PoolTypeArgumentError) - .map_err(FetchServiceError::from)?; + .map_err(NodeBackedIndexerServiceError::from)?; // Note conversion here is safe due to the use of [`ValidatedBlockRangeRequest::new_from_block_range`] let start = validated_request.start() as u32; let end = validated_request.end() as u32; - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); - let snapshot = fetch_service_clone - .indexer - .snapshot_nonfinalized_state() - .await?; + let service_clone = self.clone(); + let service_timeout = self.config.service.timeout; + let (channel_tx, channel_rx) = mpsc::channel(self.config.service.channel_size as usize); + let snapshot = service_clone.indexer.snapshot_nonfinalized_state().await?; tokio::spawn(async move { let timeout_result = timeout( @@ -1268,7 +1418,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // Use the snapshot tip directly, as this function doesn't support passthrough let chain_height = non_finalized_snapshot.best_tip.height.0; - match fetch_service_clone + match service_clone .indexer .get_compact_block_stream( &snapshot, @@ -1378,7 +1528,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { let (hex, height) = if let GetRawTransaction::Object(tx_object) = tx { (tx_object.hex().clone(), tx_object.height()) } else { - return Err(FetchServiceError::TonicStatusError( + return Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::not_found("Error: Transaction not received"), )); }; @@ -1391,7 +1541,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { // TODO: This probably shouldn't be an error. // this is an improvement over previous behaviour of // acting as if we are only synced to the genesis block - return Err(FetchServiceError::UnavailableNotSyncedEnough); + return Err(NodeBackedIndexerServiceError::UnavailableNotSyncedEnough); }; non_finalized_snapshot.best_tip.height.0 as u64 } @@ -1402,7 +1552,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { height, }) } else { - Err(FetchServiceError::TonicStatusError( + Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::invalid_argument("Error: Transaction hash incorrect"), )) } @@ -1426,17 +1576,15 @@ impl LightWalletIndexer for FetchServiceSubscriber { ) -> Result { let chain_height = self.chain_height().await?; let txids = self.get_taddress_txids_helper(request).await?; - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; - let (transmitter, receiver) = - mpsc::channel(self.config.common.service.channel_size as usize); + let service_clone = self.clone(); + let service_timeout = self.config.service.timeout; + let (transmitter, receiver) = mpsc::channel(self.config.service.channel_size as usize); tokio::spawn(async move { let timeout = timeout( time::Duration::from_secs((service_timeout * 4) as u64), async { for txid in txids { - let transaction = - fetch_service_clone.get_raw_transaction(txid, Some(1)).await; + let transaction = service_clone.get_raw_transaction(txid, Some(1)).await; if handle_raw_transaction::( chain_height.0 as u64, transaction, @@ -1483,9 +1631,9 @@ impl LightWalletIndexer for FetchServiceSubscriber { let checked_balance: i64 = match i64::try_from(balance.balance()) { Ok(balance) => balance, Err(_) => { - return Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Error converting balance from u64 to i64.", - ))); + return Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown("Error: Error converting balance from u64 to i64."), + )); } }; Ok(Balance { @@ -1499,10 +1647,10 @@ impl LightWalletIndexer for FetchServiceSubscriber { &self, mut request: AddressStream, ) -> Result { - let fetch_service_clone = self.clone(); - let service_timeout = self.config.common.service.timeout; + let service_clone = self.clone(); + let service_timeout = self.config.service.timeout; let (channel_tx, mut channel_rx) = - mpsc::channel::(self.config.common.service.channel_size as usize); + mpsc::channel::(self.config.service.channel_size as usize); let fetcher_task_handle = tokio::spawn(async move { let fetcher_timeout = timeout( time::Duration::from_secs((service_timeout * 4) as u64), @@ -1512,8 +1660,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { match channel_rx.recv().await { Some(taddr) => { let taddrs = GetAddressBalanceRequest::new(vec![taddr]); - let balance = - fetch_service_clone.z_get_address_balance(taddrs).await?; + let balance = service_clone.z_get_address_balance(taddrs).await?; total_balance += balance.balance(); } None => { @@ -1561,11 +1708,11 @@ impl LightWalletIndexer for FetchServiceSubscriber { Ok(Ok(())) => {} Ok(Err(e)) => { fetcher_task_handle.abort(); - return Err(FetchServiceError::TonicStatusError(e)); + return Err(NodeBackedIndexerServiceError::TonicStatusError(e)); } Err(_) => { fetcher_task_handle.abort(); - return Err(FetchServiceError::TonicStatusError( + return Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::deadline_exceeded( "Error: get_taddress_balance_stream request timed out in address loop.", ), @@ -1579,21 +1726,23 @@ impl LightWalletIndexer for FetchServiceSubscriber { Err(_) => { // TODO: Hide server error from clients before release. // Currently useful for dev purposes. - return Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Error converting balance from u64 to i64.", - ))); + return Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown( + "Error: Error converting balance from u64 to i64.", + ), + )); } }; Ok(Balance { value_zat: checked_balance, }) } - Ok(Err(e)) => Err(FetchServiceError::TonicStatusError(e)), + Ok(Err(e)) => Err(NodeBackedIndexerServiceError::TonicStatusError(e)), // TODO: Hide server error from clients before release. // Currently useful for dev purposes. - Err(e) => Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - format!("Fetcher Task failed: {e}"), - ))), + Err(e) => Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown(format!("Fetcher Task failed: {e}")), + )), } } @@ -1618,7 +1767,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { for (i, excluded_id) in request.exclude_txid_suffixes.iter().enumerate() { if excluded_id.len() > 32 { - return Err(FetchServiceError::TonicStatusError( + return Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::invalid_argument(format!( "Error: excluded txid {} is larger than 32 bytes", i @@ -1634,9 +1783,8 @@ impl LightWalletIndexer for FetchServiceSubscriber { } let mempool = self.indexer.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); + let service_timeout = self.config.service.timeout; + let (channel_tx, channel_rx) = mpsc::channel(self.config.service.channel_size as usize); tokio::spawn(async move { let timeout = timeout( @@ -1735,9 +1883,8 @@ impl LightWalletIndexer for FetchServiceSubscriber { #[allow(deprecated)] async fn get_mempool_stream(&self) -> Result { let indexer = self.indexer.clone(); - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); + let service_timeout = self.config.service.timeout; + let (channel_tx, channel_rx) = mpsc::channel(self.config.service.channel_size as usize); let snapshot = indexer.snapshot_nonfinalized_state().await?; tokio::spawn(async move { let timeout = timeout( @@ -1818,23 +1965,16 @@ impl LightWalletIndexer for FetchServiceSubscriber { /// The block can be specified by either height or hash. async fn get_tree_state(&self, request: BlockId) -> Result { let hash_or_height = blockid_to_hashorheight(request).ok_or( - crate::error::FetchServiceError::TonicStatusError(tonic::Status::invalid_argument( - "Invalid hash or height", - )), + crate::error::NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::invalid_argument("Invalid hash or height"), + ), )?; - let treestate_response = ::z_get_treestate( - self, - hash_or_height.to_string(), - ) - .await?; + let treestate_response = + ::z_get_treestate(self, hash_or_height.to_string()).await?; Ok(super::tree_state_from_treestate_response( - self.config - .common - .network - .to_zebra_network() - .bip70_network_name(), + self.data.network().bip70_network_name(), treestate_response, )) } @@ -1852,8 +1992,8 @@ impl LightWalletIndexer for FetchServiceSubscriber { #[allow(deprecated)] fn timeout_channel_size(&self) -> (u32, u32) { ( - self.config.common.service.timeout, - self.config.common.service.channel_size, + self.config.service.timeout, + self.config.service.channel_size, ) } @@ -1884,17 +2024,21 @@ impl LightWalletIndexer for FetchServiceSubscriber { let checked_index = match i32::try_from(output_index.index()) { Ok(index) => index, Err(_) => { - return Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Index out of range. Failed to convert to i32.", - ))); + return Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown( + "Error: Index out of range. Failed to convert to i32.", + ), + )); } }; let checked_satoshis = match i64::try_from(satoshis) { Ok(satoshis) => satoshis, Err(_) => { - return Err(FetchServiceError::TonicStatusError(tonic::Status::unknown( - "Error: Satoshis out of range. Failed to convert to i64.", - ))); + return Err(NodeBackedIndexerServiceError::TonicStatusError( + tonic::Status::unknown( + "Error: Satoshis out of range. Failed to convert to i64.", + ), + )); } }; let utxo_reply = GetAddressUtxosReply { @@ -1924,9 +2068,8 @@ impl LightWalletIndexer for FetchServiceSubscriber { super::validate_utxo_address_count(request.addresses.len())?; let taddrs = GetAddressBalanceRequest::new(request.addresses); let utxos = self.z_get_address_utxos(taddrs).await?; - let service_timeout = self.config.common.service.timeout; - let (channel_tx, channel_rx) = - mpsc::channel(self.config.common.service.channel_size as usize); + let service_timeout = self.config.service.timeout; + let (channel_tx, channel_rx) = mpsc::channel(self.config.service.channel_size as usize); tokio::spawn(async move { let timeout = timeout( time::Duration::from_secs((service_timeout * 4) as u64), @@ -2018,7 +2161,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { .to_string(); let latest_upgrade = super::latest_network_upgrade(blockchain_info.upgrades()) - .map_err(FetchServiceError::TonicStatusError)? + .map_err(NodeBackedIndexerServiceError::TonicStatusError)? .into_parts(); let nu_name = latest_upgrade.0; @@ -2041,7 +2184,6 @@ impl LightWalletIndexer for FetchServiceSubscriber { zcashd_subversion: self.data.zebra_subversion(), donation_address: self .config - .common .donation_address .as_ref() .map(DonationAddress::encode) @@ -2056,7 +2198,7 @@ impl LightWalletIndexer for FetchServiceSubscriber { /// /// NOTE: Currently unimplemented in Zaino. async fn ping(&self, _request: Duration) -> Result { - Err(FetchServiceError::TonicStatusError( + Err(NodeBackedIndexerServiceError::TonicStatusError( tonic::Status::unimplemented( "Ping not yet implemented. If you require this RPC \ please open an issue or PR at the Zaino github \ diff --git a/packages/zaino-state/src/lib.rs b/packages/zaino-state/src/lib.rs index 548c1da16..a2c0d67a6 100644 --- a/packages/zaino-state/src/lib.rs +++ b/packages/zaino-state/src/lib.rs @@ -56,18 +56,16 @@ pub use indexer::{ ZcashService, }; -pub(crate) mod backends; - -#[allow(deprecated)] -pub use backends::{ - fetch::{FetchService, FetchServiceSubscriber}, - state::{StateService, StateServiceSubscriber}, +pub use indexer::node_backed_indexer::{ + ChainTipSubscriber, NodeBackedIndexerService, NodeBackedIndexerServiceSubscriber, }; pub mod chain_index; // Core ChainIndex trait and implementations -pub use chain_index::{ChainIndex, NodeBackedChainIndex, NodeBackedChainIndexSubscriber}; +pub use chain_index::{ + ChainIndex, ChainIndexRpcExt, NodeBackedChainIndex, NodeBackedChainIndexSubscriber, +}; // Source types for ChainIndex backends pub use chain_index::source::{BlockchainSource, State, ValidatorConnector}; // Supporting types @@ -102,16 +100,14 @@ pub mod test_dependencies { pub(crate) mod config; -#[allow(deprecated)] pub use config::{ - BackendConfig, BackendType, ChainIndexConfig, CommonBackendConfig, DonationAddress, - FetchServiceConfig, StateServiceConfig, + ChainIndexConfig, CommonBackendConfig, DirectConnectionConfig, DonationAddress, + NodeBackedIndexerServiceConfig, ValidatorConnectionType, }; pub(crate) mod error; -#[allow(deprecated)] -pub use error::{FetchServiceError, StateServiceError}; +pub use error::NodeBackedIndexerServiceError; pub(crate) mod status; diff --git a/packages/zainod/Cargo.toml b/packages/zainod/Cargo.toml index 4fc9a786d..beba27ad9 100644 --- a/packages/zainod/Cargo.toml +++ b/packages/zainod/Cargo.toml @@ -70,7 +70,6 @@ zaino-state = { workspace = true } zaino-serve = { workspace = true } # Zebra -zebra-chain = { workspace = true } zebra-state = { workspace = true } # Runtime @@ -81,7 +80,6 @@ clap = { workspace = true, features = ["derive"] } # Tracing tracing = { workspace = true } -tracing-subscriber = { workspace = true, features = ["fmt", "env-filter", "time"] } # Network / RPC http = { workspace = true } @@ -97,8 +95,8 @@ thiserror = { workspace = true } # Formats toml = { workspace = true } config = { workspace = true } -tempfile = { workspace = true } [dev-dependencies] +tempfile = { workspace = true } zcash_address = { workspace = true } zcash_protocol = { workspace = true } diff --git a/packages/zainod/src/config.rs b/packages/zainod/src/config.rs index aeb97dbbe..889359c65 100644 --- a/packages/zainod/src/config.rs +++ b/packages/zainod/src/config.rs @@ -21,11 +21,32 @@ use zaino_common::{ try_resolve_address, AddressResolution, Network, ServiceConfig, StorageConfig, ValidatorConfig, }; use zaino_serve::server::config::{GrpcServerConfig, JsonRpcServerConfig}; -#[allow(deprecated)] use zaino_state::{ - BackendType, CommonBackendConfig, DonationAddress, FetchServiceConfig, StateServiceConfig, + CommonBackendConfig, DirectConnectionConfig, DonationAddress, NodeBackedIndexerServiceConfig, + ValidatorConnectionType, }; +/// On-disk selector for the validator connection (`backend = "direct" | "rpc"` in the +/// config file), mapped to [`zaino_state::ValidatorConnectionType`] at spawn. +/// +/// The legacy values `"state"` / `"fetch"` remain accepted as aliases for backward +/// compatibility with existing `zainod.toml` files. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum BackendType { + /// Direct Zebra `ReadStateService` access (formerly `state`). + /// + /// More efficient but requires running on the same machine as Zebra. + #[serde(alias = "state")] + Direct, + /// JSON-RPC access (formerly `fetch`). + /// + /// Compatible with Zcashd, Zebra, or another Zaino instance. + #[default] + #[serde(alias = "fetch")] + Rpc, +} + /// Header for generated configuration files. pub const GENERATED_CONFIG_HEADER: &str = r#"# Zaino Configuration # @@ -82,7 +103,8 @@ pub struct ZainodConfig { /// When enabled, Zaino does not use a persistent on-disk finalised-state database. Finalised /// state reads are served from the configured validator/source instead. pub ephemeral_finalised_state: bool, - /// Network to connect to (Mainnet, Testnet, or Regtest). + /// Network to connect to (Mainnet, PubTestnet — The Public Testnet — or Regtest; + /// `"Testnet"` is accepted as a legacy spelling of PubTestnet). pub network: Network, /// Prometheus metrics endpoint listen address. /// @@ -222,11 +244,6 @@ impl ZainodConfig { Ok(()) } - - /// Returns the network type currently being used by the server. - pub fn get_network(&self) -> Result { - Ok(self.network.to_zebra_network()) - } } impl Default for ZainodConfig { @@ -250,7 +267,7 @@ impl Default for ZainodConfig { storage: StorageConfig::default(), ephemeral_finalised_state: false, zebra_db_path: default_zebra_db_path(), - network: Network::Testnet, + network: Network::PubTestnet, donation_address: None, } } @@ -363,59 +380,56 @@ pub fn load_config_with_env( Ok(parsed_config) } -#[allow(deprecated)] -impl TryFrom for StateServiceConfig { +impl TryFrom for NodeBackedIndexerServiceConfig { type Error = IndexerError; fn try_from(cfg: ZainodConfig) -> Result { - let grpc_listen_address = cfg - .validator_settings - .validator_grpc_listen_address - .as_ref() - .ok_or_else(|| { - IndexerError::ConfigError( - "Missing validator_grpc_listen_address in configuration".to_string(), - ) - })?; - - let validator_grpc_address = - fetch_socket_addr_from_hostname(grpc_listen_address).map_err(|e| { - let msg = match e { - IndexerError::ConfigError(msg) => msg, - other => other.to_string(), + let connection = match cfg.backend { + BackendType::Rpc => ValidatorConnectionType::Rpc, + BackendType::Direct => { + let grpc_listen_address = cfg + .validator_settings + .validator_grpc_listen_address + .as_ref() + .ok_or_else(|| { + IndexerError::ConfigError( + "Missing validator_grpc_listen_address in configuration".to_string(), + ) + })?; + + let validator_grpc_address = fetch_socket_addr_from_hostname(grpc_listen_address) + .map_err(|e| { + let msg = match e { + IndexerError::ConfigError(msg) => msg, + other => other.to_string(), + }; + IndexerError::ConfigError(format!( + "Invalid validator_grpc_listen_address '{grpc_listen_address}': {msg}" + )) + })?; + + let validator_state_config = zebra_state::Config { + cache_dir: cfg.zebra_db_path.clone(), + ephemeral: false, + delete_old_database: true, + debug_stop_at_height: None, + debug_validity_check_interval: None, + should_backup_non_finalized_state: true, + debug_skip_non_finalized_state_backup_task: false, }; - IndexerError::ConfigError(format!( - "Invalid validator_grpc_listen_address '{grpc_listen_address}': {msg}" - )) - })?; - - let validator_state_config = zebra_state::Config { - cache_dir: cfg.zebra_db_path.clone(), - ephemeral: false, - delete_old_database: true, - debug_stop_at_height: None, - debug_validity_check_interval: None, - should_backup_non_finalized_state: true, - debug_skip_non_finalized_state_backup_task: false, - }; - let validator_cookie_auth = cfg.validator_settings.validator_cookie_path.is_some(); - - Ok(StateServiceConfig { - common: build_common(cfg), - validator_state_config, - validator_grpc_address, - validator_cookie_auth, - }) - } -} + let validator_cookie_auth = cfg.validator_settings.validator_cookie_path.is_some(); -#[allow(deprecated)] -impl TryFrom for FetchServiceConfig { - type Error = IndexerError; + ValidatorConnectionType::Direct(DirectConnectionConfig { + validator_state_config, + validator_grpc_address, + validator_cookie_auth, + }) + } + }; - fn try_from(cfg: ZainodConfig) -> Result { - Ok(FetchServiceConfig { + Ok(NodeBackedIndexerServiceConfig { common: build_common(cfg), + connection, }) } } @@ -555,7 +569,8 @@ key_path = "{}" let config_path = create_test_config_file(&temp_dir, &toml_content, "full_config.toml"); let config = load_config(&config_path).expect("load_config failed"); - assert_eq!(config.backend, BackendType::Fetch); + // legacy `backend = "fetch"` still parses via the serde alias + assert_eq!(config.backend, BackendType::Rpc); assert!(config.json_server_settings.is_some()); assert_eq!( config @@ -588,7 +603,7 @@ key_path = "{}" let toml_content = r#" backend = "state" -network = "Testnet" +network = "PubTestnet" zebra_db_path = "/opt/zebra/data" [storage.database] @@ -605,7 +620,9 @@ listen_address = "127.0.0.1:8137" let config = load_config(&config_path).expect("load_config failed"); let default_values = ZainodConfig::default(); - assert_eq!(config.backend, BackendType::State); + // legacy `backend = "state"` still parses via the serde alias + assert_eq!(config.backend, BackendType::Direct); + assert_eq!(config.network, Network::PubTestnet); assert!(config.json_server_settings.is_none()); assert_eq!( config.validator_settings.validator_user, @@ -617,6 +634,33 @@ listen_address = "127.0.0.1:8137" ); } + /// The pre-rename config spelling of The Public Testnet still parses, + /// via the `#[serde(alias = "Testnet")]` on `Network::PubTestnet`. + #[test] + fn legacy_testnet_spelling_parses_as_the_pub_testnet() { + let _guard = EnvGuard::new(); + let temp_dir = TempDir::new().unwrap(); + + let toml_content = r#" +backend = "state" +network = "Testnet" +zebra_db_path = "/opt/zebra/data" + +[storage.database] +path = "/opt/zaino/data" + +[validator_settings] +validator_jsonrpc_listen_address = "127.0.0.1:18232" + +[grpc_settings] +listen_address = "127.0.0.1:8137" +"#; + + let config_path = create_test_config_file(&temp_dir, toml_content, "legacy_testnet.toml"); + let config = load_config(&config_path).expect("load_config failed"); + assert_eq!(config.network, Network::PubTestnet); + } + #[test] fn test_cookie_dir_logic() { let _guard = EnvGuard::new(); @@ -625,7 +669,7 @@ listen_address = "127.0.0.1:8137" // Scenario 1: auth enabled, cookie_dir empty (should use default ephemeral path) let toml_content = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" zebra_db_path = "/zebra/db" [storage.database] @@ -655,7 +699,7 @@ listen_address = "127.0.0.1:8137" // Scenario 2: auth enabled, cookie_dir specified let toml_content2 = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" zebra_db_path = "/zebra/db" [storage.database] @@ -682,7 +726,7 @@ listen_address = "127.0.0.1:8137" // Scenario 3: cookie_dir not specified (should be None) let toml_content3 = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" zebra_db_path = "/zebra/db" [storage.database] @@ -793,7 +837,7 @@ listen_address = "127.0.0.1:8137" let temp_dir = TempDir::new().unwrap(); let toml_content = r#" -network = "Testnet" +network = "PubTestnet" [validator_settings] validator_jsonrpc_listen_address = "127.0.0.1:18232" @@ -884,7 +928,7 @@ listen_address = "127.0.0.1:8137" let toml_content = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" [validator_settings] validator_jsonrpc_listen_address = "192.168.1.10:18232" @@ -915,7 +959,7 @@ listen_address = "127.0.0.1:8137" let toml_content = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" [validator_settings] validator_jsonrpc_listen_address = "8.8.8.8:18232" @@ -1138,66 +1182,56 @@ listen_address = "127.0.0.1:8137" /// `CARGO_PKG_VERSION` at the boundary so the wire reflects the /// deployed binary, not zaino-state's library version. #[test] - #[allow(deprecated)] fn indexer_version_is_zainod_pkg_version() { let _guard = EnvGuard::new(); - let cfg = ZainodConfig::default(); - - let state_cfg = StateServiceConfig::try_from(cfg.clone()) - .expect("StateServiceConfig conversion should succeed for default ZainodConfig"); - assert_eq!(state_cfg.common.indexer_version, env!("CARGO_PKG_VERSION")); - - let fetch_cfg = FetchServiceConfig::try_from(cfg) - .expect("FetchServiceConfig conversion should succeed for default ZainodConfig"); - assert_eq!(fetch_cfg.common.indexer_version, env!("CARGO_PKG_VERSION")); + let service_cfg = NodeBackedIndexerServiceConfig::try_from(ZainodConfig::default()) + .expect("service config conversion should succeed for default ZainodConfig"); + assert_eq!( + service_cfg.common.indexer_version, + env!("CARGO_PKG_VERSION") + ); } - /// `StateServiceConfig::try_from` and `FetchServiceConfig::try_from` - /// share a single `build_common` helper, so the two backends can - /// never quietly disagree on the common payload they hand to a - /// service. Locks that property in across every field: a future - /// hand-rolled divergence (e.g. one path stops applying the - /// missing-credentials sentinel, or a new common field gets - /// populated on only one side) makes this fail. Pretty-Debug - /// equality is used because not every constituent of - /// `CommonBackendConfig` derives `PartialEq`, and a single - /// stringified compare future-proofs the test against fields added - /// later. + /// The `Rpc` and `Direct` connections share a single `build_common` helper, so the + /// common payload handed to the service is connection-independent. Locks that in + /// across every field: a future divergence (e.g. one path stops applying the + /// missing-credentials sentinel, or a new common field gets populated on only one + /// side) makes this fail. Pretty-Debug equality is used because not every constituent + /// of `CommonBackendConfig` derives `PartialEq`, and a single stringified compare + /// future-proofs the test against fields added later. #[test] - #[allow(deprecated)] - fn state_and_fetch_common_payloads_agree() { + fn common_payload_is_connection_independent() { let _guard = EnvGuard::new(); - let cfg = ZainodConfig::default(); + let rpc_cfg = NodeBackedIndexerServiceConfig::try_from(ZainodConfig { + backend: BackendType::Rpc, + ..ZainodConfig::default() + }) + .expect("Rpc conversion should succeed for default ZainodConfig"); + let direct_cfg = NodeBackedIndexerServiceConfig::try_from(ZainodConfig { + backend: BackendType::Direct, + ..ZainodConfig::default() + }) + .expect("Direct conversion should succeed for default ZainodConfig"); - let state_config = StateServiceConfig::try_from(cfg.clone()) - .expect("StateServiceConfig conversion should succeed for default ZainodConfig"); - let fetch_config = FetchServiceConfig::try_from(cfg) - .expect("FetchServiceConfig conversion should succeed for default ZainodConfig"); + assert!(matches!(rpc_cfg.connection, ValidatorConnectionType::Rpc)); + assert!(matches!( + direct_cfg.connection, + ValidatorConnectionType::Direct(_) + )); assert_eq!( - format!("{:#?}", state_config.common), - format!("{:#?}", fetch_config.common), + format!("{:#?}", rpc_cfg.common), + format!("{:#?}", direct_cfg.common), ); - let cfg = ZainodConfig { + let ephemeral_cfg = NodeBackedIndexerServiceConfig::try_from(ZainodConfig { ephemeral_finalised_state: true, ..ZainodConfig::default() - }; - - let state_config = StateServiceConfig::try_from(cfg.clone()) - .expect("StateServiceConfig conversion should succeed for ephemeral finalised state"); - let fetch_config = FetchServiceConfig::try_from(cfg) - .expect("FetchServiceConfig conversion should succeed for ephemeral finalised state"); - - assert!(state_config.common.ephemeral_finalised_state); - assert!(fetch_config.common.ephemeral_finalised_state); - - assert_eq!( - format!("{:#?}", state_config.common), - format!("{:#?}", fetch_config.common), - ); + }) + .expect("conversion should succeed for ephemeral finalised state"); + assert!(ephemeral_cfg.common.ephemeral_finalised_state); } /// Builds a default config with the JSON-RPC server bound to `addr`. @@ -1286,7 +1320,7 @@ listen_address = "127.0.0.1:8137" let toml_content = r#" backend = "fetch" -network = "Testnet" +network = "PubTestnet" ephemeral_finalised_state = true [validator_settings] @@ -1305,12 +1339,9 @@ listen_address = "127.0.0.1:8137" assert!(config.ephemeral_finalised_state); - #[allow(deprecated)] - { - let fetch_config = FetchServiceConfig::try_from(config) - .expect("FetchServiceConfig conversion should succeed"); + let service_config = NodeBackedIndexerServiceConfig::try_from(config) + .expect("service config conversion should succeed"); - assert!(fetch_config.common.ephemeral_finalised_state); - } + assert!(service_config.common.ephemeral_finalised_state); } } diff --git a/packages/zainod/src/error.rs b/packages/zainod/src/error.rs index 38a2433e4..0c27cecac 100644 --- a/packages/zainod/src/error.rs +++ b/packages/zainod/src/error.rs @@ -3,12 +3,10 @@ use zaino_fetch::jsonrpsee::error::TransportError; use zaino_serve::server::error::ServerError; -#[allow(deprecated)] -use zaino_state::{FetchServiceError, StateServiceError}; +use zaino_state::NodeBackedIndexerServiceError; /// Zingo-Indexer errors. #[derive(Debug, thiserror::Error)] -#[allow(deprecated)] pub enum IndexerError { /// Server based errors. #[error("Server error: {0}")] @@ -19,12 +17,9 @@ pub enum IndexerError { /// JSON RPSee connector errors. #[error("JSON RPSee connector error: {0}")] TransportError(#[from] TransportError), - /// FetchService errors. - #[error("FetchService error: {0}")] - FetchServiceError(Box), - /// FetchService errors. - #[error("StateService error: {0}")] - StateServiceError(Box), + /// NodeBackedIndexerService errors. + #[error("NodeBackedIndexerService error: {0}")] + NodeBackedIndexerServiceError(Box), /// HTTP related errors due to invalid URI. #[error("HTTP error: Invalid URI {0}")] HttpError(#[from] http::Error), @@ -43,16 +38,8 @@ pub enum IndexerError { Restart, } -#[allow(deprecated)] -impl From for IndexerError { - fn from(value: StateServiceError) -> Self { - IndexerError::StateServiceError(Box::new(value)) - } -} - -#[allow(deprecated)] -impl From for IndexerError { - fn from(value: FetchServiceError) -> Self { - IndexerError::FetchServiceError(Box::new(value)) +impl From for IndexerError { + fn from(value: NodeBackedIndexerServiceError) -> Self { + IndexerError::NodeBackedIndexerServiceError(Box::new(value)) } } diff --git a/packages/zainod/src/indexer.rs b/packages/zainod/src/indexer.rs index 1b23120a8..cfcdc2064 100644 --- a/packages/zainod/src/indexer.rs +++ b/packages/zainod/src/indexer.rs @@ -8,10 +8,9 @@ use zaino_serve::{ rpc::grpc_routes, server::{config::GrpcServerConfig, grpc::TonicServer, jsonrpc::JsonRpcServer}, }; -#[allow(deprecated)] use zaino_state::{ - BackendType, FetchService, FetchServiceConfig, IndexerService, LightWalletService, - StateService, StateServiceConfig, StatusType, ZcashIndexer, ZcashService, + IndexerService, LightWalletService, NodeBackedIndexerService, NodeBackedIndexerServiceConfig, + StatusType, ZcashIndexer, ZcashService, }; use crate::{config::ZainodConfig, error::IndexerError}; @@ -61,21 +60,13 @@ pub async fn spawn_indexer( info!(uri = %zebrad_uri, "Connected to node via JsonRPSee"); - #[allow(deprecated)] - match config.backend { - BackendType::State => { - let state_config = StateServiceConfig::try_from(config.clone())?; - Indexer::::launch_inner(state_config, config) - .await - .map(|res| res.0) - } - BackendType::Fetch => { - let fetch_config = FetchServiceConfig::try_from(config.clone())?; - Indexer::::launch_inner(fetch_config, config) - .await - .map(|res| res.0) - } - } + // Both the JSON-RPC (`Rpc`) and direct-`ReadStateService` (`Direct`) connections are + // now served by the single `NodeBackedIndexerService`; the connection is selected + // inside the config conversion from `config.backend`. + let service_config = NodeBackedIndexerServiceConfig::try_from(config.clone())?; + Indexer::::launch_inner(service_config, config) + .await + .map(|res| res.0) } impl Indexer diff --git a/tools/makefiles/lints.toml b/tools/makefiles/lints.toml index cac50b8d3..e6824f01f 100644 --- a/tools/makefiles/lints.toml +++ b/tools/makefiles/lints.toml @@ -82,6 +82,11 @@ description = "Assert Dockerfile.deterministic's stagex/pallet-rust tag matches command = "cargo" args = ["run", "-q", "--manifest-path", "tools/workbench/Cargo.toml", "--bin", "check-toolchain-pin"] +[tasks.lint-crypto-provider] +description = "Assert rustls resolves with aws-lc-rs + prefer-post-quantum and that the only ring-selecting feature edges are the tolerated zebra path (ADR-0006). Fails loud in both directions: new edges and vanished edges." +command = "cargo" +args = ["run", "-q", "--manifest-path", "tools/workbench/Cargo.toml", "--bin", "check-crypto-provider"] + [tasks.lint-no-openssl-apt] description = "Fail if any Dockerfile installs a libssl*/openssl* apt package. TLS is rustls; the system-package companion to deny.toml's openssl crate ban." command = "cargo" @@ -106,6 +111,7 @@ dependencies = [ "lint-boundary-conversions", "lint-workbench", "lint-toolchain-pin", + "lint-crypto-provider", "lint-no-openssl-apt", "deny", "lint-shell", diff --git a/tools/scripts/help.sh b/tools/scripts/help.sh index 524c80b6c..7f962451c 100755 --- a/tools/scripts/help.sh +++ b/tools/scripts/help.sh @@ -12,7 +12,7 @@ echo "" echo "Common usage:" echo " makers test # packages/* tests, no live validator (default)" echo " makers test live # both live partitions + combined summary" -echo " makers test all # everything: package then live" +echo " makers test all # everything: packages then live" echo "" echo "If you modify '.env.testing-artifacts', the test command will \ automatically:" @@ -21,15 +21,15 @@ echo " - Build a new local container image if needed" echo "" echo "Available commands:" echo "" -echo " test [SET] Front door. SET = package (default) | e2e | \ +echo " test [SET] Front door. SET = packages (default) | e2e | \ clientless | live | all" -echo " package packages/* tests, no live \ +echo " packages packages/* tests, no live \ validator" echo " e2e the e2e live partition" echo " clientless the clientless live partition" echo " live both live partitions + \ combined summary" -echo " all package then live (everything)" +echo " all packages then live (everything)" echo " container-test Engine: run nextest in the container \ (used by the front door; invoke directly to forward engine flags)" echo " live-clientless Engine: run the clientless live-test \ diff --git a/tools/test-runner/src/bin/live-summary.rs b/tools/test-runner/src/bin/live-summary.rs index 47d3ff3bd..2ca93c47e 100644 --- a/tools/test-runner/src/bin/live-summary.rs +++ b/tools/test-runner/src/bin/live-summary.rs @@ -1,10 +1,12 @@ -//! `live-summary` — run both live-test partitions and print a combined summary. +//! `live-summary` — run the test partitions and print a combined summary. //! -//! Invoked from `makers test live` (and `makers test all`) as -//! `cargo run --bin live-summary -- `. Runs the `clientless` then `e2e` -//! partition (each in its own CI container via its own `makers` task), streams -//! each run's output while capturing it, parses the nextest summary line, and -//! aggregates the totals. +//! Invoked from `makers test live` as `cargo run --bin live-summary -- ` +//! and from `makers test all` with the extra `--all` flag. The live set runs +//! the `clientless` then `e2e` partition (each in its own CI container via its +//! own `makers` task); `--all` first runs the `packages` set (`makers +//! container-test`) and adds its row to the table. The runner streams each +//! run's output while capturing it, parses the nextest summary line for +//! tallies, and measures wall-clock duration via `Instant`. //! //! Unlike a cargo-make `dependencies` list (which is fail-fast), this runs BOTH //! partitions even when the first fails, so the summary reflects the whole @@ -18,6 +20,15 @@ use std::error::Error; use std::io::{BufRead, BufReader}; use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +/// Which partitions this invocation runs and summarizes. +enum TestSet { + /// The two live partitions (`makers test live`). + Live, + /// The packages set plus the two live partitions (`makers test all`). + All, +} /// One nextest run's tallies, zero where the summary line was absent. #[derive(Default)] @@ -26,6 +37,8 @@ struct Summary { passed: u64, failed: u64, skipped: u64, + /// Wall-clock duration measured by the runner (includes build + test time). + elapsed: Duration, } impl Summary { @@ -35,6 +48,7 @@ impl Summary { passed: self.passed + other.passed, failed: self.failed + other.failed, skipped: self.skipped + other.skipped, + elapsed: self.elapsed + other.elapsed, } } } @@ -115,8 +129,7 @@ fn parse_summary(log: &str) -> Summary { let line = log .lines() .map(strip_ansi) - .filter(|l| l.contains("run:") && l.contains("test")) - .last() + .rfind(|l| l.contains("run:") && l.contains("test")) .unwrap_or_default(); Summary { @@ -125,37 +138,82 @@ fn parse_summary(log: &str) -> Summary { passed: count_before(&line, "passed"), failed: count_before(&line, "failed"), skipped: count_before(&line, "skipped"), + ..Summary::default() } } fn print_row(label: &str, s: &Summary) { + let total_secs = s.elapsed.as_secs(); + let millis = s.elapsed.subsec_millis(); + let secs = format!("{total_secs}.{millis:03}"); println!( - " {label:<18} {:>4} run, {:>4} passed, {:>4} failed, {:>4} skipped", + " {label:<18} {:>4} run, {:>4} passed, {:>4} failed, {:>4} skipped, {secs:>10}s", s.run, s.passed, s.failed, s.skipped ); } fn main() -> Result<(), Box> { let with_zcashd = std::env::args().any(|a| a == "--with-zcashd"); + let set = if std::env::args().any(|a| a == "--all") { + TestSet::All + } else { + TestSet::Live + }; + + let packages = match set { + TestSet::All => { + println!(">>> all: running packages set"); + let start = Instant::now(); + let (rc, log) = run_partition("container-test", with_zcashd)?; + let elapsed = start.elapsed(); + let mut s = parse_summary(&log); + s.elapsed = elapsed; + Some((rc, s)) + } + TestSet::Live => None, + }; println!(">>> live: running clientless partition"); + let cl_start = Instant::now(); let (cl_rc, cl_log) = run_partition("live-clientless", with_zcashd)?; + let cl_elapsed = cl_start.elapsed(); println!(">>> live: running e2e partition"); + let e2e_start = Instant::now(); let (e2e_rc, e2e_log) = run_partition("live-e2e", with_zcashd)?; + let e2e_elapsed = e2e_start.elapsed(); - let cl = parse_summary(&cl_log); - let e2e = parse_summary(&e2e_log); + let mut cl = parse_summary(&cl_log); + cl.elapsed = cl_elapsed; + let mut e2e = parse_summary(&e2e_log); + e2e.elapsed = e2e_elapsed; + + let header = match packages { + Some(_) => "test summary", + None => "live summary", + }; + let mut total = cl.add(&e2e); + if let Some((_, p)) = &packages { + total = total.add(p); + } println!(); - println!("====================== live summary =========================="); + println!("====================== {header} =========================="); + if let Some((_, p)) = &packages { + print_row("packages:", p); + } print_row("clientless:", &cl); print_row("e2e:", &e2e); - print_row("TOTAL:", &cl.add(&e2e)); + print_row("TOTAL:", &total); println!("=============================================================="); // A partition that errored without producing a summary line likely failed // to build; call it out so the zeros above aren't read as "all clear". + if let Some((pkg_rc, p)) = &packages { + if *pkg_rc != 0 && p.run == 0 { + println!(" warning: packages produced no nextest summary (build failure?) — see output above."); + } + } if cl_rc != 0 && cl.run == 0 { println!(" warning: clientless produced no nextest summary (build failure?) — see output above."); } @@ -163,8 +221,9 @@ fn main() -> Result<(), Box> { println!(" warning: e2e produced no nextest summary (build failure?) — see output above."); } - // Fail the front door if either partition failed. - if cl_rc != 0 || e2e_rc != 0 { + // Fail the front door if any set failed. + let pkg_rc = packages.map_or(0, |(rc, _)| rc); + if pkg_rc != 0 || cl_rc != 0 || e2e_rc != 0 { std::process::exit(1); } Ok(()) diff --git a/tools/workbench/src/bin/check-crypto-provider.rs b/tools/workbench/src/bin/check-crypto-provider.rs new file mode 100644 index 000000000..b3926a1cf --- /dev/null +++ b/tools/workbench/src/bin/check-crypto-provider.rs @@ -0,0 +1,206 @@ +//! Guard: one rustls CryptoProvider — aws-lc-rs — plus the single tolerated +//! ring path through zebra (ADR-0006). +//! +//! Two checks against the production workspace's resolved dependency graph: +//! +//! 1. rustls must resolve with `aws_lc_rs` and `prefer-post-quantum`. +//! 2. The set of feature edges that select ring anywhere in the graph must +//! equal [`RING_EDGE_ALLOWLIST`] — today, the single chain rooted in +//! zebra-node-services' `rpc-client` feature (its reqwest `rustls-tls`; +//! reqwest 0.12 offers no aws-lc alternative). That ring is compiled but +//! dormant: reqwest consults the process-default provider first and zaino +//! installs aws-lc-rs at both TLS boundaries. +//! +//! Loud in both directions. A NEW edge means a second provider path is +//! creeping back in via feature unification — the regression this guard +//! exists to catch (zingolabs/zaino#1360). A MISSING edge — in particular +//! ring vanishing from the graph entirely — means zebra dropped its ring +//! dependence: delete [`RING_EDGE_ALLOWLIST`], make this check assert ring's +//! absence, and close the classical-deprecation tracking issue's zebra item. + +use std::collections::BTreeSet; +use std::path::Path; +use std::process::Command; + +use workbench::{repo_root, run}; + +/// Feature names that select the ring provider when enabled on any crate. +const RING_SELECTING_FEATURES: [&str; 3] = ["ring", "__rustls-ring", "tls-ring"]; + +/// Every feature edge expected to select ring, all downstream of +/// zebra-node-services' reqwest `rustls-tls` (see module docs). +const RING_EDGE_ALLOWLIST: [&str; 5] = [ + "hyper-rustls feature \"ring\"", + "reqwest feature \"__rustls-ring\"", + "rustls feature \"ring\"", + "rustls-webpki feature \"ring\"", + "tokio-rustls feature \"ring\"", +]; + +fn main() { + run("check-crypto-provider", check, |()| { + println!( + "check-crypto-provider: ok — rustls resolves aws-lc-rs + prefer-post-quantum; \ + ring edges match the tolerated zebra path" + ); + }) +} + +fn check() -> Result<(), Vec> { + let root = repo_root()?; + + let rustls_features = cargo_tree(&root, &["--package", "rustls", "--depth", "0"], "{f}")?; + check_preferred_provider(&rustls_features)?; + + let ring_tree = match cargo_tree(&root, &["--invert", "ring", "--edges", "features"], "{p}") { + Ok(out) => out, + Err(diag) + if diag + .iter() + .any(|l| l.contains("did not match any packages")) => + { + return Err(vec![ + "ring is GONE from the dependency graph — zebra dropped it!".to_string(), + "Tighten this guard: delete RING_EDGE_ALLOWLIST and assert ring's absence," + .to_string(), + "and close the zebra item on the classical-deprecation tracking issue.".to_string(), + ]); + } + Err(diag) => return Err(diag), + }; + check_ring_edges(&ring_tree) +} + +/// Assert the resolved rustls feature set contains the preferred-provider pair. +fn check_preferred_provider(features_line: &str) -> Result<(), Vec> { + let features: BTreeSet<&str> = features_line.trim().split(',').collect(); + let missing: Vec<&str> = ["aws_lc_rs", "prefer-post-quantum"] + .into_iter() + .filter(|f| !features.contains(f)) + .collect(); + if missing.is_empty() { + Ok(()) + } else { + Err(vec![format!( + "rustls resolved without {} (got: {}) — the preferred provider per \ + ADR-0006 is aws-lc-rs with prefer-post-quantum; check \ + packages/zaino-common/Cargo.toml's rustls features", + missing.join(" + "), + features_line.trim(), + )]) + } +} + +/// Assert the ring-selecting feature edges equal the tolerated allowlist. +fn check_ring_edges(ring_tree: &str) -> Result<(), Vec> { + let found = ring_selecting_edges(ring_tree); + let expected: BTreeSet = RING_EDGE_ALLOWLIST + .iter() + .map(ToString::to_string) + .collect(); + + if found == expected { + return Ok(()); + } + + let mut msg = Vec::new(); + for new in found.difference(&expected) { + msg.push(format!( + "NEW ring-selecting feature edge: {new} — a second CryptoProvider path is \ + creeping back in (zingolabs/zaino#1360); remove the enabling feature", + )); + } + for gone in expected.difference(&found) { + msg.push(format!( + "tolerated ring edge disappeared: {gone} — if zebra dropped ring, tighten \ + this guard (see the module docs in check-crypto-provider.rs)", + )); + } + msg.push("see docs/adr/0006-aws-lc-rs-preferred-crypto-provider.md".to_string()); + Err(msg) +} + +/// The deduplicated ` feature ""` edges in a +/// `cargo tree --edges features` output whose feature name selects ring. +fn ring_selecting_edges(tree: &str) -> BTreeSet { + tree.lines() + .map(|line| { + line.trim_start_matches(['│', '├', '└', '─', ' ']) + .trim_end_matches(" (*)") + }) + .filter(|node| { + node.split_once(" feature ").is_some_and(|(_, feature)| { + RING_SELECTING_FEATURES + .iter() + .any(|f| feature == format!("\"{f}\"")) + }) + }) + .map(ToString::to_string) + .collect() +} + +/// Run `cargo tree --format ` against the production workspace +/// at `root`, returning stdout; diagnostics carry stderr so callers can +/// distinguish "package not in graph" from other failures. +fn cargo_tree(root: &Path, args: &[&str], format: &str) -> Result> { + let output = Command::new("cargo") + .arg("tree") + .args(args) + .args(["--format", format]) + .current_dir(root) + .output() + .map_err(|e| vec![format!("failed to run cargo tree: {e}")])?; + if !output.status.success() { + return Err(vec![format!( + "`cargo tree {}` failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr).trim(), + )]); + } + String::from_utf8(output.stdout).map_err(|e| vec![format!("cargo output not utf-8: {e}")]) +} + +#[cfg(test)] +mod ring_selecting_edges { + use super::*; + + #[test] + fn extracts_and_dedups_ring_edges_from_a_feature_tree() { + let tree = "\ +ring v0.17.14 +├── rustls v0.23.41 +│ ├── rustls feature \"ring\" +│ │ └── reqwest feature \"__rustls-ring\" +│ └── rustls feature \"ring\" (*) +└── rustls-webpki v0.103.13 + └── rustls-webpki feature \"ring\" +"; + let edges = ring_selecting_edges(tree); + assert_eq!( + edges.into_iter().collect::>(), + [ + "reqwest feature \"__rustls-ring\"", + "rustls feature \"ring\"", + "rustls-webpki feature \"ring\"", + ] + ); + } + + #[test] + fn ignores_package_nodes_and_non_ring_features() { + let tree = "\ +rustls v0.23.41 +├── rustls feature \"aws_lc_rs\" +├── reqwest feature \"rustls-tls\" +└── tokio-rustls feature \"logging\" +"; + assert!(ring_selecting_edges(tree).is_empty()); + } + + #[test] + fn matches_feature_names_exactly_not_by_substring() { + // "tls-ring-extra" must not match the "tls-ring" selector. + let tree = "└── tonic feature \"tls-ring-extra\"\n"; + assert!(ring_selecting_edges(tree).is_empty()); + } +} diff --git a/zainod-heights-from-validator-spec.md b/zainod-heights-from-validator-spec.md new file mode 100644 index 000000000..d78f94b0b --- /dev/null +++ b/zainod-heights-from-validator-spec.md @@ -0,0 +1,170 @@ +# Spec: zainod learns regtest activation heights from the Validator + +Solves zingolabs/zaino#1076. Requested by infras (`zingolabs/infrastructure#278`, +`bump_to_NU6.3`) in coordination with zaino's `ironwood_activation` e2e suite +(zingolabs/zaino#1368). Companion spec on the infras side: +`infras/dev/zaino-ironwood-activation-infra-spec.md` (devtool wallets on +arbitrary regtest heights — "Provider heights"). + +Governing invariant, decided 2026-07-06 (recorded as infras ADR 0003): + +> **The Validator's configured activation heights are authoritative. Every +> other component mirrors them by the strongest mechanism its protocol +> allows.** No component may treat compiled-in or config-file regtest +> heights as chain truth. + +Concretely, per component: + +- **Indexer (this spec)**: queries the Validator's `getblockchaininfo` at + startup and adopts the reported schedule. +- **Wallet client (infras side, same release as the guard lift)**: the + light-client protocol doesn't expose the schedule, so the wallet can't + ask the Validator itself — the harness queries `getblockchaininfo` on + the wallet's behalf and writes the *derived* heights into the wallet's + `activation-heights.toml`. The derivation is enforced at compile time: + the wallet config's regtest variant demands an opaque + `ValidatorHeights`, whose only public constructor is + `WalletNetwork::from_validator(&validator)`. There is no + caller-supplied-heights escape hatch; writing a height vector into a + wallet config is unrepresentable. + +Consequence for zaino's e2e suite: once both sides land, your +`ironwood_activation` fixture heights get typed in exactly one place — the +zebrad launch config. The wallet derives them via +`WalletNetwork::from_validator`; zainod adopts them via this spec; no +test-side constant needs to mirror anything. Expect the zcash_local_net +pin bump to be loud: `ZainodConfig.network` narrows to a payload-free +kind (Mainnet / PubTestnet / Regtest), `ZcashDevtoolConfig::faucet()/recipient()` +take a `WalletNetwork` parameter, and any test that assigned heights into +a wallet config stops compiling — the fix is always to launch the +validator first and derive. + +## Problem + +zainod's regtest activation heights are a compiled-in constant, silently +assumed to match whatever zebrad was launched with: + +- The zainod TOML carries only the string `network = "Regtest"`. + Deserialization inflates it to + `Network::Regtest(ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS)` + (`packages/zaino-common/src/config/network.rs:66`); the constant + (`network.rs:13-26`) documents itself as "must equal zcash_local_net's + `supported_regtest_activation_heights`" — a cross-repo mirror maintained + by hand across three repos (zebra config, zaino constant, infras canonical + heights). +- That `Network` flows into everything downstream as truth: + `StateService::spawn` passes `config.common.network.to_zebra_network()` + into `init_read_state_with_syncer` (`zaino-state/src/backends/state.rs:213-218`) + and into `NodeBackedChainIndex::new` (`state.rs:267-276`); + `FetchService::spawn` does the equivalent (`backends/fetch.rs:122+`). +- When the running zebrad's heights differ, block commitment verification + computes against the wrong upgrade schedule and the chain-index sync loop + dies with `InvalidData("Block commitment could not be computed")` + (`zaino-state/src/chain_index/types/helpers.rs:183`, + `block.commitment(network)`). + +The mismatch is not hypothetical: infras is lifting its client-side guard so +devtool wallets run on **arbitrary** regtest heights, and zaino's own +`ironwood_activation` fixture (`ORCHARD_THEN_IRONWOOD_ACTIVATION_HEIGHTS`) +puts NU6.3 at height 6 with everything else at 1–2. No compiled constant can +ever match caller-chosen heights. The only correct source is the running +zebrad — and it already publishes its schedule: `getblockchaininfo` returns +an `upgrades` map with per-upgrade `activationheight`, which zaino already +parses (`GetBlockchainInfoResponse.upgrades`, +`zaino-fetch/src/jsonrpsee/response.rs:356-359`, round-trip tested at +`response.rs:530+`). Both backends already call the validator at startup +(`get_info` at `state.rs:198`, `get_blockchain_info` in the tip-wait loop at +`state.rs:226`). The handshake exists; the heights are simply never adopted +from it. + +## Requested change + +1. **Learn heights at first contact.** When the configured network kind is + Regtest, each backend fetches `getblockchaininfo` from the validator + **before** constructing anything that consumes a + `zebra_chain::parameters::Network` — the ReadStateService syncer, the + chain index, the mempool source. Build the runtime regtest `Network` from + the reported `upgrades` map. The RPC client is already constructed first + (`state.rs:190-196`), so the ordering is achievable without restructuring. +2. **Learned heights are the only source.** `ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS` + ceases to exist as a truth source. Recommended shape: the *config-level* + network type degrades to a kind (Mainnet / PubTestnet / Regtest, no payload), + and only the *runtime* type constructed after the handshake carries + heights — making pre-handshake height reads unrepresentable. A minimal + variant (populate the existing payload from the handshake before first + use) is acceptable if the type split is too invasive, but then nothing may + read the payload before adoption. Behavior is mandated; shape is + implementer's choice. +3. **An upgrade absent from the validator's map is never-activated.** This + matches zebra's own semantics and the devtool's absent-key encoding. + Do not backfill absent upgrades from any default. +4. **Failure is loud, never defaulted.** If the network kind is Regtest and + the upgrades map cannot be fetched or parsed, startup fails with an error + naming the validator endpoint. Under no circumstances fall back to a + compiled or configured height set — a silently wrong schedule is the + exact failure mode this spec removes. + +## Acceptance + +The fixture that must work end-to-end (as zebrad regtest config heights): + +``` +overwinter = 1, sapling = 1, blossom = 1, heartwood = 1, canopy = 1, +nu5 = 2, nu6 = 2, nu6_1 = 2, nu6_2 = 2, nu6_3 = 6 +``` + +1. **Boundary sync**: launch zebrad regtest with the fixture; launch zainod + against it with only `network = "Regtest"` in its TOML. Mine past height + 6. zainod's chain index syncs across the NU6.3 boundary with no + commitment error, and serves compact blocks for both eras (Orchard + coinbase at heights 2–5, Ironwood from 6). +2. **No-recompile proof**: the same zainod binary and config, pointed at a + zebrad running a *different* schedule (the canonical all-at-2 set), also + syncs — demonstrating heights come from the validator, not the build. +3. **Contract test on the upgrades map**: pin the `getblockchaininfo → + heights` mapping with a test against real zebrad output (live regtest or + a recorded golden response) — not hand-written assertions about what the + RPC is believed to return. In particular, establish from the real output + (not from reasoning) which upgrades zebrad includes in the map (e.g. + whether anything pre-Overwinter appears) and encode exactly that. +4. **No residual truth source**: no non-test code path consumes + `ZEBRAD_DEFAULT_ACTIVATION_HEIGHTS` or any other compiled regtest + schedule as chain truth. Test-only fixtures that *are* their own chain + (mockchain sources, proptest block generators) legitimately keep explicit + heights — there, the test is the validator. + +## Known unknowns to check while implementing + +- **StateService init ordering**: `init_read_state_with_syncer` receives the + network at construction and the ReadStateService validates blocks with it. + Confirm nothing else needs a `Network` before the validator RPC is + reachable; if something does, that is a design conflict to surface, not + paper over. +- **zcashd as validator**: the legacy e2e path (`devtool_zcashd.rs`) drives + zcashd, whose `getblockchaininfo` also reports an `upgrades` map. Prefer + making the adoption path validator-generic so zcashd rides along; if its + map shape differs materially, scope this spec to zebrad and record the + divergence in the zcashd test docs. +- **zaino-testutils**: `live-tests/zaino-testutils/src/lib.rs:497` uses the + constant to *launch zebrad* — that is legitimate harness-side + configuration of the truth source, not adoption of it. Keep the height + set, but move/rename it so it can no longer be mistaken for (or imported + as) zainod's own schedule. +- **Reorg/restart**: heights are learned once at startup. A validator + restarted mid-session with different heights invalidates the indexer's + world; document that zainod must be restarted with its validator (regtest + harnesses already do this), rather than attempting live re-adoption. + +## Out of scope + +- Mainnet / The Public Testnet: compiled zebra network parameters are correct there; no + handshake is needed and none should be added. +- The infras-side client guard lift (Provider heights) — lands on + `infrastructure#278` independently. + +## Delivery + +A rev on a zaino branch that `live-tests` can exercise. Downstream effects +when it lands: zaino#1368's `ironwood_activation` tests gain a zainod that +can actually serve their fixture, and infras' nu6_3-at-6 integration test +(currently `#[ignore]`d naming #1076) flips live.