migrate-zcashd-wallet: transparent change-address gap limit gets exhausted on import, permanently blocking z_sendmany

[pacu]

Summary

After running migrate-zcashd-wallet, z_sendmany permanently fails on essentially any migrated wallet that has meaningful transparent history, with:

{'code': -4, 'message': 'Failed to propose transaction: The underlying datasource
produced the following error: The proposal cannot be constructed until a transaction
with outputs to a previously reserved transparent change address has been mined.
The address at index 10 could not be safely reserved.'}

Manually sending dust to the named address doesn't fix it — it just moves the blocked index forward by the gap limit (5) each time, requiring dozens of manual, individually-mined transactions to walk past the whole backlog.

Root cause

Verified against the exact crate revisions pinned in zallet's Cargo.lock (zcash_client_sqlite 0.22.0-rc.1, zcash_client_backend 0.24.0-rc.1, zcash_keys 0.15.0, zewif 1.0.0-rc.2, zewif-zcashd 0.1.0-rc.2):

1. zewif-zcashd never sets exposed_at_height on any transparent address it exports. migrate/addresses.rs::convert_transparent_addresses iterates every keypair in the zcashd wallet (wallet.keys().keypairs()) — i.e. the entire keypool, both addresses that were actually handed out/used and ones that were merely pre-generated reserve keys sitting unused — and includes all of them as account.addresses() with no per-address exposure height.

2. zcash_client_sqlite::zewif::import_wallet's mark_addresses_exposed (src/zewif.rs:1102-1118, feeding from import_account's per-address loop at src/zewif.rs:1359-1373) falls back to the account's birthday height whenever exposed_at_height() is None:

   let exposure_height = address
       .exposed_at_height()
       .map_or(birthday.height(), |h| BlockHeight::from(u32::from(h)));

   This applies uniformly to every recognized address regardless of transparent key scope (external/receiving vs. internal/change), and regardless of whether the address was ever actually used on-chain.

3. The transparent gap-limit machinery that z_sendmany relies on to pick a fresh change address (zcash_client_sqlite::wallet::transparent::find_gap_start) only slides its reservation window past addresses with a mined first-use (v_address_first_use.first_use_height IS NOT NULL). Being merely "exposed" does not count.

4. The default internal (change) gap limit is only 5 (zcash_keys::keys::transparent::gap_limits::GapLimits::default()). zcashd keeps a keypool of ~100 pre-generated addresses per chain by default, and any wallet with meaningful history accumulates far more than 5 change addresses over its lifetime.

Net effect: migration marks dozens of change-branch addresses "exposed" at once, none of which have (or, for unused keypool reserve keys, ever will have) a mined first-use. The gap window gets pinned at/near index 0 forever, so reserve_next_n_internal_addresses can never find a free slot, and z_sendmany fails with ReachedGapLimit on the very first send that needs transparent change.

This also explains the "index 10 → index 15" progression in the reporter's follow-up: bad_index = gap_start + gap_limit, and gap_limit=5, so each individually-mined address only advances the window by exactly 5.

Not related to zcash/librustzcash#2594 (that issue is about making the ReachedGapLimit error type backend-agnostic for shared conformance tests — an API/architecture concern, not this behavioral bug).

Reproduction

Any zcashd wallet with more than 5 historical transparent change outputs, migrated via migrate-zcashd-wallet, then any z_sendmany call that needs to construct transparent change (including a fully-shielded send whose change happens to route through the transparent legacy pool). A minimal in-repo reproduction (no zcashd wallet fixture needed) is included in the linked PR: reserving internal_gap_limit change addresses without ever mining a transaction to any of them, then attempting to reserve one more, reproduces the exact ReachedGapLimit failure.

Impact

Blocks the core "migrate from zcashd, then transact" flow that beta users are actively relying on. Affects essentially all migrated wallets with prior transparent activity, not an edge case.

Suggested fix

The root cause lives upstream, in zewif-zcashd's blanket keypool export and/or zcash_client_sqlite's blanket birthday-height exposure fallback — neither crate is vendored in this repository, so a full fix needs upstream coordination (e.g. not exposing internal/ephemeral-scope addresses that have no independent evidence of use, or not exporting never-issued keypool reserve keys as "addresses" at all).

As a scoped, immediately-usable mitigation entirely within this repository, the linked PR adds note_management.transparent_{external,internal,ephemeral}_gap_limit config options, threaded through to WalletDb::with_gap_limits. Important caveat verified by the PR's test: a wallet's transparent address rows for a given key scope are pre-generated once, at account-creation time, using whatever gap limit is active then — later reservations only select among those existing rows. So this option must be set before running migrate-zcashd-wallet (to the size of the source wallet's zcashd keypool); it cannot retroactively unstick a wallet that has already hit the lockout. A wallet that's already stuck must be re-migrated into a fresh data directory with the option set first.

/cc @nuttycom

[linear]

COR-1588

[nullcopy]

Oh I bet this has to do with manual address exposure marking. During recovery, we mark receive addresses as exposed if they appear in stored transactions, but we might not check and mark the change addresses the same way. Let me dig into this.

[nullcopy]

Actually, no, this is due to the opposite problem. We are overeagerly marking addresses as exposed, which drains the gap limit. I have a proposed fix almost ready