Wallet crash-loops permanently when a data-requested transaction is reorged out

[s1ckoleg]

Version: zallet v0.1.0-beta.1, zebrad 6.0.0, testnet (post-NU6.3)

Summary

When a transaction the wallet has a pending TransactionDataRequest for is reorged out
of the best chain, the data-requests sync task aborts the whole process. Because the
failed request persists in the wallet DB and is re-serviced first on every startup, the
wallet enters a permanent crash-loop (systemd start-limit-hit after a few restarts).
The only recovery we found is a manual repair truncate-wallet to a height below
the reorged block.

Reproduction

1. zebrad + zallet (zaino backend), miner_address = wallet-owned t-addr.
2. Mine blocks to the wallet; keep mining until one of your blocks is reorged out
   (easy on testnet; on mainnet this is any orphaned block).
3. The wallet queues TransactionDataRequest::GetStatus/Enhancement for the coinbase
   txid while the block is in the best chain; after the reorg, zebra's
   getrawtransaction returns RPC -5.

Observed behavior

INFO zallet_core::commands::start: Wallet data-requests sync task exited
wallet_sync_result=Err(Error(Context { kind: Sync, source: Some(Chain(Backend(ChainIndexError {
  kind: InternalServerError,
  message: "critical error in backing block source: could not fetch transaction data:
            unexpected error response from server: RPC Error (code: -5):
            No such mempool or main chain transaction" })))
INFO zallet_core::commands::start: Exiting Zallet because an ongoing task exited; asking other tasks to stop
...
zallet.service: Failed with result 'start-limit-hit'.

Restarts do not help: the same persisted request fails first every time.

[linear]

COR-1556