fix(zakura): keep Zakura reactors alive when the stall watchdog falls back to legacy sync

CHANGELOG.md

@@ -44,6 +44,66 @@ and this project adheres to [Semantic Versioning](https://semver.org).
 - Fixed a near-tip sync restart loop when a timed-out `AwaitUtxo` lookup in the
   transaction verifier was converted to `InternalDowncastError` instead of a
   missing transparent input.
+- Fixed a permanent header-sync wedge where a fork-recovery re-commit left
+  stale rows in the stored difficulty-validation window: every extension
+  range was then rejected with `InvalidDifficultyThreshold`, each rejection
+  scored and disconnected the (honest) serving peer, and the wedge survived
+  restarts because the stale rows are on disk (observed live at height
+  4148005 for hours). Contextual validation failures are now classified as
+  `ContextMismatch`: they no longer score peers, and repeated identical
+  failures from independent peers feed the fork-recovery walk-back — each
+  deepening re-commits a longer range, rewriting the poisoned window until
+  the chain extends again.
+- Zakura block-sync liveness disconnects are now deferred while the local
+  apply pipeline holds unfinished bodies: "no accepted block progress" cannot
+  be blamed on a peer while the node's own commits are not landing. During a
+  ~45-minute local commit stall the old behavior exiled healthy peers one by
+  one into the no-progress cooldown. A genuinely dead peer is still caught as
+  soon as the pipeline drains. Deferrals are counted in
+  `sync.block.liveness.deferred_local_stall`.
+- Fixed a chain-tip reset (fork-recovery invalidation, `invalidateblock`)
+  being silently discarded by the Zakura chain-tip mirror. The mirror maxed
+  the reset tip against a `latest_chain_tip` watch that may not have observed
+  the reset yet, republished the stale higher tip, and turned the downstream
+  `VerifiedReset` into a no-op — the block-sync sequencer then pinned one
+  block above the real tip, never requested the missing parent, and every
+  body commit timed out until restart. On `TipAction::Reset` the action's tip
+  is now published verbatim; the anti-regression maxing only applies to
+  `Grow`.
+- Fixed Zakura block-sync peers being serially disconnected and parked at the
+  chain tip by a stale liveness deadline. When the download floor passed a
+  request because other peers' deliveries satisfied its heights, the floor GC
+  removed the request but only cleared the liveness deadline for peers whose
+  last delivery was newer than their last request — a healthy peer with an
+  older delivery kept a deadline it could no longer answer, was disconnected
+  for "no accepted block progress" with zero outstanding requests, and parked
+  in the 180-second no-progress cooldown. At tip this exiled peers one by one
+  until the block-sync peer set collapsed (reproduced live: 5 → 0 peers within
+  ~25 minutes of a fleet-wide restart, wedging body sync when the next gap
+  formed). Heights satisfied below the floor now always clear an idle peer's
+  deadline; genuinely unresponsive peers are still caught by request timeouts,
+  which deliberately keep the deadline armed. The liveness disconnect log was
+  also promoted to info.
+- After a legacy fallback, the dual-stack watchdog now re-promotes Zakura to
+  the body-sync driver once legacy `ChainSync` is caught up and stable (three
+  consecutive exhausted sync rounds with the tip advancing by at most two
+  blocks). Fallback was previously permanent per process lifetime, so every
+  min-difficulty burst demoted one more node's Zakura sync to a serve-only
+  bridge until restart. The hand-back reuses the apply-gate commit barrier in
+  reverse and is counted in `sync.zakura.repromoted`.
+- The dual-stack Zakura stall watchdog no longer shuts down the Zakura header-
+  and block-sync reactors when it falls back to legacy `ChainSync`. Killing
+  them turned every fallback into a fleet-wide Zakura outage: the fallback
+  node — often the only peer with working block ingest — stopped advertising
+  statuses, serving headers and bodies, and forwarding `NewBlock`s, which
+  pinned every Zakura peer's frontier and starved zakura-only nodes until a
+  manual restart. Legacy `ChainSync` now resumes as the body-sync driver while
+  the Zakura reactors stay alive, follow local commits through the chain-tip
+  mirror, and keep serving the Zakura network as a bridge. Fallback engagement
+  is counted in `sync.zakura.legacy_fallback.engaged`. Inbound block-gossip
+  ingest failures and queue drops — previously silent at default log levels,
+  which made tip-ingest freezes unattributable — are now logged at info and
+  counted (`gossip.ingest.failed.count`, `gossip.ingest.duplicate.count`).
 - Fixed Zakura nodes gossiping and following side-chain blocks. An inbound
   `NewBlock` that committed as a side chain (for example a testnet
   min-difficulty branch) still advanced the node's Zakura header and verified

zebra-consensus/src/block.rs

@@ -303,6 +303,7 @@ where
             let mut block_miner_fees = Ok(Amount::zero());
 
             use futures::StreamExt;
+            let tx_checks_started = std::time::Instant::now();
             while let Some(result) = async_checks.next().await {
                 tracing::trace!(?result, remaining = async_checks.len());
                 let response = result
@@ -323,6 +324,18 @@ where
                 }
             }
 
+            // A slow drain here means transaction verification (usually a UTXO
+            // await on an out-of-order parent) held the block; this stage log
+            // attributes driver-side commit timeouts to their pipeline stage.
+            if tx_checks_started.elapsed() > std::time::Duration::from_secs(20) {
+                tracing::warn!(
+                    ?height,
+                    ?hash,
+                    elapsed = ?tx_checks_started.elapsed(),
+                    "slow transaction verification stage while committing block"
+                );
+            }
+
             // Check the summed block totals
 
             if sigops > MAX_BLOCK_SIGOPS {
@@ -378,13 +391,25 @@ where
                 };
             }
 
-            match state_service
+            let state_commit_started = std::time::Instant::now();
+            let state_commit_result = state_service
                 .ready()
                 .await
                 .map_err(|source| VerifyBlockError::StateService { source, hash })?
                 .call(zs::Request::CommitSemanticallyVerifiedBlock(prepared_block))
-                .await
-            {
+                .await;
+            // A slow response here means the block sat in the state's
+            // parent-waiting queue or behind the write task; this stage log
+            // attributes driver-side commit timeouts to their pipeline stage.
+            if state_commit_started.elapsed() > std::time::Duration::from_secs(20) {
+                tracing::warn!(
+                    ?height,
+                    ?hash,
+                    elapsed = ?state_commit_started.elapsed(),
+                    "slow state commit stage while committing block"
+                );
+            }
+            match state_commit_result {
                 Ok(zs::Response::Committed(committed_hash)) => {
                     assert_eq!(committed_hash, hash, "state must commit correct hash");
                     Ok(hash)

zebra-consensus/src/transaction.rs

@@ -743,14 +743,33 @@ where
 
                     utxo
                 } else {
+                    let lookup_started = std::time::Instant::now();
                     let response = state
                         .clone()
                         .oneshot(zebra_state::Request::AwaitUtxo(*outpoint))
                         .await
                         .map_err(|boxed_error| match boxed_error.downcast::<Elapsed>() {
-                            Ok(_) => TransactionError::TransparentInputNotFound,
+                            Ok(_) => {
+                                // A timed-out lookup held this transaction's whole
+                                // block for UTXO_LOOKUP_TIMEOUT; chains of these
+                                // serialize into multi-minute commit stalls that
+                                // are otherwise unattributable from logs.
+                                tracing::warn!(
+                                    ?outpoint,
+                                    elapsed = ?lookup_started.elapsed(),
+                                    "transparent input UTXO lookup timed out"
+                                );
+                                TransactionError::TransparentInputNotFound
+                            }
                             Err(boxed_error) => TransactionError::from(boxed_error),
                         })?;
+                    if lookup_started.elapsed() > std::time::Duration::from_secs(20) {
+                        tracing::warn!(
+                            ?outpoint,
+                            elapsed = ?lookup_started.elapsed(),
+                            "slow transparent input UTXO lookup delayed transaction verification"
+                        );
+                    }
 
                     if let zebra_state::Response::Utxo(utxo) = response {
                         utxo