Skip to content

Zakura header-sync anchor goes stale on an equal-height non-finalized reorg #14

Description

@czarcas7ic

Summary

Zakura header-sync anchors its forward range on ReadRequest::BestHeaderTip. On an equal-height non-finalized reorg, that read returns a stale block hash, so the next header range fails FirstHeaderDoesNotLink against an honest peer that reorged; header-sync transiently bans the peer and lags a block before recovering.

The node recovers, so this is not failing CI — it was surfaced while investigating the nightly Zakura e2e traces, where FirstHeaderDoesNotLink @ tip+1 appears in both passing and failing runs (benign today). Filing as correctness hardening.

Evidence

zebra-state/src/service.rs:1710-1723BestHeaderTip handler:

ReadRequest::BestHeaderTip => {
    let best_disk_header_tip = state.db.best_header_tip();           // finalized DB + Zakura header store
    let verified_block_tip = read::tip(state.latest_best_chain(), &state.db); // non-finalized best chain
    Ok(ReadResponse::BestHeaderTip(match (best_disk_header_tip, verified_block_tip) {
        (Some(header_tip), Some(block_tip)) if block_tip.0 > header_tip.0 => Some(block_tip),
        (Some(header_tip), _) => Some(header_tip),   // EQUAL height: returns the stale disk hash
        (None, block_tip) => block_tip,
    }))
}

At equal height the non-finalized chain's (post-reorg) hash block_tip is discarded in favour of the disk hash.

zebra-state/src/service/finalized_state/zebra_db/block.rs:391-402best_header_tip() reads only the finalized body tip and the Zakura header store, never the non-finalized chain.

The Zakura header store is reconciled with the chain only on finalized transitions (finalized commit via prepare_zakura_header_release_from_committed_block; finalized rollback via delete_zakura_headers_above, whose only non-test caller is the offline zebrad rollback CLI). There is no non-finalized-reorg hook. The serving side (headers_by_height_range) is reorg-correct, so this is purely requester-anchor staleness.

Proposed fix

  1. Read-layer (core): in BestHeaderTip, when the non-finalized block tip is at the same height as the disk header tip, prefer the non-finalized chain hash — it reflects reorgs.
  2. Deeper (header-ahead case): when the Zakura header store has rows above the non-finalized block tip and a non-finalized reorg passes through those heights, reconcile/invalidate the affected rows (a non-finalized analogue of delete_zakura_headers_above). Consensus-adjacent — scope carefully.

Also revisit the header-sync re-anchor recovery in zebra-network/src/zakura/header_sync/reactor.rs (handle_possible_stale_anchor_link_failure), which bails when best_header_tip <= verified_block_tip (the tip-reorg case).

TDD plan

zebra-state test: commit to height N (non-finalized) + seed a Zakura header at N; reorg the best non-finalized chain at N to a different block; assert BestHeaderTip returns the new hash. Fails today, passes after fix (1). A second test covers the header-ahead case for (2).

Notes

Not failing CI; node recovers. The non-blocking block-sync reactor (PR 199) did not introduce this — the read handler and header store predate it; the faster reactor only widens the window. AI-assisted analysis (Claude Code); confirm with the failing test above before fixing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions