Skip to content

sync::initialize busy-loops with no backoff when suggested scan range exceeds current chain tip #636

Description

@pacu

Summary

The initialize step of the sync task in zallet-core/src/components/sync.rs can enter a tight busy-loop with no backoff, burning a full CPU core and writing dozens of log lines per second indefinitely.

Details

The initialize loop (L231-341) is structured as:

let (current_tip, starting_boundary) = loop {
    let chain_view = chain.snapshot().await?;
    let current_tip = chain_view.tip().await?;
    info!("Latest block height is {}", current_tip.height());
    db_data.update_chain_tip(current_tip.height())?;
    let starting_boundary = update_boundary(BlockHeight::from_u32(0), current_tip.height());
    let scan_range = match db_data.suggest_scan_ranges()? ... ;
    match steps::scan_blocks(chain_view, db_data, params, &scan_range, &decryptor, shutdown_height).await {
        Ok(flow) if flow.is_break() => break (current_tip, starting_boundary),
        Ok(_) => {}                                       // no sleep, immediate re-loop
        Err(error) if is_retryable(&error) => { time::sleep(REORG_RETRY_BACKOFF).await; }
        Err(error) => return Err(error),
    }
};

The retryable-error arm backs off via REORG_RETRY_BACKOFF (200ms), but the "made no progress" arm (Ok(_) => {}) does not. If scan_range ends up above the current chain tip, scan_blocks has nothing to scan, returns Ok without breaking, and the loop immediately restarts with no delay and no state change that could resolve the condition — spinning until the chain view happens to advance past the range on its own.

Reproduction

Environment: Zallet 0.1.0-beta.1, mainnet.

The following two log lines repeated continuously, ~40 times/second:

19:05:39.948  INFO initialize: zallet_core::components::sync: Latest block height is 3413617
19:05:39.950  INFO initialize: zallet_core::components::sync::steps: Scanning blocks Historic(3413618..3414133)
19:05:39.951  INFO initialize: zallet_core::components::sync: Latest block height is 3413617
19:05:39.952  INFO initialize: zallet_core::components::sync::steps: Scanning blocks Historic(3413618..3414133)

Note the suggested scan range (3413618..3414133) starts above the tip height the same iteration just logged (3413617) — there is nothing in that range to scan.

Likely cause (inference, not confirmed): the wallet DB had already recorded a higher chain tip (~3414133) from an earlier run, while the current chain view (Zebra's read-only secondary, serving finalized on-disk state) only serves up to 3413617 — i.e. the chain view can lag or regress relative to what the DB has already observed. Regardless of root cause, any scan range suggestion starting above current_tip reproduces the same busy-loop.

Impact

Burns a full CPU core and writes ~40 log lines/second to journald for as long as the condition holds. Not data-threatening, but on a production host it is indistinguishable from a hang, and the log volume is significant over time.

Suggested fix

Any of:

  1. When scan_range.start > current_tip, skip scanning and await a chain-tip change instead of immediately retrying.
  2. Apply a backoff on the Ok(_) arm when zero blocks were actually scanned (mirroring the existing retryable-error backoff).
  3. Clamp suggested scan ranges to current_tip before selecting one, so an above-tip range is never handed to scan_blocks in the first place.

Reported by a mining pool operator running Zallet in production.

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions