Skip to content

Reduce flakiness in block checks#268

Merged
simonlechner merged 4 commits into
mainfrom
simon/reduce-flakiness
Jul 23, 2026
Merged

Reduce flakiness in block checks#268
simonlechner merged 4 commits into
mainfrom
simon/reduce-flakiness

Conversation

@simonlechner

Copy link
Copy Markdown
Contributor

This PR refactors the blocks rolling and block height checks to be less flaky on CI infrastructure. These changes surfaced an issue the norma configuration when only a single node is present in the network.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce CI flakiness in network progress checks by (a) making the block-height check sample nodes “at the same time” and (b) making the blocks-rolling check rely on forward observation rather than historical samples, and it also introduces a “network bootstrap” signal to avoid single-node startup deadlocks around doublesign protection.

Changes:

  • Add NETWORK_BOOTSTRAP plumbing (Go → Docker env → run_sonic.sh) to conditionally disable DoublesignProtection only when bootstrapping a brand-new single-validator network.
  • Refactor blocksRolling to observe progress over a time window (with updated tests).
  • Read block heights concurrently in blockHeight checker (with updated tests) to avoid false lag detection due to sequential sampling.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
scripts/run_sonic.sh Gate DoublesignProtection disabling behind NETWORK_BOOTSTRAP for single-validator bootstrap.
scenarios/rules_upgrades/allegro_to_brio_mixed_versions.yml Update scenario description and remove the v2.1.5 node step.
driver/norma/run.go Adjust node log dumping output formatting and severity.
driver/node/opera.go Add NetworkBootstrap to node config and pass it to the container env as NETWORK_BOOTSTRAP.
driver/network/local/local.go Infer and set NetworkBootstrap when creating nodes.
driver/checking/blocks_rolling.go Replace historical sliding-window logic with forward observation window logic.
driver/checking/blocks_rolling_test.go Rewrite tests to match the new forward-observation behavior.
driver/checking/block_height.go Fetch block heights concurrently to reduce skew between node samples.
driver/checking/block_height_test.go Update mocks/expectations for concurrent RPC calls; add a genesis-within-slack test.

Comment thread driver/checking/block_height.go
Comment thread scripts/run_sonic.sh Outdated
Comment thread driver/network/local/local.go
Comment thread driver/checking/blocks_rolling.go Outdated
@simonlechner
simonlechner requested a review from LuisPH3 July 23, 2026 09:30
LuisPH3
LuisPH3 previously approved these changes Jul 23, 2026
Comment thread driver/checking/block_height.go Outdated
@simonlechner
simonlechner merged commit d1ffc24 into main Jul 23, 2026
3 checks passed
@simonlechner
simonlechner deleted the simon/reduce-flakiness branch July 23, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants