Skip to content

Classify failures, report block height, bound whole steps - #19

Merged
emilbob merged 1 commit into
masterfrom
feat/failure-kinds-block-height-deadlines
Jul 25, 2026
Merged

Classify failures, report block height, bound whole steps#19
emilbob merged 1 commit into
masterfrom
feat/failure-kinds-block-height-deadlines

Conversation

@emilbob

@emilbob emilbob commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Four gaps found reviewing the merged code. One of them contradicted a rule this project had already adopted.

1. The JSON couldn't say why it failed

Every failure was ok: false plus a prose error string. A monitor wanting to page differently for "unreachable" vs "too few peers" had to regex it — which is exactly what #16 avoided for the genesis hash and then didn't do for the failure itself.

Errors are now a ProbeError carrying a Failure kind, surfaced as a stable failure field. error stays for humans and is free to be reworded; failure is the part safe to alert on.

kind meaning
config the command line asked for something impossible
connect the connection was never established
timeout a wait expired
transport the connection broke part-way through
protocol the node answered with something unusable
rpc_error the node refused a call the probe needs
genesis_mismatch wrong chain
requirement_unmet reachable and correct, but failed a --require-* bar

connect and requirement_unmet are both "exited 1", but one means your monitoring can't see the node and the other means the node is up and telling you it's unhealthy. Those want different alerts.

2. It never reported block height

The report couldn't answer "is this node stuck?" — the most common question asked of a node. chain_getHeader joins the pipelined query step and lands as best_block.

No staleness check: a Substrate header carries no timestamp, so the honest thing is to report the number and let the caller compare across runs or against a second node. Claiming to judge freshness would have meant inventing a block time.

3. Timeouts bounded each frame, not each operation

The read loops skip frames they don't recognise, so a node emitting one unrelated frame every 9s would keep a 10s probe alive forever — every individual wait inside the limit, the total unbounded. A Deadline is now set once per step and every read inside draws down the same clock. --follow gets a fresh budget per header, which is the one place a long wait is legitimate.

There's a regression test that spams unrelated frames every 20ms; it hangs on the old per-frame bound, so it's wrapped in an outer timeout to fail rather than stall the suite.

4. Timeouts were compiled in

--connect-timeout, --rpc-timeout, --head-timeout. Zero is rejected — an instant-fail probe is never what someone means by asking for one.

Verification

Each kind produced against real conditions rather than asserted:

$ ... wss://rpc.polkadot.io --json                       {"ok":true,"failure":null,"best_block":32264078,"peers":69}
$ ... --genesis-hash <kusama>                            {"ok":false,"failure":"genesis_mismatch","genesis_hash":"0x91b1…"}
$ ... --require-peers 999999                             {"ok":false,"failure":"requirement_unmet","peers":75,"best_block":32264078}
$ ... --node-address ws://127.0.0.1:9                    {"ok":false,"failure":"connect"}
$ ... --node-address ws://10.255.255.1:9944 --connect-timeout 1   {"ok":false,"failure":"timeout"}
$ ... --genesis-hash nothex                              {"ok":false,"failure":"config"}

(The nulls are jq rendering an absent key — the field is genuinely omitted on success, pinned by a test.)

Tests 24 → 29. cargo fmt --check, cargo clippy --all-targets, cargo doc and the live Polkadot test all pass.

🤖 Generated with Claude Code

Four gaps, one of which contradicted a rule this project had already
adopted.

1. Every failure was `ok: false` plus a prose string, so a monitor that
   wants to page differently for "unreachable" and "too few peers" had
   to regex it — exactly what #16 avoided for the genesis hash and then
   didn't do for the failure itself. Errors are now a ProbeError
   carrying a Failure kind, surfaced as a stable `failure` field:
   config, connect, timeout, transport, protocol, rpc_error,
   genesis_mismatch, requirement_unmet. `error` stays for humans and is
   free to be reworded; `failure` is the part safe to alert on.

2. The report never carried the node's height, so it could not answer
   "is this node stuck?" — the most common question asked of a node.
   chain_getHeader joins the pipelined query step and lands as
   `best_block`. No staleness check: a Substrate header carries no
   timestamp, so the honest thing is to report the number and let the
   caller compare across runs or against another node.

3. Timeouts bounded each frame, not each operation. The read loops skip
   frames they don't recognise, so a node emitting one unrelated frame
   every 9s would keep a 10s probe alive forever — every wait short, the
   total unbounded. A Deadline is now set once per step and every read
   inside draws down the same clock; --follow gets a fresh budget per
   header, which is the one place a long wait is legitimate.

4. The timeouts were compiled in. --connect-timeout, --rpc-timeout and
   --head-timeout expose them, rejecting 0 since an instant-fail probe
   is never what someone means.

Verified against wss://rpc.polkadot.io: healthy run reports best_block
32264078 and no failure key; a genesis mismatch, an unmeetable
--require-peers, a refused connection, a bad --genesis-hash and a
1s connect timeout each report their own kind. Tests 24 -> 29,
including a chattering-node test that would hang on the old per-frame
bound.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@emilbob
emilbob merged commit 679d9a7 into master Jul 25, 2026
1 check passed
@emilbob
emilbob deleted the feat/failure-kinds-block-height-deadlines branch July 25, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant