Skip to content

fix(operator): wait for inclusion on rollup-message injection - #39

Open
devon-n wants to merge 2 commits into
trilitech:mainfrom
devon-n:fix/operator-inject-wait-inclusion
Open

fix(operator): wait for inclusion on rollup-message injection#39
devon-n wants to merge 2 commits into
trilitech:mainfrom
devon-n:fix/operator-inject-wait-inclusion

Conversation

@devon-n

@devon-n devon-n commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Problem

Shields intermittently fail with the operator recording:

cannot be added because the mempool already contains a conflicting operation

Both inject_direct_message call sites (process_submission direct inbox, and the DalPointer submit after attestation) used octez-client -w none — fire-and-forget. The op enters the mempool but the call returns before it is baked, releasing advance_lock while the on-chain counter is still stale. octez-client derives each op's counter from on-chain state, so the next injection from the operator key reuses the same counter and L1 rejects it. The shield is marked failed (not retried); the next one self-recovers. ~3% of injections.

Fix

1. Wait for inclusion. Both sites now use -w 1, so each op is baked (counter advances) before advance_lock releases — the conflict can't occur.

2. Bounded wait. -w 1 blocks under the lock, so run_command_collect_output gets a 180s ceiling (capture to temp files, poll try_wait, kill on expiry). Without it, one never-confirmed op would hang the operator indefinitely.

3. Idempotent on post-injection error. octez-client prints the op hash the moment it injects, before waiting. If the wait then errors (timeout / RPC blip) the op already reached the node, so both call sites record it as SubmittedToL1 rather than Failed / re-injecting — otherwise the reconciler would inject a duplicate pointer op next tick.

Cost / assumptions

  • Adds ~1 block (~8–16s) of latency per injection, held under advance_lock. Acceptable for the operator's throughput; removes the race deterministically.
  • Correct under the operator's existing single-writer assumption (one process signs for the operator key). A second signer for the same key would reintroduce the counter race.
  • (3) records an injected op as submitted without verifying inclusion; a genuinely dropped op is still caught by the caller's settlement timeout. On-chain inclusion verification is a possible future follow-up.

Test

cargo test -p tzel-services --bin tzel-operator → 33/33; build + clippy clean. New: pointer_inject_error_with_op_hash_is_recorded_not_retried, run_command_times_out_and_keeps_partial_output, and a -w 1 assertion on attested_dal_submission_sends_pointer_message.

Both inject_direct_message call sites used `-w none` (fire-and-forget): the op
was injected into the mempool but the call returned before it was baked, so it
released advance_lock while the on-chain counter was still stale. octez-client
computes the next op's counter from on-chain state, so the following injection
from the operator key reused the same counter and L1 rejected it:

  cannot be added because the mempool already contains a conflicting operation

That marked the shield `failed` (not retried); the next shield self-recovered.

Switch both sites to `-w 1` so each op is baked before advance_lock releases —
the counter always advances before the next injection is built, so the conflict
can't occur. Costs ~1 block of latency per injection held under the lock.

Test: attested_dal_submission_sends_pointer_message now asserts the pointer
inject waits (`-w 1`, not `none`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devon-n
devon-n force-pushed the fix/operator-inject-wait-inclusion branch from cc8b9b6 to b9ce2df Compare August 6, 2026 10:46
Follow-up to waiting on inclusion (-w 1). Two hardening changes:

Timeout: `-w 1` blocks until the op is baked, under advance_lock. Without a
bound, one op that never confirms would hang the injection forever and wedge
the whole operator (reconciler + every submit stall on the lock). Add a 180s
ceiling in run_command_collect_output: capture stdout/stderr to temp files
(pipes could deadlock on a long wait, and files let us keep partial output),
poll try_wait to a deadline, kill on expiry, and return a timeout error that
still carries whatever octez-client printed.

Idempotency: octez-client prints the op hash the moment it injects, before it
waits. So an injection can reach the node and *then* error (a confirmation
timeout or RPC blip). Both inject_direct_message call sites now check for an op
hash in the error: if present, the op is on the node, so record it as
SubmittedToL1 instead of Failed / instead of propagating the error. This stops
the reconciler from injecting a duplicate pointer op on the next tick. Inclusion
is not verified here — a genuinely dropped op is still caught by the caller's
settlement timeout.

Tests: pointer_inject_error_with_op_hash_is_recorded_not_retried (idempotency),
run_command_times_out_and_keeps_partial_output (timeout + partial-output). 33/33.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant