ML0 StateChannelBinarySender retries indefinitely despite GL0 accepting snapshots

[ottobot-ai]

Description

ML0's StateChannelBinarySender keeps retrying the same binary hash even after GL0 has accepted and included the state channel snapshot in its global snapshots.

Evidence

• ML0 logs show repeated [Queue] Processing N unsent binaries with growing N
• Same binary hash retried: f7bc1f81d9569441ce954255a10dcd40a9fa234659983f661d13f621ec807598
• GL0's /global-snapshots/latest confirms stateChannelSnapshots contains the metagraph entry (keyed by token ID)
• GL0 ordinal: 5415, ML0 ordinal: 3347

Impact

• Cosmetic (snapshots ARE landing in GL0), but the retry queue grows unbounded
• Likely ML0 not receiving/processing GL0's acknowledgment response

Environment

• Tessellation v4.0.0-rc.10
• Local cluster (3x GL0, 3x ML0, 3x DL1)

Likely Root Cause

Framework-level issue in tessellation's binary sender acknowledgment handling. To be investigated from tessellation side.

[ottobot-ai]

Root Cause Analysis

Bug: BinaryPoster.pickPeerAndSend returns early without calling binaryTracker.markAsSent() when PeerSelector determines that another ML0 node (not selfId) should be the deterministic sender.

Effect: sendsSoFar stays 0 for delegated binaries. processNormalMode filters for sendsSoFar === 0 ("unsent") every 5 seconds, so those binaries are re-enqueued indefinitely even after GL0 has already confirmed them via a peer that actually sent them.

Why GL0 ordinal is ahead: GL0 was accepting snapshots (sent by the elected deterministic peer), but ML0's confirm() hadn't yet pruned the entries — because ML0 was still behind on processing global snapshots at the time the bug manifested.

Fix

Call binaryTracker.markAsSent(binary.hash) in the delegation branch before returning. Retry mode (which ignores sendsSoFar) will still re-send if the delegated peer fails.

PR with fix + regression test: scasplte2/tessellation#2

Once that merges and a new tessellation/metakit release is cut, bump V.metakit in project/Dependencies.scala to pick up the fix.

[ottobot-ai]

CodeBot triage check — 2026-03-09

Upstream fix PR scasplte2/tessellation#2 is OPEN and awaiting review/merge. No CI checks configured on that fork yet.

Current status:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() call in delegation branch
• Fix + regression test implemented in tessellation#2
• Blocker: tessellation#2 needs merge + new metakit release cut

Next step after tessellation#2 merges:
Bump V.metakit in project/Dependencies.scala to pick up the fix — this issue can then be closed.

[ottobot-ai]

CodeBot triage check — 2026-03-10

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet.

Status unchanged:

• Root cause confirmed, fix + regression test in tessellation#2
• Blocker: tessellation#2 needs merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-10 (10:28 AM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned.

Status unchanged:

• Root cause confirmed, fix + regression test in tessellation#2
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-10 (2:28 PM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned.

Status unchanged:

• Root cause confirmed, fix + regression test in tessellation#2
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-10 (10:22 PM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (2:23 AM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (6:23 AM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (10:23 AM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (2:23 PM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (6:23 PM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.

[ottobot-ai]

CodeBot triage check — 2026-03-11 (10:24 PM CT)

Upstream fix PR scasplte2/tessellation#2 remains OPEN — no merge yet, no reviewers assigned, no CI configured on the fork.

Status unchanged:

• Root cause confirmed: BinaryPoster.pickPeerAndSend missing markAsSent() in delegation branch
• Fix + regression test implemented in tessellation#2 (all 14 tests passing)
• Blocker: tessellation#2 needs review/merge + new metakit release

Action needed: Review/merge tessellation#2. Once a new metakit version is released, bump V.metakit in project/Dependencies.scala to pull in the fix and close this issue.