Skip to content

Loop guard across pool members: the trip deletes peers' backlog without interrupting their turns, and the counters lose increments #1038

Description

@zfy0701

The loop guard is global in its destruction and local in its enforcement, and its counters lose increments under concurrent members.

What is wrong

The trip destroys peers' work but never stops them. daemon.ts:13696:

const purged = this.purgeLoopScopeInbox(scope)
for (const [key, entry] of this.activeGateEntries) {  }
for (const pending of this.pending.values()) {  }
for (const [key, queued] of this.serialQueue) {  }

purgeLoopScopeInbox (daemon.ts:13664) scans listInboxBySessionKeyFifo() — the whole install-wide inbox — and calls store.removeInbox(row.id) on every match, including rows queued on other members whose liveInboxIds this process cannot see. The interruption half that follows only walks this process's own maps.

The counters are a read-modify-write with an absolute-value upsert. local-store.ts:3787 then :3811:

const current = this.getLoopGuard(scopeKey)

const totalCount = totalWindowExpired ? 1 : current.totalCount + 1

ON CONFLICT(scopeKey) DO UPDATE SET
  totalCount=excluded.totalCount,
  automaticCount=excluded.automaticCount, 

scopeKey is a conversation, not an agent, so two agents in one Slack channel held by two members increment concurrently: both read n, both write n + 1, one turn goes uncounted. Wrapping it in recordLoopGuardTurnForInbox's BEGIN (:3855) does not save it, because the sync facade rewrites BEGIN IMMEDIATE to a plain BEGIN (packages/daemon/src/store/postgres-store-worker.js:11), i.e. READ COMMITTED — the second writer blocks on the row lock and then overwrites with its stale-derived value.

The failing scenario

A runaway conversation involves two agents held by two members. The counters undercount exactly when the loop is fastest, because that is when concurrency is highest — the circuit trips late or not at all. When it does trip on member A, A deletes member B's durable backlog rows out from under B's in-memory queue, but B's live ACP turns are never interrupted, so the loop keeps running on B. The durable loop_guard latch does block new admissions on B, so the outcome is a partial, confusing failure: work disappears, the loop continues, and the logs on the two members tell different stories.

Smallest correct fix

  • purgeLoopScopeInbox: skip rows whose agentId this member does not hold.
  • Propagate the trip so peers interrupt their own turns — a durable loop_guard.trippedAt epoch each member reads on its idle sweep is enough; it does not need a new frame.
  • Make the counter update relative and window-aware in SQL rather than derived in JS, e.g. totalCount = CASE WHEN loop_guard.windowStartedAt <= excluded.windowStartedAt - @windowMs THEN 1 ELSE loop_guard.totalCount + 1 END, with RETURNING so the verdict is computed from what was actually stored.

Separately worth fixing while here: the BEGIN IMMEDIATEBEGIN rewrite at postgres-store-worker.js:11 silently downgrades every transaction that was written against SQLite's exclusive-writer semantics. Mapping it to BEGIN ISOLATION LEVEL REPEATABLE READ (with serialization-failure retry at the facade) would make the keyword keep meaning what its callers assume, instead of leaving each SELECT-then-write inside a BEGIN individually responsible for its own CAS.

Part of the "shared by accident" half of the class tracked by #955.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions