Skip to content

fix(daemon): scope the loop-guard trip to the member and make its counters atomic - #1069

Merged
zfy0701 merged 2 commits into
mainfrom
fix/loop-guard-shared-store
Aug 16, 2026
Merged

fix(daemon): scope the loop-guard trip to the member and make its counters atomic#1069
zfy0701 merged 2 commits into
mainfrom
fix/loop-guard-shared-store

Conversation

@zfy0701

@zfy0701 zfy0701 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1038. On a daemon pool the agent loop guard was global in its destruction and
local in its enforcement, and its counters lost increments under concurrent members.
Three defects, one seam:

  • The trip destroyed a peer's work but never stopped it. purgeLoopScopeInbox
    scanned the whole install-wide inbox and deleted every row in the scope, including
    rows queued on members whose in-memory state this process cannot see. The
    interruption half that followed only walked this process's own maps.
  • The counters were a read-modify-write with an absolute upsert. The scope key is
    a conversation, not an agent, so two agents in one channel held by two members
    charged concurrently: both read n, both wrote n + 1, and one turn went uncounted.
  • The write lock the path assumed does not exist on the shared store. The Postgres
    facade rewrites BEGIN IMMEDIATE to a plain BEGIN, so the enclosing transaction
    bought no exclusivity — the second writer merely blocked on the row lock and then
    overwrote with its stale-derived value.

Failure scenario

A runaway conversation involves two agents held by two members. The counters undercount
exactly when the loop is fastest, because that is when concurrency is highest, so the
circuit trips late or not at all. When it finally trips on member A, A deletes member B's
durable backlog rows out from under B's in-memory queue, but B's live ACP turns are never
interrupted and keep running. The durable latch does block new admissions on B, so the
outcome is a partial, confusing failure: work disappears, the loop continues, and the two
members' logs tell different stories.

Fix

The trip acts only on what this member serves. purgeLoopScopeInbox skips rows whose
agent this member does not hold, through the same servesAgent gate that already scopes
crons, deadlines, the sandbox sweep and — since #1065 — the session TTL/GC sweeps. A peer's
queued row is left for its holder, never destroyed. The interrupt half is split out as
interruptLoopScopeTurns, which is member-local by construction, and a member that meets a
circuit a peer latched stops its own turns once per latch, on the first admission it
refuses. The trip's operator warning still has exactly one owner.

The counters are relative and window-aware in SQL. Both the charge and the structural
trip are single statements with RETURNING, identical on SQLite and Postgres: an upsert
whose DO UPDATE increments from loop_guard.totalCount (guarded by
WHERE loop_guard.trippedAt IS NULL, so a latched scope is never recharged and returns no
row), and a guarded UPDATE for the latch itself. The verdict is therefore computed from
what was actually stored, and the latch is a CAS that elects exactly one member to run the
trip's side effects.

That removes the need for the lock rather than faking one. The remaining transaction in
recordLoopGuardTurnForInbox buys atomicity between the charge and the inbox marker only —
no SELECT … FOR UPDATE path was added, because nothing reads-then-writes any more. The
BEGIN IMMEDIATE rewrite in the Postgres worker now states why a shared-store statement has
to be a CAS or a relative write, instead of leaving each caller to assume a writer lock it
never gets.

Local single-daemon behavior is unchanged: with no duty enforcement every agent is served
here, so the purge still clears the whole conversation backlog. No schema change.

Out of scope, per the issue: the k8s driver, the sweeps (#1065), and the outboxes (#1023).

Test plan

New packages/daemon/test/daemon-loop-guard-pool.test.ts — two members over one store:

  • a trip on A leaves B's durable row in place and does not mark B's live gate head
    cancelled; B then cancels its own head and purges its own row when it enforces the
    latch, and does so once per latch, not on every subsequent refusal;
  • a single local daemon still purges the whole conversation backlog.

New cases in packages/daemon/test/loop-guard.test.ts — two LocalStore members over one
SQLite file, with a peer's charge interleaved at the exact moment the member's own write is
about to execute:

  • ten charges across the two members sum to exactly ten, including the interleaved one;
  • two members that exceed the budget in the same window elect exactly one trippedNow
    owner, and so do two concurrent structural trips;
  • a latched scope is never recharged, whichever member reads it.

Both new store cases and the pool case fail on main and pass here.

Run: pnpm --filter @agentconnect.md/daemon typecheck, the loop-guard, duty, sweep-pool,
durable-inbox, serial-gate, hook, interrupt-safety and local-store suites (389 tests),
pnpm lint, pnpm format:check.

…nters atomic

On a daemon pool the loop guard was global in its destruction and local in its
enforcement, and its counters lost increments under concurrent members.

`purgeLoopScopeInbox` scanned the whole install-wide inbox and deleted every row
in the scope, including rows queued on peers whose in-memory state this process
cannot see; the interruption half that followed only walked this process's own
maps. So a trip on one member destroyed a peer's durable backlog without ever
stopping the peer's live turns. The purge now skips rows whose agent this member
does not serve, exactly like the sweeps and the replay path, and a member stops
its own turns once per latch on the first admission its open circuit refuses.

The counters were a JS read-modify-write followed by an absolute upsert, so two
members charging the same conversation both read n and both wrote n + 1 — the
undercount is worst exactly when the loop is fastest. Both the charge and the
latch are now single relative, window-aware statements with `RETURNING`, so the
verdict is computed from what was actually stored and the latch is a CAS that
elects exactly one owner for the trip's side effects.

That also removes the path's dependence on an exclusive writer, which the shared
store cannot give it: the Postgres facade rewrites `BEGIN IMMEDIATE` to a plain
`BEGIN`. The remaining transaction buys atomicity between the charge and the
inbox marker only, and the rewrite now says why a shared-store statement has to
be a CAS or a relative write.

Local single-daemon behavior is unchanged: with no duty enforcement every agent
is served here, so the purge still clears the whole conversation backlog.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@agentconnect-md-test agentconnect-md-test Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 162231126f9f420057b42c6d7cf1396225f39d8d against base 52e7754a2e672a4c464709e27d86af10f8b8bcf0.

One blocking pool-concurrency gap remains. The relative counter updates and member-scoped inbox purge look sound, but the new structural-trip CAS makes overlapping callers on losing members return trippedNow: false. The malformed-DM dispatch branch handles that result by purging the member’s inbox only; it does not call enforceLatchedLoopScope, so existing ACP turns on each losing member continue until some later admission happens to encounter the open latch. This directly undercuts the PR’s member-local interruption guarantee and is amplified by this change because overlapping structural trips previously let both callers report trippedNow: true and run their local interruption side effects.

Please enforce the already-latched scope on the structural-trip loser path as well. No additional tests are required for this review.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

)
.run({ ...row, trippedAt: now, reason })
.get({ scopeKey, now, reason }) as { totalCount: number; automaticCount: number } | undefined
if (!latched) return this.latchedLoopGuardVerdict(scopeKey)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CAS correctly elects one side-effect owner, but it changes overlapping structural trips so every losing member returns trippedNow: false. In Daemon.dispatch’s malformed-DM branch, that result only calls purgeLoopScopeInbox; unlike the normal counter-latch path, it never calls enforceLatchedLoopScope. The winner can interrupt only its own in-memory turns, so losers leave their live ACP turns running until another admission arrives. Please enforce the latch in that false branch too (alongside the member-scoped purge).

The trip's CAS gives overlapping callers on losing members `trippedNow: false`,
and the malformed-DM branch answered that with a member-scoped purge alone. So a
member that lost the race kept its live ACP turns running until some later
admission happened to hit the open latch. It now enforces the latch there as
well, exactly like the counter-latch path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zfy0701

zfy0701 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Good catch — fixed in 34f9187.

The structural trip's CAS gives overlapping callers on losing members trippedNow: false,
and Daemon.dispatch's malformed-DM branch answered that with the member-scoped purge alone,
so a loser kept its live ACP turns running until some later admission happened to hit the
open latch. That branch now calls enforceLatchedLoopScope next to the purge, the same as
the counter-latch path in admitLoopGuardTurn.

New case in daemon-loop-guard-pool.test.ts: B latches the DM circuit, A is made to read it
as still closed a moment earlier — the exact race the CAS resolves — and A's malformed-DM
dispatch now cancels A's own live head. It fails without the added call.

Re-ran the loop-guard, duty, sweep-pool, durable-inbox, serial-gate, hook, interrupt-safety
and local-store suites (390 tests), plus daemon typecheck, pnpm lint and
pnpm format:check. Rebased on current main.

@agentconnect-md-test agentconnect-md-test Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 34f9187c657b966dbd11ba26218becc355bd1bea against base 52e7754a2e672a4c464709e27d86af10f8b8bcf0.

Approved. The follow-up closes the prior blocking race: when a member loses the structural-trip CAS, the malformed-DM path now performs its member-scoped inbox purge and calls enforceLatchedLoopScope, so the elected winner remains the sole warning owner while every member still interrupts its own live ACP turns. The focused pool regression test exercises that loser path. I found no remaining blocking regressions in the relative counter updates, latch election, or member-scoped purge behavior.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

@agentconnect-md-test

Copy link
Copy Markdown
Contributor

Confirmed—I re-reviewed 34f9187 and approved it. The structural-trip loser now enforces the latch locally, and the regression case covers the original race. Thanks for the quick fix.

sent by review-bot (Codex · gpt-5.6-sol) · open in session

@zfy0701
zfy0701 merged commit b17f981 into main Aug 16, 2026
11 checks passed
@zfy0701
zfy0701 deleted the fix/loop-guard-shared-store branch August 16, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loop guard across pool members: the trip deletes peers' backlog without interrupting their turns, and the counters lose increments

1 participant