Symptom
After a daemon-pool rollout, every new member logs, repeatedly and indefinitely:
WARN event/session snapshot deferred after N failures for session <id> (WireError: cannot resolve organization for event/session-sync
Same session id on all three members, N climbing on each. Nothing recovers it.
Where it comes from
packages/daemon/src/cp/client.ts scopedFrame(): in frame mode a non-install-wide frame must carry an orgId; it is derived from the payload's agent id via orgForAgent. For this event/session-sync snapshot that resolution returns nothing (the agent is not served by this member — or the sync payload does not carry an agent id the resolver knows), so the client throws SCOPE_DENIED locally before the frame is sent. The snapshot lives in the session-metadata outbox, which pool members share through the data-plane store, so every member picks up the same row, fails the same way, and defers it again (runSessionMetadataDrain / drainSessionMetadataSnapshots).
The session in question was one whose webchat continuation had just failed after the rollout ("Connection error", see #987), i.e. a session recorded on a retired member.
What should happen
Two things, either fixes the loop, both are probably right:
- A member should only drain outbox rows for agents it currently serves (duty holder or placement), or the row should be claimable by whichever member serves the agent — a shared outbox drained by every member is a fan-out bug on its own.
- A
SCOPE_DENIED raised locally (organization unresolvable) is not a transient error; the drain should stop retrying it (park the row with a reason, or drop it after resolving the org from the session record itself), not defer forever.
Observed on 2026-08-16 (rc.129) during the #1016 rollout verification. Refs #955, #987.
Symptom
After a daemon-pool rollout, every new member logs, repeatedly and indefinitely:
Same session id on all three members, N climbing on each. Nothing recovers it.
Where it comes from
packages/daemon/src/cp/client.tsscopedFrame(): in frame mode a non-install-wide frame must carry anorgId; it is derived from the payload's agent id viaorgForAgent. For thisevent/session-syncsnapshot that resolution returns nothing (the agent is not served by this member — or the sync payload does not carry an agent id the resolver knows), so the client throwsSCOPE_DENIEDlocally before the frame is sent. The snapshot lives in the session-metadata outbox, which pool members share through the data-plane store, so every member picks up the same row, fails the same way, and defers it again (runSessionMetadataDrain/drainSessionMetadataSnapshots).The session in question was one whose webchat continuation had just failed after the rollout ("Connection error", see #987), i.e. a session recorded on a retired member.
What should happen
Two things, either fixes the loop, both are probably right:
SCOPE_DENIEDraised locally (organization unresolvable) is not a transient error; the drain should stop retrying it (park the row with a reason, or drop it after resolving the org from the session record itself), not defer forever.Observed on 2026-08-16 (rc.129) during the #1016 rollout verification. Refs #955, #987.