Role/user/schema mutations are eventually-consistent across worker threads (async ITC not awaited) — spurious 403s + flaky tests

[kriszyp]

Summary

Mutating operations that change auth or schema state — add_role / alter_role / drop_role, user create/alter/drop, and schema DDL — return HTTP 200 after the durable write + a local cache update + an ITC broadcast send, but before other worker threads have rebuilt their in-memory caches. The cross-worker propagation is fire-and-forget, so a request routed to a lagging worker observes the old state.

This is an internal consistency-invariant gap, not just a test artifact. The invariant we'd expect: once a role/user/schema mutation returns success, the change is in effect for subsequent requests. Today it's eventually-consistent across the worker pool.

Where it lives

• security/role.ts — alterRole (~L122), dropRole (~L189), add-role path: await signalling.signalUserChange(new UserEventMsg(process.pid)).
• utility/signalling.ts — signalUserChange / signalSchemaChange: call serverItcHandlers.user(...) (updates the local worker only, synchronously) then return sendItcEvent(...) (broadcasts to other workers). The awaited value is the send, not remote acknowledgment.
• security/user.ts — each worker's usersWithRolesMap is rebuilt by setUsersWithRolesCache() only when it processes the inbound ITC event.
• security/permissionsTranslator.js — translateRolePermissions expands per-attribute permissions from the (possibly stale) cached role; a stale role with no attribute_permissions for a table yields an empty attr-perm map.

Observable symptoms

1. Spurious 403 on grant. Admin grants attribute access via alter_role, gets 200, then a query for the newly-granted column hits a lagging worker → 403 … "Attribute 'X' does not exist on 'schema.table'" (the empty attr-perm map path in checkAttributePerms, utility/operation_authorization.ts:765-773).
2. Brief continued access on revoke. A revoke is similarly delayed on lagging workers — a short window where the old (broader) permissions still apply. (Live subscriptions have separate continuous re-auth; this is about request-path auth.)
3. Intermediate/partial-looking states. Because different workers can be at different role versions, a 403 can appear with a different body than the final state (e.g. empty unauthorized_access, a subset of invalid_schema_items).
4. Schema variant. The same pattern affects schema propagation — e.g. describe_all on a worker that hasn't yet seen a just-created schema returns it as missing/empty (integrationTests/apiTests/blob.test.mjs "all three device-type schemas are visible in describe_all").

CI impact

This is the root cause behind the recurring Integration Tests shard 5/6 flakes (and others) in northwind.test.mjs (cross-schema-join role tests) and the schema-visibility flake in blob.test.mjs. Test-side mitigations are in flight (#1474, #1475 — poll for propagation with waitFor), but those are per-call-site whack-a-mole and easy to get subtly wrong (the first waitFor predicate in #1475 was too loose and still flaked). Related closed test-side issue: #1222.

Proposed direction (needs design buy-in)

Make mutating role/user/schema ops await cross-worker propagation before returning, OR expose an internal propagation barrier:

• Add an ack to the ITC round-trip: sendItcEvent resolves once all live workers have processed the user/schema event (each replies after setUsersWithRolesCache() / schema reload). Mutating ops await that instead of the bare send.
• Bound it with a timeout + worker-death handling so a stuck/dying worker can't hang an admin op (degrade to best-effort + log, since the durable write already succeeded).

Trade-offs

• Adds latency to admin/DDL ops (rare, so likely acceptable) and some ITC complexity.
• Eliminates the entire mutate-then-read flake class (auth and schema) at the source, instead of per-test polling.
• Alternative (lighter) option: keep ops async but provide a documented/internal "propagation settled" signal the test harness (and careful clients) can await.

Filing for discussion before any product change — the test-side PRs (#1474/#1475) unblock CI in the meantime.

[kriszyp]

Dug into this — the gap is real, but narrower than the proposed direction: the cross-worker ack round-trip we'd want is already built, and six call sites just drop it.

The ack machinery already exists and is correct:

• signalUserChange / signalSchemaChange return broadcastWithAcknowledgement (server/threads/manageThreads.js), which resolves only after every live worker has acked (a port close-listener releases the wait if a worker dies, so a dying worker can't hang it).
• Each worker acks from its ITC handler (server/threads/itc.js) only after awaiting userHandler / schemaHandler (server/itc/serverHandlers.js) — i.e. after setUsersWithRolesCache() / schema reload finishes.

So an ack genuinely means "that worker rebuilt its cache." The awaited value is not just the send.

The actual bug is that these entry points dropped the returned promise, while their siblings already awaited it:

Mutating op	awaited?
alterRole, addUser / alterUser / dropUser	✅
addRole, dropRole	❌
createSchema, dropSchema, createAttribute, dropAttribute	❌

Fix = add await at those six sites, matching the correct siblings. That enforces the invariant directly, no new ITC design needed, and makes the per-call-site waitFor mitigations (#1474 / #1475) redundant — the flaky northwind / blob integration tests become the regression coverage.

PR: #1499.

Deliberately left out (flagging for the discussion):

1. tokenAuthentication refresh-token persist signals on every login — awaiting would add cross-worker latency for no auth-visibility gain, so left fire-and-forget.
2. createTable / dropTable propagate via a different layer (harperBridge → ResourceBridge → Table apply path), not a bare dropped signal — needs separate verification.
3. A timeout for a wedged-but-alive worker (the one piece genuinely missing) is pre-existing exposure shared by the already-awaited paths; bounding broadcastWithAcknowledgement touches all callers and needs a degrade-on-timeout policy, so better as its own follow-up than bundled here.

— KrAIs

[kriszyp]

Heads-up from a multi-worker QA re-test (harper main f447404, 4 workers, 40-req no-settle bursts, real low-priv credentials): the client-reachable impact here appears smaller than the issue states — in particular the security-critical stale-allow window does not reproduce.

Measured:

• Grant-lag: 0 spurious 403s
• Revoke-lag: 0 stale-allow — drop_user → 401, alter_role-revoke → 403, all 40/40 at 0 ms
• Schema-lag: 0 errors

Why (code trace): dropUser/alterRole/alterUser/addUser all await broadcastWithAcknowledgement (server/threads/manageThreads.js:473) — a genuine all-worker-ACK rendezvous — so the mutation is synchronous before the HTTP 200. The only unawaited path is addRole/dropRole's fire-and-forget signalUserChange() (security/role.ts:90/189), but that's client-unreachable: a new role does nothing until addUser assigns it, and addUser awaits.

So for client-reachable mutations this looks WAD / not-reproducible. If you'd still like belt-and-suspenders, the zero-risk consistency fix is to add await to the two signalUserChange() calls in addRole/dropRole. Either way, no de-authorized-user-still-reading window was found — a positive security result.

Repro test: qa376-multiworker-auth-propagation.test.ts.

— from Harper exploratory QA (KrAIs)