Skip to content

A successful any() writes a circuit failure to every member that lost the race #706

Description

@rejifald

Scenario: orphaned-half-write
Proofs: docs/scenarios/proofs/orphaned-half-write/ (8 scripts, 189 checks)

Found while measuring multi-step resource creation. This is a composition of two individually
reasonable behaviours, which is why it is invisible from either side.

The finding

pipe.ts auto-cancels the members that can no longer affect the result — finally { ctrl.abort() }
at :115 (all), :135 (any) and :155 (race). An abort reaches engine.ts:903-914, which
catches everything except CircuitOpenError and calls circuit.onFailure() (this is
#705).

Give each member its own circuit, and:

arrangement member outcome circuit.failures
successful any([a,b,c]) mirror.a won 0
mirror.b auto-cancelled, healthy 1
mirror.c auto-cancelled, healthy 1
failing all([x,y,z]) step.x genuine 503 1 (earned)
step.y auto-cancelled, healthy 1
step.z auto-cancelled, healthy 1

Three successful any() calls open a failures: 3 breaker (measured 1→2→3, tripped on the
third). After that an innocent caller — no signal, no group — gets StitchError "circuit open",
status 503, having sent 0 requests, to a vendor that is up and answering 200. The winner's
own breaker is untouched.

The failure mode is the inverse of the intent: any() exists for failover across mirrors, and a
circuit exists to take a sick mirror out of rotation. Used together, every successful failover
marks the healthy mirrors as failing
, so the breaker removes the mirrors that were fine.

Causation, established three ways

Not inferred from the source:

  1. Control Z — the same three stitches run standalone: 0 failures each. The endpoints are
    healthy and the stitches are well-formed.
  2. The losers' recorded error is This operation was aborted, and the vendor's ledger recorded
    2 hang-ups with completedBeforeHangUp = false — sockets destroyed mid-handler.
  3. Control A′ — a standalone success on mirror.b clears its count back to 0, so the number
    being read is the breaker's own live counter, not an artefact.

C1 was re-run three times with identical numbers.

Why it cannot be classified at the seam

createCircuit (resilience.ts:337-406, md5-identical to origin/main) exposes onSuccess() and
onFailure() only, and onFailure() takes no argument. The stored record
{"failures":1,"tripped":false} is byte-identical to the one a genuine 503 leaves. There is no
information at the circuit seam that could distinguish a group's own cancellation from an outage.

Ask

Either of two places:

  • In pipe — the group's abort is the library cancelling its own work, not a caller's. It could
    be marked so the engine skips onFailure() for it, without touching the caller-signal case.
  • In the engine — skip circuit.onFailure() when the run ended in an abort. That is the
    #705 fix and it closes this one too.

The engine-side fix is strictly better because it also covers a caller who aborts a member directly,
but the pipe case is the one with the surprising blast radius: the caller never cancelled anything
and the damaged endpoints are healthy.

Documentation, whichever is chosen: pipe.ts:19-21 describes auto-cancellation as a courtesy
("members that can no longer affect the result"). It is worth saying there that a cancelled member is
currently a circuit failure for that member.

The workaround, measured

Hand-rolled Promise.all([a.safe(), b.safe(), c.safe()])3 executable lines — runs the
identical arrangement and leaves both healthy breakers at 0, because there is no per-group
AbortSignal. Cost: 3 root runs on 3 traceIds instead of one fan, and no cancellation at all.

Related: the module comment says there is deliberately no allSettled and directs the reader to
"compose .safe() members by hand" — but .safe() is not a combinator member. It carries no
__stitch/__composable brand, and forcing it past the type gives
TypeError: node.__runWith is not a function. The documented substitute means leaving pipe
entirely, which is worth stating outright in that comment.

Two smaller findings from the same suite

  • idempotency.keyOf derivations are unvalidated. A derivation reading an absent field yields
    the stable key signup:undefined, which silently collapses two different signups into one. A
    key that is constant across distinct inputs is always a bug; a construction-time or first-use
    warning when a derivation produces undefined would catch it.
    (idempotency: true warns at construction that its UUID is per-logical-call — good — and measured,
    it collapses 0 creates across a re-run, so it is no better than nothing for this scenario.)
  • hooks.onError fires for an auto-cancelled write with no payload — handed exactly
    { attempt, error, name }, no req, no res. It is a trigger with nothing to act on, which
    matters here because it is the only per-member hook that fires for a cancelled member.

Reproduction

npx tsx docs/scenarios/proofs/orphaned-half-write/c1-circuit-and-autocancel.ts
npx tsx docs/scenarios/proofs/orphaned-half-write/c7-observable-surface.ts

A real node:http vendor on 127.0.0.1 keeping a durable create/delete book, with every held handler
released by an explicit promise — never a timer. circuit.failures is read directly out of
StitchAPI's own store record written by createCircuit.onFailure(); the orphan counts are the
vendor's book, deliberately, because the caller's belief is the thing under test and cannot also be
the instrument.

Which engine

  • pipe.ts — zero diff against origin/main. :19-21 and the three ctrl.abort() sites are
    identical.
  • engine.ts circuit block — byte-identical, shifted: :880-891 in this worktree, :903-914 on
    origin/main.
  • resilience.ts createCircuit — md5-identical. The file's 56-line diff is entirely
    RateLimitError (now extends StitchError, PR refactor(core)!: RateLimitError extends StitchError #662), untouched by anything here.
  • surface.ts classifyStatus — zero diff.

Only this worktree was executed. Since every load-bearing block is byte-identical, the result should
reproduce on origin/main, but I did not run it there.

No proof declares an input/output schema, so the worktree's zod 3.25.76 against a declared
^4.4.3 is not load-bearing for any claim.

Not tested

race() — it shares the same linkedController + finally { ctrl.abort() } shape as all and
any, so I expect the same result, but C1 and C6 measured only any and all. A distributed
(Redis-backed) breaker shared across workers would multiply the blast radius; every arm here uses
memoryStore().


Found while writing scenario 52 ("the third call failed and the first two already happened") for the docs. Every claim is backed by a runnable offline proof in docs/scenarios/proofs/orphaned-half-write/.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Correctness bug, live on mainbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions