handoff: continuation affinity — ReactorSynchronizationContext spec + session findings index

[MDA2AV]

Handoff/tracking issue: an analysis session (2026-07-02, repo at cf2fce344eea6d4dc023745c4d4d9687a720a448) produced a set of findings and one designed-but-unimplemented feature. This issue indexes the findings and carries the full spec so anyone — human or agent — can pick the work up with no other context. If you are an agent continuing this: read this issue top to bottom, then start at "The deliverable".

What already landed (do not redo)

• perf: adopt direct descriptors (registered file table) for accepted sockets #91 — direct descriptors (registered file table) for accepted sockets. Includes corrected cost accounting: multishot recv takes its file ref once at arm time; the recurring fget/fput is per send op.
• incremental: accept beyond MaxConnections crashes the reactor (Stack.Pop on empty gid stack) #92–opt-in perf experiments: NAPI busy-poll (6.7+) and min-wait batched completions (6.12+) #103 — review findings, filed with severity:* labels, permalinks pinned to cf2fce3 (line numbers may have drifted). Headlines: incremental: accept beyond MaxConnections crashes the reactor (Stack.Pop on empty gid stack) #92 gid-exhaustion reactor crash (critical), recv: -ENOBUFS tears down healthy connections instead of stalling #93 -ENOBUFS tears down healthy connections, faulted handlers leak connections; incremental loop doesn't observe handler faults #94 faulted-handler connection leak, ring: set IORING_SETUP_CQSIZE - multishot decouples CQ pressure from SQ depth #95 CQSIZE, coalesce eventfd wakes across all cross-thread queues (only _postQ coalesces today) #96 wake coalescing, hardening: idle/keep-alive timeouts and per-reactor connection caps #97 idle timeouts/caps, register the ring fd (IORING_REGISTER_RING_FDS) to skip per-enter fdget/fdput #98 registered ring fd, arm the 250ms timer only when tickers exist; consider IORING_TIMEOUT_MULTISHOT #99 timer, ZeroCopySend is silently ignored in incremental mode #100 UseZc/incremental, observability: per-reactor counters (ENOBUFS, overflows, accepts, sheds, pool hit rate) #101 counters, validate ServerConfig at startup (power-of-two sizes) instead of kernel EINVAL #102 config validation, opt-in perf experiments: NAPI busy-poll (6.7+) and min-wait batched completions (6.12+) #103 NAPI/min-wait experiments.
• Decision: IORING_RECVSEND_BUNDLE is not worth adopting. Send side already coalesces (write slab → one SEND/SENDMSG per flush = one SQE+CQE per response; bundles can't go lower). Recv side: bundles and big-buffers/incremental are competing solutions to buffer granularity — ioxide picked 32 KB shared buffers and IOU_PBUF_RING_INC, and incremental's contiguous per-connection assembly is better for HTTP parsing than bundle-scattered buffers anyway (bundles likely don't compose with INC rings; verify on target kernel if ever revisited). Don't re-litigate without new evidence.

The open thread: continuation affinity (work stealing vs the tick)

Experiment: setting RunContinuationsAsynchronously = true on the connection value-task sources collapsed performance. Diagnosis — the inline tick is a transaction (reap CQE batch → run handlers on-reactor → their SQEs land via fast paths → one SubmitAndWait), and offloading continuations to the ThreadPool dissolves it into four separable costs:

1. Submit-batch collapse — reactor finishes dispatch with an empty SQ, parks, then gets dribbled flushes ≈ one io_uring_enter per response.
2. Eventfd storm — off-reactor EnqueueFlush/EnqueueReturnQ do one write(2) per item (coalesce eventfd wakes across all cross-thread queues (only _postQ coalesces today) #96; only _postQ coalesces).
3. ThreadPool global-queue hop — the reactor is not a pool thread, so every continuation goes through the pool's global queue.
4. Locality loss — connection state touched on whatever core stole the continuation.

Two hard-won conclusions:

• A strict "wait for all woken handlers to enqueue before submitting" barrier is unsafe: a handler may never enqueue (foreign await, CPU work, completion), and may await something that requires the reactor to advance → deadlock. Any rendezvous must be deadline-bounded.
• The right mechanism already exists: ScheduleOnReactor + DrainPostQ (Reactor.Post.cs). Loop order runs drained callbacks on-reactor before SubmitAndWait, so posted continuations regain batch coherence by construction, with a coalesced wake (_postSignalPending). The Kestrel bridge (ReactorPipeScheduler → ScheduleOnReactor; HopDuplexPipe reader schedulers; ReactorPinReader first-read pin) already routes pipe continuations this way. The remaining gap: awaits that don't go through the pipes (HttpClient, EF, Task.Delay, Task.Run results) resume on the pool and drift until the next pipe/connection await.

Also note when benchmarking any off-thread mode: check for #93 (-ENOBUFS teardowns) polluting the numbers, and fix #96 first or the comparison indicts the wake path, not work stealing.

The deliverable: ReactorSynchronizationContext

Close the gap with a per-reactor SynchronizationContext (the roadmap's "BCL bridge" item in IoxideRuntime.cs). Background for humans: SynchronizationContext.Current is a per-thread ambient slot read by the C# await machinery — captured at the await point, and the completing thread calls capturedContext.Post(continuation) instead of running it locally. Installing one on the reactor thread makes every await (not just pipe ones) resume on the reactor, transitively, unless explicitly opted out via ConfigureAwait(false).

Implementation, file by file:

1. New ioxide/Reactor/ReactorSynchronizationContext.cs

public sealed class ReactorSynchronizationContext : SynchronizationContext
{
    private readonly Reactor _reactor;
    internal ReactorSynchronizationContext(Reactor reactor) => _reactor = reactor;
    public Reactor Reactor => _reactor;

    public override void Post(SendOrPostCallback d, object? state)
        => _reactor.ScheduleOnReactor(d, state);          // MUST always queue, never invoke inline

    public override void Send(SendOrPostCallback d, object? state)
    {
        if (_reactor.OnReactorThread) { d(state); return; }
        using var done = new ManualResetEventSlim();
        Exception? ex = null;
        _reactor.ScheduleOnReactor(_ => { try { d(state); } catch (Exception e) { ex = e; } finally { done.Set(); } }, null);
        done.Wait();
        if (ex is not null) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(ex).Throw();
    }

    public override SynchronizationContext CreateCopy() => this;
}

2. Reactor.Runner.cs Run() — install right after _reactorThreadId is set: SynchronizationContext.SetSynchronizationContext(new ReactorSynchronizationContext(this));. Thread-lifetime; nothing to uninstall.

3. Reactor.Post.cs — two changes:

• Add a ScheduleOnReactor(SendOrPostCallback, object?) overload. SendOrPostCallback and Action<object?> have identical signatures but are distinct delegate types; converting allocates per post. Let PostItem hold a Delegate and type-test at invoke.
• Harden DrainPostQ: wrap each callback in try/catch (log + count). async void exceptions are delivered by rethrow inside Posted callbacks; unhandled, one unwinds Run() and kills the reactor (same failure class as incremental: accept beyond MaxConnections crashes the reactor (Stack.Pop on empty gid stack) #92). This becomes mandatory the moment arbitrary continuations flow through the queue.

4. The hot-path trap — Connection.Read.cs and Connection.Write.Flush.cs IValueTaskSource.OnCompleted: ManualResetValueTaskSourceCore captures SynchronizationContext.Current when the awaiter passes UseSchedulingContext (awaits always do), and on SetResult it unconditionally Posts to a captured context — RunContinuationsAsynchronously = false only covers the null-context case. Once the context is installed, every ReadAsync/FlushAsync resume would silently become SetResult → Post → postQ → next-loop drain instead of an inline call. Fix: strip the flag before forwarding —

_readSignal.OnCompleted(continuation, state, _readSignal.Version,
    flags & ~ValueTaskSourceOnCompletedFlags.UseSchedulingContext);

Safe because these sources always complete on the owning reactor thread (the continuation already runs where the context would post it), and the thread-wide slot keeps downstream awaits captured. Keep FlowExecutionContext. Consider a config toggle to retain the posted mode: "inline vs posted-but-on-reactor" isolates pure queue-deferral cost with zero core migration — a useful benchmark point (see below).

5. ioxide.Kestrel — make context installation a transport option, default on. Rationale: ASP.NET Core normally runs context-free, so legacy sync-over-async middleware (.Result) merely risks pool starvation there; under a reactor context it hard-deadlocks that reactor (the loop is blocked, the mailbox never drains). Document the ban; keep the escape hatch. Optional cleanup: IoxideReactor.TryCurrent() can be reimplemented as SynchronizationContext.Current is ReactorSynchronizationContext r ? r.Reactor : null.

Document: ConfigureAwait(false) in handler code opts that await out of affinity (library-internal CA(false) is desirable — their plumbing stays off-reactor; the boundary await comes home). Pipes keep useSynchronizationContext: false.

Acceptance criteria

• Smoke: a handler asserting reactor.OnReactorThread after await Task.Delay(1), after an HttpClient call, and after await Task.Run(...); an async void-throwing callback does not kill the reactor.
• Perf gate: wrk plaintext (scripts/static-bench.sh) with context ON vs OFF for pure-ioxide handlers must be flat — the §4 strip is what guarantees this; if it regresses, the flag strip is missing or wrong.
• Decomposition benchmark (feeds the HTTP Workshop 2026 talk — keep modes as toggles, don't delete alternate paths): (a) inline baseline, (b) posted-on-reactor mode (§4 toggle), (c) RCA=true ThreadPool mode raw, (d) ThreadPool mode + coalesce eventfd wakes across all cross-thread queues (only _postQ coalesces today) #96 wake coalescing. The (a)–(b) gap = queue discipline; (b)–(d) gap = migration + pool hop; (c)–(d) = the eventfd storm.

Notes for whoever picks this up

• Analysis was read-only: no code changes exist from that session; everything is in issues perf: adopt direct descriptors (registered file table) for accepted sockets #91–opt-in perf experiments: NAPI busy-poll (6.7+) and min-wait batched completions (6.12+) #103 and this spec.
• Permalinks pin to cf2fce3; re-locate lines if the tree moved.
• Suggested order: faulted handlers leak connections; incremental loop doesn't observe handler faults #94 (framework-owned handler ref) → incremental: accept beyond MaxConnections crashes the reactor (Stack.Pop on empty gid stack) #92 → recv: -ENOBUFS tears down healthy connections instead of stalling #93 first if stabilizing; this issue first if the talk needs the decomposition data. The DrainPostQ hardening here overlaps faulted handlers leak connections; incremental loop doesn't observe handler faults #94's wrapper work — do them together.
• The analysis machine was WSL2 kernel 6.6: IORING_RECVSEND_BUNDLE (6.10) and IOU_PBUF_RING_INC (6.12) are untestable there; use the incremental-mode test box for anything kernel-gated.

[MDA2AV]

Follow-ups filed after the tick/submission-ordering discussion: 105: IOSQE_CQE_SKIP_SUCCESS analysis (#105), IORING_OP_LINK_TIMEOUT analysis (#106), and the SQ submit-accounting edge (#107 — compute toSubmit from the kernel-consumed head; EBUSY can strand published SQEs). The last one refines this issue's 'guaranteed CQE' reasoning: the guarantee assumes every staged SQE is eventually counted into a successful enter.