Interpreted event loop: socket/TLS workloads degrade 50-100x under ambient process load (investigation)

[nickna]

Symptom

TlsServerAndClientHandshake_Interpreted (guest-code TLS server + client handshake over loopback) runs in 0.27-0.32s in isolation but 13.6-30.3s inside the full SharpTS.Tests suite. This is the single largest contributor to the suite''s wall-clock critical path.

This is filed as a runtime concern, not a test concern: whatever stalls the interp event loop ~100x under load likely affects real interpreted socket/TLS workloads on busy hosts.

What the investigation established (all measured)

• Not the test or TLS: the pure-C# twin (TlsServerAndClientHandshake, raw SslStream, no interpreter) runs in 0.09-0.12s in the same full-suite runs. Machine, Defender, lsass, loopback all exonerated.
• Not class state: the whole TlsModuleTests class alone -> 0.32s.
• Not any single neighbor category: pairing with NetModule/HttpModule, Cluster/ChildProcess, or Crypto/Https classes -> 0.18-0.22s.
• Partially the subprocess tier: pairing with IntegrationTests+PackagingTests (dotnet child-process spawns, DLL compiles to disk) -> 2.6s (10x). Full effect needs suite breadth.
• Not raw CPU parallelism, and inversely correlated with it: maxParallelThreads=8 -> 30.3s, default aggressive (~16) -> 19.6s, =24 -> 13.6s.
• Parallel-knob tuning is a dead end: suite wall is flat 53-62s across aggressive/conservative/8/24; DOTNET_ThreadPool_ForceMinWorkerThreads=128 was catastrophic (294s wall, 10x test-second inflation). Current xunit config is near-optimal; do not re-tune without new evidence.
• Stack samples during the stall window show heavy GC churn (PollGC workers, spin-waiters), in-process IL compilation (MetadataBuilder/Ecma335 frames), and compiled-test threads parked in emitted $Runtime..cctor via InitClassSlow.
• The ScheduleTimer(0, ...) hop path used by SharpTSTlsSocket to marshal handshake completions onto the event loop does wake a blocked loop (verified) - though Interpreter.ScheduleTimer: cross-thread timers with delay > 10ms never wake a blocked event loop (worst-case latency up to 60s) #194 is adjacent (delays >10ms do not wake).

Mechanism (test harness side)

Every test runs as Task.Run(work) + task.Wait(timeout) (SharpTS.Tests/Infrastructure/TestHarness.cs), so each concurrent test parks a thread-pool thread for its full duration, with the interp event loop itself running on a TP thread and its I/O completions queueing behind parked workers. The TLS test makes many small loop-hops per handshake stage, so per-hop latency multiplies.

Next step

Black-box sampling is exhausted. Needs in-situ instrumentation: env-gated stage timestamps in SharpTSTlsSocket (connect / authenticate / per-hop ScheduleTimer-to-callback latency) plus event-loop wake-to-dispatch latency, run inside the full suite to localize which hop class eats the time (TP queue delay vs GC suspension vs TryTake wake).

Related: #94 (dotnet test speed - this is its biggest single line item), #194 (timer wake gap), #110 (Test262 Pass->Timeout under parallel-regen contention - plausibly the same amplification mechanism on the Test262 side).

[nickna]

Re-confirmed (2026-06-20): still present, and now the dominant single line item in the suite

Re-measured on a 12-core box, dotnet test -c Release, full suite (13,986 tests), aggressive parallelism, the committed MinThreads=16. Per-test durations pulled from the trx logger:

Scenario	TlsServerAndClientHandshake_Interpreted
Isolation (--filter to just this test)	0.455 s
Inside the full suite	6 m 23 s (383 s)

~840× degradation — even worse than the 50–100× originally documented (this box has 12 cores vs. the 16-core reference; thread-pool starvation worsens nonlinearly as cores drop, so I'd treat the absolute number as machine-specific, not a regression claim). The next-slowest test in the entire suite is < 10 s, so this one test is ~40× anything else and owns the wall-clock critical path (suite wall here was 10 m 6 s).

Telling detail confirming the mechanism: the test has its own 15 s gate (task.WaitAsync(TimeSpan.FromSeconds(15))) yet passed after 383 s — under starvation the timeout's own Timer/ContinueWith callbacks can't get a pool thread to fire on time. The guard is meaningless under the exact condition it exists to catch.

Root cause is already localized — the proposed instrumentation pass isn't needed

Black-box sampling + a code read pin it conclusively to thread-pool starvation, structural to how the harness and event loop interact:

1. Harness (SharpTS.Tests/Infrastructure/TestHarness.cs): every test is Task.Run(work) + task.Wait(timeout). Each concurrent test parks a pool thread for its full duration, and the interp event loop (RunEventLoop → blocks in _callbackQueue.TryTake) runs inside that work delegate — so it's also on a pool thread.
2. TLS handshake (SharpTSTlsSocket.cs:95, SharpTSTlsServer.cs:192/:207): connect/handshake is Task.Run(async …) → ConnectAsync → AuthenticateAsClientAsync (many loopback round-trips, each an await continuation) → StartReading (async ReadAsync loop) → ScheduleTimer(0,0,…) marshal-backs. Every one of those continuations needs a fresh pool thread.
3. With the pool saturated by parked task.Wait workers, .NET throttles thread injection to ~1–2/sec → each handshake hop stalls ~0.5–1 s × ~15–30 hops → tens of seconds to minutes.

Note on #194: it was closed (8e5cc912, "wake unconditionally on any timer delay") after this issue was filed, but it only touches the >10 ms path. The TLS hops use ScheduleTimer(0,…), which already woke the loop — so #194's fix does not touch this starvation. Independent problem.

Recommendation: do the work, but reframe it

• Drop the instrumentation step. The sampling already done plus the code read identify the cause; env-gated stage timestamps would only re-confirm it.
• Test-harness fix (cheap, high-leverage, self-validating): run the interp/event-loop work on a dedicated Thread instead of Task.Run, so the synchronous-blocking event loop and the gating wait stop consuming pool workers — freeing the pool for the SslStream I/O continuations. Low risk; measurable before/after on the in-suite TLS time.
• Runtime robustness (the real product concern this issue rightly raises): consider running the interp event loop on a dedicated thread rather than a pool thread, so interpreted socket/TLS workloads on busy hosts (contended pool) don't depend on pool thread-injection. Bigger change; the harness fix is the natural first step and de-risks it.
• Not min-thread tuning — already established as a dead end here (ForceMinWorkerThreads=128 → 294 s catastrophe). The fix is structural, not a knob.

Relations unchanged: still #94's biggest single line item; #110 (Test262 Pass→Timeout under parallel-regen) is plausibly the same amplification mechanism.