Skip to content

Layer-1 flow tests flake under heavy parallel CPU load (event-loop starvation) #136

Description

@breferrari

Symptom

Layer-1 flow tests (tests/component/flows/*.test.tsx) intermittently fail under heavy parallel CPU load — when the full ~1179-test suite runs in one vitest batch on a saturated machine (CI cells, or a busy local box). The failures are non-deterministic (a different test fails each run) and every implicated test passes in isolation.

Observed instances:

  • self-update-flow scenario 7 ("first frame never contains the banner — banner is async") — failed on the v0.1.3 release run's Windows cell. Root-caused and fixed in fix(test): de-flake self-update banner flow under CPU starvation #135: checkSelfUpdate's 3s internal fetch-abort fired before the (fast, local) stub response was processed under event-loop starvation → banner never rendered → 15s waitFor timed out.
  • install-flow scenario 12 (requires.shardmindSHARDMIND_VERSION_MISMATCH, v0.2: Engine version-compatibility check on install #121) — observed failing locally only under full-parallel load; passes in isolation.

Root cause (class)

Layer-1 flow tests mount the real React tree + real async hooks + a local HTTP stub, then assert against rendered frames within fixed time windows. Two timing surfaces race event-loop starvation when the CPU is saturated:

  1. Fixed internal timeouts in the code under test (e.g. the self-update fetch's 3s AbortController) that fire before a fast local response is processed, turning a success into a silent null/no-op.
  2. waitFor windows in the tests that expire before a starved loop gets a slice to resolve + setState + re-render.

This is the same family as the known flakes #106 (hook-runner pre-throw stdout under parallel pressure) and #114 (merge-adversarial 10K tokenize timeout under parallel pressure).

Directions to evaluate

  • Audit fixed internal timeouts for env-overridability the way fix(test): de-flake self-update banner flow under CPU starvation #135 did for the self-update fetch (SHARDMIND_SELF_UPDATE_FETCH_TIMEOUT_MS), so flow harnesses can widen them under load without changing production defaults. Candidates: any AbortController/setTimeout-bounded I/O reachable from a flow test.
  • vitest pool / concurrency tuning — cap poolOptions worker count, or move Layer-1 flow tests into their own project/pool with reduced parallelism so they aren't starved by the heavier suites (e2e/contract/merge-adversarial running concurrently).
  • Per-file isolation for the flow tests if cross-file env/timer contention is contributing.
  • Generous, consistent waitFor budgets for banner/async-arrival assertions (ordered: test-timeout > internal-timeout > waitFor), as a belt-and-suspenders backstop.

Scope / acceptance

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions