Give user index jobs a worker lane render sweeps cannot occupy

[habdelra]

Background

The matrix Playwright suite runs each shard against one isolated realm server on :4205, shared by both of Playwright's parallel workers. Realm provisioning and every subsequent write funnel into one worker manager, whose job queue serves several kinds of work:

• Indexing jobs (from-scratch-index, incremental-index, copy-index) gate everything user-visible about a write: createRealm blocks on the from-scratch index, and the read-your-writes endpoints — card GET and _publishability — drain in-flight incremental indexing before reading boxel_index, so their response time includes however long the write's index job sits in the queue.
• prerender_html jobs render card HTML after an index pass. The job spawned by a from-scratch index runs a realm-wide module pre-warm sweep first, which takes tens of seconds for a fresh ~110-module workspace.

User-initiated indexing and prerender-html are deliberately co-equal in priority (runtime-common/queue.ts): a published realm's rendered HTML is as first-class as its search index. Dequeue is oldest-first within a priority floor, so a priority floor alone cannot keep the two kinds of work from occupying the same workers.

The failure

On loaded CI runners, publish-realm.spec.ts › it warns when private dependencies would cause host mode errors fails when the publish modal's private-dependency check exceeds its 15s assertion budget. The job-level timeline from a failing run shows why: at the moment the test POSTs its card sources, all three user-tier workers are held — two by concurrent pre-warm sweeps (25.8s and 17.9s) for freshly created workspaces, one by the base realm's 600-file boot reindex. The just-written cards' incremental index job waits 15.1s in the queue before a worker even starts it, so the _publishability drain alone outlives the assertion (dur=15072ms, dur=17480ms across attempts; the final retry's response landed at 14.8s — a hair too late). The same queue wait shows up on the neighboring card GETs (/index, /realm at 15–18s).

The fix

The worker-manager's userIndexCount pool now starts its workers with --indexJobsOnly, which restricts handler registration to the indexing job types. The queue's claim query only dequeues job types a worker has registered handlers for, so this makes the pool a dedicated indexing lane that a render sweep can never occupy, independent of priority tiers. The high-priority pool still carries the prerender-html sweeps alongside other user-initiated work — nothing about job priorities changes, so rendered-HTML work keeps its standing.

userIndexCount defaults to 0 and the matrix harness is its only consumer, so no deployed worker changes behavior.

With the lane in place, the drain on _publishability costs the actual indexing time of the written files (a few seconds under CI load, matching passing-run timings) instead of an unbounded queue wait.

Diagnostics

The failing runs also produced _publishability 404s whose origin the logs cannot attribute, because two failure paths are silent:

• Router.handle maps a thrown CardError to an error response without logging it, so a thrown 404 is indistinguishable in the request log from a deliberate not-found response. It now logs the method, URL, status, and message before responding.
• The _publishability handler now logs its drain duration and scan counts (instances scanned, error rows, violations, publishable) so a slow or empty-looking report can be attributed to queue wait vs. the scan itself.

If the 404 recurs, these two lines plus the request log pin down which layer produced it.

Testing

worker-job-registration-test.ts pins the registration sets: a default worker registers every job type; an --indexJobsOnly worker registers exactly the indexing job types and cannot claim prerender_html.

🤖 Generated with Claude Code

[github-actions]

Host Test Results

139 tests   137 ✅  9m 3s ⏱️
  1 suites    2 💤
  1 files      0 ❌

Results for commit c54c317.

Realm Server Test Results

    1 files      1 suites   16m 38s ⏱️
1 877 tests 1 877 ✅ 0 💤 0 ❌
1 956 runs  1 956 ✅ 0 💤 0 ❌

Results for commit c54c317.

[Copilot]

Pull request overview

This PR isolates user-initiated indexing work onto a dedicated worker “lane” by restricting handler registration for a configurable pool of workers, preventing long prerender sweeps from occupying all user-tier workers and delaying read-your-writes drains (e.g. _publishability) in CI.

Changes:

• Added an indexJobsOnly worker mode that registers handlers only for indexing job types, and used it for the worker-manager’s userIndexCount pool.
• Improved CI diagnosability by logging when routed handlers throw CardError, and by timing/logging the _publishability indexing-drain plus scan counts.
• Added a unit test to pin the job-handler registration set for default vs indexJobsOnly workers, and enabled the dedicated lane in the Matrix isolated realm-server harness.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/runtime-common/worker.ts	Introduces INDEX_JOB_TYPES and indexJobsOnly to restrict handler registration to indexing jobs.
packages/runtime-common/router.ts	Logs when a handler throws CardError to distinguish thrown errors from deliberate responses in request logs.
packages/runtime-common/realm.ts	Adds timing and summary logging around _publishability’s incremental-index drain and scan results.
packages/realm-server/worker.ts	Adds --indexJobsOnly CLI flag and wires it through to Worker.
packages/realm-server/worker-manager.ts	Starts userIndexCount workers with --indexJobsOnly and preserves that behavior on respawn.
packages/realm-server/tests/worker-job-registration-test.ts	New test ensuring handler registration differs correctly between default and indexing-only workers.
packages/realm-server/tests/index.ts	Includes the new worker registration test in the realm-server test suite.
packages/matrix/support/isolated-realm-server.ts	Configures the Matrix harness to use a dedicated indexing-only worker lane.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.