[perf] client worker

[SouradeepBera]

Fixes client worker threading

Client worker threading: measurement-fidelity impact

Compares the baseline against the dedicated-puller-thread change on the preflight gate.

	Commit	Change under test	Runs
before	e0316eb8	baseline	4
after	a337018f	+ client worker threading (dedicated puller thread)	5

uv run veeksha preflight --check-text True --check-tts True --concurrency 500 --num_sessions 1500

1500 requests per workload per run, 1500 paired, 0.000% unpaired on every run — no request
dropped or mismatched. All values are milliseconds, median across the runs on each commit
(medians because the after runs have large run-to-run spread; see Variance).

Ground truth did not move. server_ttfc_abs_error_ms p99 is 1.161 → 1.174 ms and
server_tpoc_abs_error_ms p99 is 1.018 → 1.062 ms, both against a 5 ms threshold and effectively
identical before and after. The mock servers held their pacing on both commits, so every difference
below is harness-side and the comparison is valid.

---

What gets better

chat — large tail win, exactly where the change targets

Gate p99	Threshold	before	after	Change
dispatch drift (lifecycle_ready_to_send_ms)	10	5336	563	9.5× better
request delivery	5	4527	2681	1.7× better

The win localises to the stage the commit addresses. lifecycle_dispatch_to_pickup_ms p99 drops
5336 → 546 ms (×0.10) and its mean 757 → 111 ms (×0.15). Before, intake was awaited on the event
loop, so item N+1 was not requested until task N had been created; the dedicated _pull_loop thread
(veeksha/workers/client_runner.py:71) posts items via call_soon_threadsafe and the loop converts
them to tasks in batches.

Secondary effects follow the same direction:

Metric p99	before	after
lifecycle_pickup_to_send_ms	166.32	98.88
lifecycle_ready_to_dispatch_ms	76.89	40.62
client_observed_ttfc_ms	5290	2882
client_observed_tpoc_ms	74.20	40.77

tts — neutral, as expected

Gate p99	before	after
request delivery	159.32	190.36
response delivery	0.554	0.830
dispatch drift	161.20	152.81

tts never had the dispatch-to-pickup pileup this change fixes (161 ms p99 before, not 5336 ms), so
there was nothing to win. The deltas here are within run-to-run spread.

---

What gets marginally worse — and why it doesn't matter

Regressions that stay far under threshold

Metric	Workload	Threshold	before	after
response_delivery_ms p50	chat	5	0.155	0.323
response_delivery_ms p50	tts	5	0.112	0.157
response_delivery_ms p99	tts	5	0.554	0.830
lifecycle_pickup_to_send_ms p99	tts	10	0.141	0.403
lifecycle_ready_to_dispatch_ms p99	tts	10	0.336	0.626

Sub-millisecond values against 5 ms and 10 ms gates. These ratios look large only because the
absolute numbers are tiny; none of them changes a gate outcome.

The p50/p99 trade on chat

Metric p50	before	after
request_delivery_ms	15.02	33.62
lifecycle_ready_to_send_ms	6.432	21.19
lifecycle_dispatch_to_pickup_ms	3.766	6.819

The change trades body for tail: the median request waits longer, the worst-case request waits far
less. Every gate in validator.py:76 is a p99 gate, so this trade is favourable as scored. Worth
stating explicitly that it is a trade — removing the self-throttling intake puts more requests in
flight, so the typical request contends with more work.

chat response delivery p99 (969 → 1634 ms) — confounded, not a real regression signal

This is the one metric that moves the wrong way at p99, and it is not interpretable from these runs.
Per-run chat response_delivery_ms p99:

• before: 963.6, 982.1, 964.8, 972.9 — tight
• after: 0.787, 1537, 2098, 1634, 1736 — bimodal

One after run was dramatically better on every metric (request delivery p99 266 ms vs
2681–2969 ms for the other four). One run in five landed in a much better operating point; 5 runs
cannot tell whether the change occasionally reaches a better steady state or that run simply caught a
quiet machine. Both commits are ~200× over the 5 ms threshold here, so neither produces a trustworthy
response-delivery number at this load and the change does not alter that.

The dispatch-drift and request-delivery wins are robust: they hold in all 5 after runs,
including the four "bad" ones.

---