feat(preflight): measurement-fidelity gate for the benchmark harness

[sicario001]

Summary

Veeksha reports server-side timing (ttfc/tpoc, delivery latencies) as ground truth. But those numbers are only trustworthy if the harness itself keeps time honestly — if it dispatches requests on schedule, streams paced input on schedule, and doesn't add its own queueing/transport drift on top of what the server does.

This PR adds veeksha preflight: a command that certifies exactly that. It drives the real scheduler → dispatch → client path against deterministic mock servers and gates the p99 timing drift against thresholds, so you can tell harness-induced drift apart from real server behavior before you trust a benchmark run.

It also back-ports the core instrumentation (lifecycle dispatch-drift metrics) into the normal performance evaluator as a soft WARN, so production runs surface the same signal without needing a mock.

Why

On a real benchmark you only see end-to-end numbers; you can't tell whether an inflated ttfc came from the server, from transport, or from the harness dispatching the request late. Preflight removes the server as a variable (deterministic mock with a known ttfc/tpoc schedule) and instruments both sides, so every millisecond of drift is attributable to a specific stage.

How it works — the record-book model

No timing values ever go on the wire. Each request carries only a correlation request-id header. Both sides independently stamp their own monotonic clock (comparable because mock and client run on the same host), and the two timelines are joined by request-id after the run.

Symbol legend — t_ is a timestamp; the subscript _i means "for the i-th chunk / segment" (0-indexed, so _0 is the first). c = client side, s = server side:

symbol	field	what it marks
t_cs	client_sent_at	client sent the request
t_sr	server_recv_time	server received the request
t_ss_i	server_send_times[i]	server sent response chunk i
t_cr_i	chunk_recv_times[i]	client received response chunk i
t_cs_i	input_send_times[i]	client sent input segment i (streaming-input WS)
t_sr_i	input_recv_times[i]	server received input segment i (streaming-input WS)

Concretely, the two record books hold:

• client records: client_sent_at (t_cs), per-chunk chunk_recv_times (t_cr_i), and — for streaming-input WS clients — input_send_times (t_cs_i) + the intended input_send_deadlines each segment was supposed to be sent at.
• server records: server_recv_time (t_sr), server_send_times (t_ss_i), input_recv_times (t_sr_i), and a response_start_time anchor so ttfc isn't polluted by the streaming-input phase.

The mock servers run as pinned subprocesses and serve their ground-truth record book back over /preflight/records.

Verdicts

• PASS — all gated p99 drifts within threshold.
• FAIL — the harness is off-schedule (dispatch drift and/or delivery lag exceed threshold) while the mock held its own pacing. The harness is the problem.
• SERVER_AT_CAPACITY — the mock couldn't hold its own ttfc/tpoc schedule (server-pacing gate breached). We refuse to blame the harness when the ground-truth generator itself is jittery.

The metrics

Grouped by what they certify (all in ms):

A. Lifecycle / dispatch drift (harness-only; no server needed)

metric	meaning
lifecycle_ready_to_dispatch_ms	dispatched_at − ready_at — scheduler queue lag
lifecycle_dispatch_to_pickup_ms	picked_up_at − dispatched_at — dispatcher → client-worker handoff
lifecycle_pickup_to_send_ms	t_cs − picked_up_at — client worker's own pre-send overhead
lifecycle_ready_to_send_ms	t_cs − ready_at — end-to-end dispatch drift (sum of the three)

B. Delivery lag (localhost transport/queueing; should be ~0)

metric	meaning
request_delivery_ms	t_sr − t_cs — request bytes client → server
response_delivery_ms	t_cr_i − t_ss_i per chunk — response bytes server → client

C. Server pacing fidelity (breach → SERVER_AT_CAPACITY)

metric	meaning
server_ttfc_abs_error_ms	abs((t_ss_0 − response_start) − configured_ttfc)
server_tpoc_abs_error_ms	abs((t_ss_i − t_ss_{i-1}) − configured_tpoc)

D. Streaming-input (WS clients only: realtime_tts / vajra_tts_stream / stt)

metric	meaning
input_delivery_ms	t_sr_i − t_cs_i — per-segment client → server delivery
input_pacing_abs_error_ms	abs(t_cs_i − deadline_i) — how far each paced segment slipped

E. Client-observed (informational, not gated) — what a client-only view would report

metric	meaning
client_observed_ttfc_ms	t_cr_0 − t_cs — end-to-end first-chunk latency
client_observed_tpoc_ms	t_cr_i − t_cr_{i-1} — end-to-end inter-chunk latency

Config & usage

Config is the same vidhi system benchmarks use, so every field is settable as a --flag, a --config config.yaml key, or a CLI sweep. There are four groups: which checks run, traffic, per-category workload/mock timing, and gate thresholds.

1. Which checks run

One boolean per client pathway (all True by default): --check-text (chat), --check-completions, --check-tts, --check-realtime-tts, --check-vajra-tts, --check-stt. The six clients fall into three workload categories — because their request/response shapes differ — and each category has its own config block:

category	clients	input	response the mock streams
text	chat, completions	single-shot text prompt	token stream (completions: one response)
tts	tts, realtime_tts, vajra_tts_stream	text prompt (streamed in paced segments for the two WS clients)	audio chunks
stt	stt	streamed audio	transcript deltas

2. Traffic & sizing

Traffic is closed-loop concurrency, configured with two scalar flags (preflight builds the scheduler itself with rampup_seconds=0):

• --concurrency (default 50) — how many sessions are in flight at once (the load level you're certifying at).
• --num-sessions (default 500) — total sessions to drive per check (the sample size). Raise it for more samples / tighter p99s.

# certify at concurrency 200 with 4000 sessions/check
veeksha preflight --concurrency 200 --num-sessions 4000

3. Per-category workload & mock timing

Each block controls two things: the input the harness generates, and the schedule the deterministic mock promises (its ttfc/tpoc — the ground truth that pacing fidelity is measured against).

--text.* — chat & completions

flag	default	meaning
--text.input_tokens	100	prompt length in tokens
--text.num_response_chunks	100	response tokens the mock emits (completions emits a single response)
--text.server_ttfc_ms	200	mock's promised time-to-first-chunk
--text.server_tpoc_ms	20	mock's promised inter-chunk (per-output-token) gap

--tts.* — tts (HTTP), realtime_tts & vajra_tts_stream (WS)

flag	default	meaning
--tts.input_tokens	100	prompt length in tokens
--tts.input_chunk_tokens	4	tokens per streamed input message (WS clients only)
--tts.input_pacing_tps	50	rate the input prompt is streamed at, tokens/sec (WS clients only)
--tts.num_response_chunks	100	audio chunks the mock emits
--tts.server_ttfc_ms	200	mock's promised time-to-first-audio-chunk
--tts.server_tpoc_ms	20	mock's promised inter-audio-chunk gap

The two WS clients stream the prompt in input_chunk_tokens-sized messages paced at input_pacing_tps — that pacing is exactly what the input_pacing_abs_error / input_delivery metrics grade. Plain HTTP tts sends the prompt in one shot and ignores those two fields.

--stt.* — stt (WS, audio-in)

flag	default	meaning
--stt.input_seconds	3.0	length of the generated audio clip
--stt.input_chunk_bytes	4096	audio bytes per streamed input message
--stt.sample_rate	16000	audio sample rate (Hz) — with input_seconds, fixes the clip's byte size and thus how many input messages it streams
--stt.num_response_chunks	40	transcript deltas the mock emits
--stt.server_ttfc_ms	200	mock's promised time-to-first-transcript-delta
--stt.server_tpoc_ms	50	mock's promised inter-delta gap

The audio is synthetic (from the new AudioChannelGenerator — a cached tone/silence clip), so no dataset is needed.

4. Gate thresholds & output

p99 ceilings, in ms unless noted (a breach flips the verdict): --dispatch-drift-threshold-ms (10), --delivery-lag-threshold-ms (5), --server-pacing-threshold-ms (5, breach → SERVER_AT_CAPACITY), --input-pacing-threshold-ms (10, WS-input only), --max-unpaired-fraction (0.02). --output-dir (preflight_report) is where the rendered report is written.

Examples

# defaults: concurrency 50, 500 sessions/check, all six checks
veeksha preflight

# one modality, longer text, faster mock, looser dispatch gate
veeksha preflight --check-text \
  --text.input_tokens 512 --text.server_ttfc_ms 50 --text.server_tpoc_ms 10 \
  --dispatch-drift-threshold-ms 20

# STT only, 10 s clips streamed in 8 KB messages, at concurrency 100
veeksha preflight --check-stt \
  --stt.input_seconds 10 --stt.input_chunk_bytes 8192 \
  --concurrency 100

# reproducible config file instead of flags
veeksha preflight --config preflight.yaml

Results

Environment: single unpinned 10-core macOS box, free-threaded CPython 3.14t (PYTHON_GIL=0), all six mock servers + clients contending for the same cores. All config values are the stock veeksha preflight defaults — concurrency 50, 500 sessions/check; text/tts mock timing 200 ms ttfc / 20 ms tpoc over 100 response chunks and STT 200 ms / 50 ms over 40 deltas; 100-token prompts (tts/vajra input paced at 50 tokens/s over ~25 deltas), 3 s STT audio; gates dispatch-drift-threshold 10 ms, delivery-lag-threshold 5 ms, server-pacing-threshold 5 ms, input-pacing-threshold 10 ms, max-unpaired-fraction 2%. A warmup pass precedes the measured run so first-touch costs (librosa import, synthetic-WAV generation, connection setup) don't land as cold-start outliers.

Concurrency 50, 500 sessions per client — this run

metric (p99 ms)	chat	completions	tts	realtime_tts	vajra_tts	stt
lifecycle_ready_to_send	13.7	15.7	13.6	7.0	8.8	10.4
lifecycle_dispatch_to_pickup	13.0	15.5	13.1	4.9	5.2	6.2
request_delivery	15.4	19.3	73.6	9.7	8.6	11.3
response_delivery	2.0	16.2	2.3	2.1	2.0	2.8
server_ttfc_abs_error	1.4	4.2	6.1	7.0	4.4	3.0
server_tpoc_abs_error	2.3	–	3.4	4.4	3.9	2.8
input_delivery (WS)	–	–	–	1.9	2.4	2.4
input_pacing_abs_error (WS)	–	–	–	3.7	6.4	4.4
client_observed_ttfc (info)	217.1	229.6	273.8	2214.0	2215.1	3216.2
client_observed_tpoc (info)	22.5	–	23.1	23.8	23.2	53.1
verdict	FAIL	FAIL	SERVER_AT_CAPACITY	SERVER_AT_CAPACITY	FAIL	FAIL

– = metric doesn't apply (completions is single-response → no tpoc; input metrics are WS-only).

---

PR Checklist (Click to Expand)

Thank you for your contribution to Veeksha! Before submitting the pull request, please ensure the PR meets the following criteria. This helps Veeksha maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [Feat] for new features.
• [Core] for changes in the core benchmarking logic
• [CI/Build] for build or continuous integration improvements.
• [Docs] for documentation fixes and improvements.
• [Tests] for changes in the test suite.
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR needs to meet the following code quality standards:

• Pass all linter checks. Please use make format to format your code.
• The code needs to be well-documented so future contributors can easily understand it.
• Please add documentation under docs/ if the PR changes user-facing behavior. This helps users understand and adopt the new behavior.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to Veeksha. Your contributions make Veeksha a great tool for everyone!