HTTP server perf excellence — beat vLLM/SGLang on every bottleneck

[AgrawalAmey]

Goal

No HTTP-server bottleneck should make Vajra slower than vLLM or SGLang. Every per-request / per-token / per-chunk cost on the wire path gets audited against both references, with file:line citations for any claim.

Baseline (what we already do better)

Vajra's streaming write path is genuinely ahead of both:

• C++ AnthropicFormatter / ChatCompletionFormatter / ResponsesFormatter emit pre-framed SSE bytes into an os.pipe(). Python forwards bulk chunks via asyncio.StreamReader — zero per-token Python work, zero per-chunk Pydantic serialization.
• vLLM and SGLang both do yield f"data: {chunk.model_dump_json()}\n\n" per token in Python (vllm/entrypoints/openai/chat_completion/serving.py:1044-1045, python/sglang/srt/entrypoints/openai/serving_chat.py:949,970,1018,1058,1110). We should not regress this.

Gaps (tracked as sub-issues)

1. VJ-185 — Abort session on client disconnect. AsyncEngine.abort() is a no-op. Both references have well-developed disconnect-to-abort paths; we have none.
2. VJ-186 — Real usage token counts from C++ engine. All endpoints return usage.prompt_tokens=0 today.
3. Move auth middleware off BaseHTTPMiddleware to raw ASGI. vLLM did this deliberately (vllm/entrypoints/openai/server_utils.py:32-68); the wrapper-task overhead and receive() interference are both real.
4. Drop the Responses API response round-trip: responses_processor.py:169-182 does ResponsesCreateResponse(**response.model_dump()) to "validate" an already-validated Pydantic model, then model_dump() is called again at responses_routes.py:89. Triple round-trip.
5. Switch non-streaming responses from JSONResponse (stdlib json) to ORJSONResponse. Non-critical (not the hot path) but cheap.
6. Move the OpenAI Realtime setup/completion events (6-10 per response) to C++ formatter instead of Python model_dump() + json.dumps loop in base_transport.py:29.
7. Batch detokenization for logprobs. openai/handlers/base.py:52, 65 decodes tokens one at a time in a Python loop when logprobs requested.
8. SSE keepalive comment frames (: keep-alive\n\n) every ~30s, for deployments behind proxies with short idle timeouts. Neither reference does this; only ship if we observe proxy drops in practice.

Non-gaps (after verification)

• orjson everywhere: not needed on streaming path — we already do better by assembling in C++. For non-streaming it's a marginal win (item 5 above).
• msgspec: nobody uses it for OpenAI-compat models. Pydantic v2 parse is not measurable next to first-token latency.
• uvloop tuning: already loop="uvloop" in server.py:198.
• Pydantic extra="allow": matches vLLM's deliberate choice (vllm/entrypoints/openai/engine/protocol.py:24-45). SGLang's default-ignore is arguably the worse choice.
• Conversation-store OrderedDict "race": asyncio single-loop semantics make dict ops atomic between awaits; our helpers don't await mid-op.

Novel patterns worth stealing

• vLLM with_cancellation decorator (vllm/entrypoints/utils.py:34-72) — captured in VJ-185.
• SGLang StreamingResponse(..., background=create_abort_task(...)) pattern (python/sglang/srt/entrypoints/http_server.py:1171) — captured in VJ-185.
• vLLM OpenAIBaseModel validator logging unrecognized fields (vllm/entrypoints/openai/engine/protocol.py:24-45) — low-effort API-drift detector.

References

• Full comparison walkthrough: vLLM vllm/entrypoints/openai/api_server.py, chat_completion/serving.py, v1/engine/async_llm.py; SGLang python/sglang/srt/entrypoints/http_server.py, openai/serving_chat.py, managers/tokenizer_manager.py.

Tracking

This issue is the umbrella. Individual work lives in the sub-issues. Close this when all child sub-issues close or we have evidence via daily perf comparison (see VJ-63 if revived) that Vajra's server-path latency is at or below both references on every workload.