Surface real usage token counts from C++ engine (all endpoints)

[AgrawalAmey]

Problem

Every Vajra HTTP endpoint returns hardcoded prompt_tokens=0, completion_tokens=0, total_tokens=0 in the OpenAI/Anthropic usage field. This is broken for anyone running real clients — it breaks billing, rate limiting, evaluation harnesses, and anything that reads usage off the response.

We deliberately stubbed these (commit faca00f85) to avoid double-tokenizing Python-side; the engine already tokenizes every prompt in TokenizerPool and generates token IDs. The counts exist in C++ — they just never reach the Python layer.

Current state

Endpoint	Usage source	Status
OpenAI chat non-streaming	hardcoded zero (chat_processor.py:95-97)	broken
OpenAI chat streaming	— (no usage in streaming chunks yet)	broken
OpenAI completions	hardcoded zero (completions_processor.py:143-144)	broken
OpenAI responses	hardcoded zero (responses_processor.py:108-110)	broken
Anthropic messages non-streaming	AnthropicUsage(input_tokens=0, output_tokens=0)	broken
Anthropic messages streaming	input_tokens=0 in message_start, output_tokens=0 in message_delta (emitted by AnthropicFormatter in C++)	broken

Canonical token-count owners (already in C++):

• Prompt tokens: csrc/vajra/native/core/tokenizer/TokenizerPool.cpp:54 (after Tokenizer->Encode(input.prompt))
• Generated tokens: csrc/vajra/native/llm/controller/replica/utils/OutputGenerator.cpp:92+ (every sampled token passes through here)

Desired behavior

1. DeltaOutput carries cumulative_output_tokens (or equivalent) so streaming can emit real counts in-flight.
2. The first delta (or a dedicated initial record) carries prompt_tokens.
3. generate_complete returns (text, finish_reason, prompt_tokens, completion_tokens).
4. All Python processors read these values directly — zero re-tokenization.
5. Anthropic SSE writer fills in usage.input_tokens and usage.output_tokens with real counts.

Implementation sketch

1. DeltaOutput schema: add prompt_tokens: optional<int32> (set on first delta) and completion_tokens: int32 (running count). Update pybind + Python bindings.
2. OutputGenerator: emit the running count each delta.
3. Session ingestion: after TokenizerPool::Encode, thread prompt token count through SessionData so OutputGenerator can stamp it on the first delta.
4. Python processors: replace every prompt_tokens=0 / input_tokens=0 with the real value from the engine result.
5. Anthropic SSE writer (csrc/vajra/native/entrypoints/openai/AnthropicFormatter.cpp): replace the two output_tokens=0 stubs with the counter already tracked locally, and source input_tokens from DeltaOutput's first delta.
6. OpenAI chat streaming: add a final "usage" chunk when stream_options={"include_usage": true} is passed (matches OpenAI API).

Reference

• vLLM reads engine-surfaced counts at vllm/entrypoints/openai/chat_completion/serving.py:905, 915, 1027, 1163-1167.
• SGLang reads meta_info counts at python/sglang/srt/entrypoints/openai/serving_chat.py:867-877.

Acceptance criteria

• All five endpoints return non-zero, tokenizer-accurate usage.
• No tokenizer.encode() calls remain on the HTTP hot path (verify with grep).
• Streaming emits usage per Anthropic's message_delta and OpenAI's include_usage contract.
• Integration test with a real tokenizer confirms counts match len(tokenizer.encode(text)).

Notes

• The current Python-side stubs (commit faca00f85) are documented placeholders — this issue replaces them.
• Touches the C++ DeltaOutput contract, so coordinate with any other OutputGenerator / AsyncEngine consumers.