[TRTLLM-14511][feat] refactor per-request perf metrics for multi-process trtllm-server

[reasonsolo]

Doc: TRTLLM-Serving Performance Metrics Design

• Replaces /perf_metrics polling with per-request metrics returned through response headers or an opt-in post-[DONE] SSE event.
• Adds optional asynchronous JSONL dumping using the former /perf_metrics schema plus status and disagg_request_id.
• Supports aggregated and disaggregated serving; ctx/gen metrics are combined by the serving fleet worker, never the coordinator.
• Public metrics require both return_perf_metrics=true and X-TRTLLM-return-metrics: 1; file dumping is independently controlled by perf_metrics_output_dir.
• Adds shared typed schemas, time-breakdown support, and agg/disagg E2E coverage.

Dev Engineer Review

• Perf metrics exposure refactor (API-compatible intent): Removed /perf_metrics polling/endpoint usage and replaced it with:
  • Header-based delivery gated by both return_perf_metrics=true and X-TRTLLM-return-metrics: 1.
  • Streaming delivery via SSE as an opt-in dedicated post-[DONE] metrics event (SSE event is suppressed from user-visible streaming output).
  • Server-side JSONL output via perf_metrics_output_dir independent of header return.
• Disaggregated propagation & gating correctness:
  • DisaggServerConfig extended with return_perf_metrics and perf_metrics_output_dir; extract_disagg_cfg(...) now propagates/forces downstream context_servers and generation_servers to return internal metrics when either gating/JSONL options are enabled.
  • Updated config inheritance logic to avoid iterating directly over mutable kwargs, and switched to explicit keyword instantiation.
• Typed, schema-driven metrics:
  • Added tensorrt_llm/serve/_perf_metrics_schema.py with TypedDict records supporting aggregated + disaggregated phases and a unified PerfMetricsRecord union.
• Server pipeline overhaul:
  • tensorrt_llm/serve/perf_metrics.py now owns building metrics records, generating metric headers, SSE emission, and JSONL persistence (PerfMetricsJsonlWriter), plus disagg record combination (combine_disagg_metrics).
  • Removed the in-memory deque/locking /perf_metrics implementation; metrics records are now attached to raw_request.state.perf_metrics_record and emitted at the correct time (explicitly awaited at end of streaming/non-streaming flows).
• Client behavior changes:
  • OpenAIHttpClient now sets the return-metrics header when request_perf_metrics is enabled, extracts metrics from response headers/SSE payloads, and emits via hooks; removed the old abstract/concrete collect_metrics() plumbing.
  • Added/adjusted robustness around streaming metrics payload parsing and retry-related disagg_request_id hook invocation.
• Metrics header sizing fix: Commit fix: allow large disagg metrics headers updates implementation to support larger disaggregated metrics headers (important given the move to header-carried metrics).
• Tooling & scripts alignment:
  • Benchmark scripts and time-breakdown tooling updated to consume server-written JSONL instead of fetching /perf_metrics.
  • Streaming helpers updated so [DONE] terminates SSE iteration immediately (return), preventing accidental extra chunk processing.

CI failures noted in context: Two CI runs failed after the refactor/endpoint removal:

• PR_Github run #60916 (pipeline #49181, commit 5e0ac04)
• PR_Github run #61165 (pipeline #49414, commit d9ff361)

Recommendation for investigation: prioritize any tests asserting /perf_metrics behavior, then validate header gating correctness and SSE post-[DONE] emission/suppression, and finally confirm JSONL writing/reading expectations (including large-header handling) across aggregated + disaggregated flows.

---

QA Engineer Review

Test code/config changes detected (under tests/)

Integration / config updates

• tests/integration/defs/accuracy/test_disaggregated_serving.py
  • Updated perf metrics enablement to use perf_metrics_output_dir (derived from temp dir) instead of perf_metrics_max_requests.
  • Removed local /perf_metrics polling helper.
• tests/integration/defs/disaggregated/test_disaggregated.py
  • Updated cluster setup to accept perf_metrics_output_dir, inject return_perf_metrics, and validate via JSONL artifacts instead of HTTP /perf_metrics.
  • Updated disagg router/perf metrics assertions accordingly.
• tests/integration/defs/disaggregated/test_configs/*.yaml
  • disagg_config_conditional_deepseek_v3_v2.yaml: set top-level return_perf_metrics: true (kept/updated rationale comment).
  • disagg_config_metrics.yaml: set top-level return_perf_metrics: true.

Test utilities

• tests/test_common/perf_metrics_utils.py
  • Refactored metric retrieval to read perf_metrics-*.jsonl from output_dir.
  • Added/changed helpers: read_perf_metrics_jsonl, wait_for_perf_metrics_jsonl, updated get_timing_metrics(...).

Unittest coverage

• tests/unittest/disaggregated/test_disagg_utils.py
  • Added test_extract_disagg_metrics_controls.
• tests/unittest/llmapi/apps/_test_openai_perf_metrics.py
  • Removed test_metrics_endpoint
  • Added test_return_perf_metrics_and_jsonl_dump
  • Added test_streaming_metrics_require_request_opt_in
• tests/unittest/llmapi/apps/_test_openai_chat_harmony_perf_metrics.py
  • Updated tests to validate via JSONL output (no /perf_metrics drain/count).
• tests/unittest/llmapi/apps/test_disagg_perf_metrics_collector.py
  • Removed entire module.
• tests/unittest/llmapi/apps/test_disagg_serving_perf_metrics.py
  • Added test_return_perf_metrics_and_jsonl_dump; threaded perf_metrics_output_dir through fixtures and server config.
• tests/unittest/llmapi/apps/test_request_metrics.py
  • Added new test suite covering header formatting/mapping, middleware behavior (headers + SSE), JSONL writer persistence, and record combination/parsing.
• tests/unittest/others/test_time_breakdown.py
  • Added test_parse_disaggregated_time_breakdown
  • Added test_parse_jsonl_file
• Multi-node disagg apps:
  • _test_disagg_serving_multi_nodes.py and _test_disagg_serving_multi_nodes_service_discovery.py remove perf/timing-metrics plumbing and assertions.

API stability reference

• tests/unittest/api_stability/references/llm.yaml
  • Added perf_metrics_output_dir
  • Marked perf_metrics_max_requests as deprecated.

Coverage mapping vs test lists (CI/manual)

• No changes under tests/integration/test_lists/ are mentioned in the provided change list; however, the repo diff couldn’t be validated in this environment. Treat test-list coverage verification as not confirmed.

Verdict

needs follow-up — the change is substantial (removing /perf_metrics endpoint behavior and switching to header/SSE/JSONL), and the context includes two failing CI runs likely tied to metrics gating/emission/JSONL expectations and/or the disaggregated large-header fix.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ed099721-19b3-4634-836b-dcffcd063fa4

📥 Commits

Reviewing files that changed from the base of the PR and between 4f9117c and dbce2f7.

📒 Files selected for processing (3)

• tensorrt_llm/serve/openai_client.py
• tensorrt_llm/serve/perf_metrics.py
• tests/unittest/disaggregated/test_disagg_openai_client.py

🚧 Files skipped from review as they are similar to previous changes (2)

• tensorrt_llm/serve/openai_client.py
• tensorrt_llm/serve/perf_metrics.py

---

Walkthrough

Replaces the HTTP /perf_metrics endpoint and in-memory buffering with header/SSE metrics and optional server-side JSONL output. Adds configuration fields, typed schemas, client/server propagation, updated tooling, and tests.

Changes

Server-side JSONL performance metrics pipeline

Layer / File(s)	Summary
Configuration and metrics contracts tensorrt_llm/llmapi/disagg_utils.py, tensorrt_llm/llmapi/llm_args.py, tensorrt_llm/serve/_perf_metrics_schema.py	Adds metrics configuration, disaggregated propagation, typed records, and API metadata.
Metrics serialization and persistence tensorrt_llm/serve/perf_metrics.py, tests/unittest/llmapi/apps/test_request_metrics.py	Builds, combines, serializes, persists, and exposes performance metrics through headers and SSE.
OpenAI client metrics propagation tensorrt_llm/serve/openai_client.py, tensorrt_llm/serve/responses_utils.py	Adds request opt-in, response header/SSE parsing, hook delivery, request-ID updates, and stream completion handling.
OpenAI and disaggregated server integration tensorrt_llm/serve/openai_server.py, tensorrt_llm/serve/openai_disagg_server.py, tensorrt_llm/serve/openai_disagg_service.py	Gates collection, manages writers and middleware, combines disaggregated metrics, removes the old endpoint, and records completed requests.
Benchmark and time-breakdown tooling tensorrt_llm/serve/scripts/*, examples/disaggregated/slurm/benchmark/run_benchmark_aiperf.sh	Removes HTTP perf-metrics fetching, updates stream completion handling, and supports server-side JSONL input.
Integration and application validation tests/integration/*, tests/unittest/*, tests/test_common/perf_metrics_utils.py	Configures local JSONL output and validates metrics propagation, persistence, joining, headers, SSE, and timing parsing.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIClient
  participant PerfMetricsMiddleware
  participant PerfMetricsJsonlWriter
  OpenAIClient->>PerfMetricsMiddleware: send metrics-enabled request
  PerfMetricsMiddleware->>OpenAIClient: return headers or SSE metrics event
  PerfMetricsMiddleware->>PerfMetricsJsonlWriter: submit completed record
  PerfMetricsJsonlWriter->>PerfMetricsJsonlWriter: write JSONL

Loading

sequenceDiagram
  participant OpenAIHttpClient
  participant RawRequestResponseHooks
  participant OpenAIDisaggServer
  OpenAIHttpClient->>RawRequestResponseHooks: on_perf_metrics(server, role, metrics)
  RawRequestResponseHooks->>RawRequestResponseHooks: combine_disagg_metrics
  RawRequestResponseHooks->>OpenAIDisaggServer: store perf_metrics_record

Loading

Suggested reviewers: pcastonguay

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change and follows the repository’s ticket-and-type format.
Description check	✅ Passed	The description covers the change summary, but it lacks an explicit Test Coverage section and is lighter than the template.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #60916 [ run ] triggered by Bot. Commit: 5e0ac04 Link to invocation

[tensorrt-cicd]

PR_Github #60916 [ run ] completed with state FAILURE. Commit: 5e0ac04
/LLM/main/L0_MergeRequest_PR pipeline #49181 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[pcastonguay]

Removing /perf_metrics is a breaking change and the PR title should reflect that. Older benchmark_serving.py or run_benchmark_aiperf.sh would now fail I believe.

[reasonsolo]

I have removed /perf_metrics fetching from benchmark script, and added jsonl file parsing.

[reasonsolo]

I added save-breakdown feature back to benchmark_serving.py, but it's requiring an extra jsonl dump dir now.

[pcastonguay]

I understand we kept the functionality but since we removed the /perf_metrics endpoint, it could break clients outside TRT-LLM that are still using that endpoint. Therefore I would prefer if we mark it as a breaking change so it at least gets captured in the release notes.

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61165 [ run ] triggered by Bot. Commit: d9ff361 Link to invocation

[tensorrt-cicd]

PR_Github #61165 [ run ] completed with state FAILURE. Commit: d9ff361
/LLM/main/L0_MergeRequest_PR pipeline #49414 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61270 [ run ] triggered by Bot. Commit: dbce2f7 Link to invocation

[tensorrt-cicd]

PR_Github #61270 [ run ] completed with state FAILURE. Commit: dbce2f7
/LLM/main/L0_MergeRequest_PR pipeline #49505 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[reasonsolo]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61508 [ run ] triggered by Bot. Commit: dbce2f7 Link to invocation