[None][feat] cache transceiver test in Perf sanity#16674
Open
chuangz0 wants to merge 19 commits into
Open
Conversation
chuangz0
requested review from
BowenFu,
EmmaQiaoCh,
QiJune,
ZhanruiSunCh,
dc3671 and
nv-ananjappa
July 21, 2026 12:09
Contributor
📝 WalkthroughWalkthroughChangesAdds a disaggregated cache-transceiver precheck with validated configuration planning, deterministic KV transfer verification, authenticated MPI coordination, SLURM launch integration, node pinning, failure summaries, JUnit output, and CPU-only tests. Runtime environment setup is centralized for both regular tests and prechecks. Cache-transceiver precheck
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
BowenFu
approved these changes
Jul 21, 2026
BowenFu
left a comment
BowenFu
left a comment
There was a problem hiding this comment.
LGTM — additive CI/perf-sanity + test tooling; no tensorrt_llm/ runtime code touched. The new disagg cache-transceiver precheck gate + node-pinning is scoped to the perf-sanity launch (not the pre-merge gate) and opt-out via TRTLLM_DISAGG_CT_PRECHECK=0 / per-yaml enabled:false; the -30 is a behavior-preserving extraction of the env setup into slurm_env_setup.sh. New .py files carry SPDX headers; the new unit test is CPU-only. Regression-safe.
Collaborator
|
@chenfeiz0326 Can you review this? It's related to Perf CI |
dc3671
removed their request for review
July 22, 2026 02:58
chuangz0
force-pushed
the
perf_sanity_cache_transceiver_test
branch
from
July 22, 2026 08:36
e7f9ee6 to
5ade49d
Compare
Collaborator
Author
|
/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*" |
Collaborator
|
PR_Github #60938 [ run ] triggered by Bot. Commit: |
chenfeiz0326
approved these changes
Jul 22, 2026
Collaborator
Author
|
/bot kill |
Collaborator
|
PR_Github #60946 [ kill ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #60938 [ run ] completed with state |
Collaborator
|
PR_Github #60946 [ kill ] completed with state |
Shixiaowei02
approved these changes
Jul 22, 2026
| # Python transceiver's perf_<uuid>_<rank>.csv are per-run and appended) | ||
| # would make parse_python_bandwidth_gbps median over two runs' samples. The | ||
| # driver also job-id-stamps addr files as a second line of defense. | ||
| rm -f "$precheckDir"/rendezvous/*.addr "$precheckDir"/status/*.status \ |
Collaborator
There was a problem hiding this comment.
The stale-file cleanup misses precheck.abort.
Collaborator
Author
There was a problem hiding this comment.
Fixed in e910c35. Good catch — worse than an omission: a requeued job keeps its SLURM_JOB_ID, so the driver's job-id stamp cannot reject a stale abort flag and the rerun would be falsely fail-fast-skipped. The launch-side cleanup is the only reliable defense; comment updated to say so.
chuangz0
force-pushed
the
perf_sanity_cache_transceiver_test
branch
from
July 22, 2026 10:01
ca516e1 to
4cbab86
Compare
Collaborator
Author
|
/bot run --disable-fail-fast --stage-list "GB200PerfSanity*,GB300PerfSanity*" |
Collaborator
|
PR_Github #61271 [ run ] completed with state
|
fredricz-20070104
approved these changes
Jul 24, 2026
fredricz-20070104
left a comment
fredricz-20070104
left a comment
Collaborator
There was a problem hiding this comment.
Test passed. Approve this one.
Collaborator
|
Check the passed QA pipeline for more details. https://gitlab-master.nvidia.com/dlswqa/trtllm-ci/-/pipelines/59368729 |
…nity Standalone MPI driver that transfers deterministic KV data ctx->gen through the cache transceiver using the SAME instance topology, cache_transceiver_config, and KV shape (from the model's config.json) as a disagg perf-sanity test's yaml. Supports asymmetric parallelism (pattern seeded per (request, global layer), constant along the KV-head axis) and attention DP (per-dp-rank request pairing), multi-instance (ctx x gen) pairing via file-based rendezvous + per-pair ZMQ sessions, and reports PASS/TIMEOUT/TRANSFER_ERROR/MISMATCH/INIT_ERROR per case with per-GPU bandwidth. Config resolution is a pure-stdlib module with CPU-only unit tests. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com> [None][test] Use SELFKONLY cache type for MLA in cache transceiver precheck Real MLA serving creates the KV cache manager with CacheType.SELFKONLY (kv_factor=1: a single latent plane, no V) — see _torch/pyexecutor/_util.py. The precheck driver used CacheType.SELF unconditionally, doubling the per-token bytes for MLA models, which skews buffer chunking behavior and reported bandwidth relative to the real test. Fill/verify already read kv_factor from the buffer view shape, so no other change is needed. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…recheck Before starting the real ctx/gen servers, the generated launch script now optionally runs one precheck srun per server instance with the same node slices (deterministic SLURM_JOB_NODELIST slicing + srun -w, also applied to the real server steps), the same MPI topology, and the same UCX env: submit.py builds the precheck commands from the exact ucx_tls_cmd + CTX/GEN_WORKER_ENV_VARS strings used by the worker steps, and the shared slurm_env_setup.sh (extracted from slurm_run.sh) applies the same LD_LIBRARY_PATH / 'unset UCX_TLS=tcp' / PMIX_MCA_gds fixups. The gate logic lives in slurm_ct_precheck_gate.sh as functions; submit.py splices it into the launch script ahead of slurm_launch_draft.sh, which just calls run_cache_transceiver_precheck. On precheck failure the stage aborts before any model bring-up, with per-instance verdicts, the first error lines + tail of each failing step log, and UCX red-flag lines (sw-emul tcp fallback), and writes a synthetic junit xml to $jobWorkspace/results-ct-precheck.xml so the failure shows up as a test entry in the Jenkins report (uploadResults already collects results*.xml). On by default; disable per yaml via cache_transceiver_precheck.enabled: false, or globally with TRTLLM_DISAGG_CT_PRECHECK=0 at generation time (=1 force-enables; the env var overrides the yaml either way). The local perf flow (jenkins/scripts/perf/local) gets the same exports. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com> [None][infra] Make the precheck run script work with the local perf flow layout slurm_precheck_run.sh hardcoded the CI layout (cd $resourcePathNode; llmSrcNode=$resourcePathNode/TensorRT-LLM/src), but the local perf flow (jenkins/scripts/perf/local/submit.py) exports llmSrcNode directly and has no resourcePathNode -- with set -u the precheck step died instantly. Honor whichever layout is present, and fail with a clear message when neither is exported. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…fails The failure branch of the Slurm job tracking script only said 'Pytest failed'; the Artifactory URL of the full test output (results-<stage>*.tar.gz) was printed much later by uploadResults, far from where people read the failure. Echo the URL right at the failure line so the console points straight at the full logs. Applies to every Slurm-run stage, not just perf sanity. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
The precheck's ctx leaders bind ZMQ REP sockets reachable from the cluster network and previously pickle.loads()'d whatever arrived -- unpickling network-supplied bytes is arbitrary code execution. Control messages are now JSON with an appended HMAC-SHA256 tag, verified with a constant-time compare before parsing. Each (ctx, gen) session gets a fresh 32-byte key generated by the ctx leader and shared only through the work-dir rendezvous addr file (created 0600 on the shared filesystem -- the job's trust domain), never over the network, so a network-only attacker can neither forge nor tamper frames. ContextPhaseParams crosses the wire as its primitive fields with base64 opaque_state (mirroring disaggregated_params.py / executor/result.py) instead of being pickled. Adds CPU-only unit tests for the wire format (roundtrip, tampered frame, wrong key, short frame, addr file permissions). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Follows the proven pattern from examples/disaggregated/slurm/cache_transceiver_test (which exercises V2 with the Python transceiver end-to-end): a KvCacheConfigV2 dataclass stand-in, prepare_context/resize_context (ctx) and prepare_disagg_gen_init (gen) for allocation, free_resources() for release (also frees the IndexMapper slot), and is_disagg=True for in-flight-transfer slot headroom. fill/verify are unchanged -- get_buffers()/get_batch_cache_indices() alias the real pool on both V1 and V2. vocab_size (a V2 ctor input) now comes from the model's config.json. Manager version and transceiver runtime resolve exactly like serving (PR NVIDIA#15823): explicit yaml values win; an absent use_kv_cache_manager_v2 means 'auto' and adopts the model class's get_model_defaults() (e.g. DeepseekV4 defaults to V2), and transceiver_runtime 'auto' resolves through the REAL llm_utils._resolve_transceiver_runtime_auto against model_cls.get_preferred_transceiver_runtime() with the same NIXL gate. The model class comes from config.json architectures via MODEL_CLASS_MAPPING, mirroring the automodel path. Effective values are printed by the leader and recorded in the status json. V2 only works with the Python transceiver (see cache_transceiver_test/report.py), so a V2 + CPP pairing is rejected up front with a clear INIT_ERROR instead of a C++ binding type error. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…eused work dirs A reused --work-dir (Slurm requeue reruns the batch script with the same directories; manual reruns) could hold a previous run's rendezvous addr files: gen leaders would connect to a dead host:port and block until the hello timeout, misreporting TIMEOUT. Stale status files would also pollute the launch script's verdict aggregation. Three layers: - the gate script clears rendezvous/status artifacts before launching, - write_addr removes any stale file before atomically publishing, and - addr files are stamped with SLURM_JOB_ID; wait_for_addr skips files stamped by another run (empty job id -- manual non-Slurm runs -- disables the check), so even a crash path that skipped cleanup cannot hand a gen leader a dead endpoint. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
… all peers With multiple ctx servers, the gen sent 'done' to each ctx right after that ctx's schedule, so ctx_0's process exited while the gen went on to ctx_1 -- leaving the gen's C++ transceiver holding connections to a dead NIXL agent, a state real serving never produces (ctx servers outlive the whole run). Observed on a ctx2xdep4 -> gen1xdep16 topology: the ctx_0 session passed, then ctx_1's first chunk stalled until the chunk timeout. The gen now defers 'done' to every ctx until ALL its peers' schedules have finished (gen_release_peer), and the ctx budgets its final wait like a handshake, since it can legitimately span the gen's remaining sessions. Session order stays deterministic (every ctx serves gens in index order, every gen visits ctxs in index order), so the deferred release cannot deadlock. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
- gen: periodic progress while waiting for a receive (rid, request state, elapsed) so a stalled transfer shows whether it never matched (request-info phase) or the RDMA write never completed; per-chunk recv timing. - ctx: per-chunk send timing with final request states. - PRECHECK_DEBUG=1 raises the C++ (TLLM_LOG_LEVEL, set before the tensorrt_llm import) and Python logger levels for deep debugging. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Root cause of the pp8 ctx -> dep16 gen false positive (reproduced on three node sets while the REAL test with the same yaml passes): the C++ NIXL path pays a one-time fetchRemoteMD metadata exchange per (receiver rank, ctx rank) agent pair, serialized under a lock with an unbounded retry loop, and a cold cross-rack fetch was measured at 100-170s per pair. MLA + ctx pp8 makes every receiving rank wire up to all 8 ctx agents (dep16 -> 128 pairs over 4 nodes / 2 racks), so the first chunk blew the 180s chunk alarm; the watchdog SIGKILL mid-fetch also produced the UCX-keepalive segfaults seen on the ctx side. Real serving absorbs the wire-up as slow first requests (no hard alarm, observe-only kv_transfer_timeout), so the precheck must too. The first rep of the schedule (the warmup rep) now gets an extra wireup_timeout_s budget (default min(1800, 150 * max world size), yaml overridable) on both sides' chunk alarms, the handshake budget, the gen ZMQ backstop, and the submit.py external step timeout. Later reps keep the tight 180s alarm, preserving the fast-fail property once connections exist. Also fix rid construction: the C++ notification tag is rid & 0xFFF and all non-pair strides were multiples of 4096, so every request with the same pair index across reps/lengths shared one (agent, tag) key -- stray notifications from a previous rep could be consumed by the wrong request. Rids are now peer-session base + dense in-session sequence, keeping tags unique within any 4096-request window (sessions talk to distinct agents, so the peer stride cannot alias). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…path Quality pass over the precheck stack (no behavior change; all topologies re-validated on hardware afterwards): - precheck_config now owns every launch-facing policy once: precheck_prefix_lines() builds the gate export lines (enable / kill-switch semantics, step-timeout default) for BOTH jenkins/scripts/perf/submit.py and the local flow, which previously carried a drifting ~40-line copy each; wireup_timeout_s()/ default_step_timeout_s() replace the formula that existed in three files. - run_precheck: reuse the maintained converters instead of local re-implementations (DisaggregatedParams.get_context_phase_params for the wire decode, llm_utils._resolve_kv_cache_manager_v2_auto for the V2 'auto' rule, the real pydantic KvCacheConfig instead of a 25-field stand-in dataclass); one exception->verdict mapper replaces three hand-rolled per-peer blocks; fill/verify expand the pattern on device (was transferring heads x the unique bytes), compute shapes without a throwaway gather, skip byte-verification for warmup reps (their result is discarded), and V2 frees sync the stream once per chunk instead of per request; assorted small cleanups (dead state, redundant conditionals, duplicate loop tests, stale-file pre-remove that os.replace + job stamps already cover). - gate script: shared ct_step_passed/ct_step_excerpt keep the console summary and the junit xml from drifting; one ct_launch_step function replaces the twin gen/ctx srun loops. - per-side plan fields in resolve_plan are built by one loop; side_plan drops the unused peer_parallel; tests import run_precheck at module level (it is stdlib-only by design). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
- main() is now a short narrative (parse -> env -> topology check -> watchdog -> setup gate -> sessions -> verdict): the watchdog wiring moved to _install_watchdog(), the exception->verdict mapping to _make_peer_failure_recorder(), and the two session drivers to _serve_gen_peers() / _drive_ctx_peers() with role-level docstrings. - The leader-to-leader session protocol now has an ASCII sequence diagram at the section header (hello/welcome, go/params, deferred done/bye). - 'chunk' is defined everywhere a reader first meets it (module vocabulary block, pcfg.chunks(), the max_concurrent_pairs knob): a chunk is the batch of TRANSFER PAIRS in flight at once, NOT token/data chunking -- the term collides with chunked prefill and confused a reviewer. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com> [None][test] CPU-level coverage for multi-peer precheck orchestration Runs the REAL 2-ctx x 1-gen session protocol in-process (real PrecheckRunner, ZMQ loopback sockets, HMAC frames, StatusRecorder, rendezvous files; only the GPU transfer methods stubbed): full-pass asserts every ctx serves the complete schedule and receives the deferred done, and the failure-isolation case asserts one aborting ctx yields TRANSFER_ERROR on the gen while the healthy peer still passes and is released. This is the multi-instance logic the hardware 'B' topology exercises, now guarded without GPUs. Addr files publish loopback (SLURMD_NODENAME=127.0.0.1) since CI/sandbox hosts may not resolve their own hostname. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
The env snapshot in status/*.json exists to capture variables that influence transport behavior (UCX_/TRTLLM_/TLLM_). The only NIXL_* variable ever observed is NIXL_VERSION, a leftover marker from the NGC base image's bundled NIXL rather than the library TRT-LLM links from /opt/nvidia/nvda_nixl (it read '1.0.0' in containers shipping NIXL v1.3.0), so it misstated the transport version in every verdict. Drop the NIXL_ prefix from the snapshot entirely. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…fied pool; add Python-transceiver bandwidth Two follow-ups so the precheck covers more real configs: 1. Sparse-attention models (DeepSeek V4 / DSA: an indexer INDEX_KEY pool separate from the main K/V pool) cannot be expressed as the precheck's single (layers, kv_heads, head_dim) pool. Since the precheck is a NETWORK/transceiver availability check -- not a KV-correctness test -- it must still RUN, not skip. model_kv_shape now returns a simple MLA-flavored stand-in pool for these models (real layer count, one latent head, is_mla=True), so the transceiver is set up with SELFKONLY + AttentionType.MLA exactly like the real sparse model and the transfer exercises the same UCX/NIXL path -- only the per-token bytes are generic. The KV manager version (V1/V2) and transceiver runtime still resolve against the model + yaml (V4 -> V2 + Python), so the check stays consistent with what the model actually runs. A SIMPLIFIED note is printed and recorded. 2. The Python transceiver records bandwidth through perf_logger.py (KVSendTask throughput_mbs on the ctx side) rather than the C++ TRTLLM_KVCACHE_TIME_OUTPUT_PATH CSV, so V2xV2 verdicts showed no bandwidth. setup() now also enables TLLM_ENABLE_CACHE_TRANSFER_PERF_INFO + TLLM_KV_TRANSFER_PERF_LOG_FILE, and the result step medians throughput from the ctx-side perf CSVs on the Python path (gen-side recv CSVs on the C++ path, as before). CPU unit tests cover the simplified MLA stand-in and the Python CSV bandwidth parser. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
'chunk' collided with chunked prefill (splitting a request's tokens), which is an unrelated concept -- reviewers kept reading the precheck's concurrency batch as token chunking. Rename the concept throughout to 'wave' (a wave of transfer pairs in flight at once): waves()/wave_size/ wave_timeout_s (yaml knob), ctx_run_wave/gen_run_wave/ctx_finish_wave, and all comments/docstrings. The max_concurrent_pairs knob keeps its name (already unambiguous). The docstring now states the collision explicitly so the term doesn't drift back. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…findings Fail-fast: once any (ctx, gen) pair fails, stop testing the rest instead of running every pair against a fabric already known bad. A work-dir abort flag (job-id stamped, like the addr files, so a reused work dir does not carry a stale flag) is dropped on the first failure; the gen driver skips remaining peers and tells each to abort over the control channel, so the ctx side tears down promptly rather than waiting out its handshake alarm. Sessions that already passed still get their deferred "done". Skipped peers record a non-failing SKIP, which does not pollute the overall verdict. Review fixes: - run_precheck: broadcast the leader-only "missing context_phase_params" verdict so every rank raises together; previously only the leader raised while the others entered the next collective, deadlocking the step until the watchdog SIGKILLed it (misreported as TIMEOUT). - precheck_config: parse the TRTLLM_DISAGG_CT_PRECHECK kill switch as a boolean (1/0/true/false/on/off/yes/no) and reject ambiguous values, so a force-enable like =true is no longer silently read as "off". - precheck_config: add gate_library_content() as the single owner of loading and splicing the gate shell library; both submit.py call it (removing the drifted duplicate), and it falls back to the in-repo copy when the draft lives outside the repo instead of failing with FileNotFoundError. - gate: also clear csv/ on requeue so stale per-run bandwidth CSVs are not median'd across runs; print SKIPPED (not an empty "PASSED") when the run wrote no verdicts. Tests: cover fail-fast skip/abort, the abort-flag staleness rule, the missing-params broadcast, the kill-switch parsing, and the gate library fallback. Register the (pure-CPU) test file in l0_a10.yml. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
The failure echo used a '*' glob in place of the attempt suffix, so the printed Artifactory URL was not directly downloadable. The tracking script is regenerated per attempt with postTag in scope, and the teardown upload uses the same postTag, so interpolating it yields the exact uploaded filename. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…d collective read - precheck_config: _spec_nextn falls back to max_draft_len (the field num_nextn_predict_layers is its deprecated alias); several disagg yamls spell the MTP depth only as max_draft_len and resolved to 0 spec layers. - gate script: also remove a stale precheck.abort before launching -- a requeued job keeps its SLURM_JOB_ID, so the driver's job-id stamp cannot reject it and the rerun would be falsely fail-fast-skipped. - run_precheck: read the fail-fast flag collectively (leader reads, bcast) so ranks cannot race the flag write into divergent branches whose MPI collective sequences differ (gen_run_peer vs gen_abort_peer). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
…ache-transceiver precheck driver The precheck driver (run_precheck.py) had no test coverage and reached into unstable TRT-LLM internals from eight scattered call sites, so an upstream refactor would only surface as INIT_ERROR aborts on the SLURM disagg perf pipeline. - run_precheck.py: centralize every tensorrt_llm import behind a lazy load_internal_apis() (single failure point; --dry-run and pure-logic tests still work without torch installed). - test_cache_transceiver_precheck_run.py: pure-logic tests (HMAC wire format, rid/seed scheme, rendezvous/abort-flag files, StatusRecorder, bandwidth CSV parsing, schedule/timeouts) plus internal-API contract tests that pin the constructor/signature shapes the driver relies on, so internal-API drift fails in the refactorer's pre-merge CI. - test_cache_transceiver_precheck_e2e.py: drives run_precheck.py like the SLURM gate (one mpirun world per ctx/gen instance, shared work-dir, synthetic model config.json) on one GPU: symmetric 1x1, multi-instance 2x2, asymmetric tp2->tp1, and a fingerprint-mismatch fail-fast negative path asserting FAIL verdicts instead of hangs. - Wire the new tests into l0_a10 (unit/contract) and l0_h100 (e2e, next to the existing single-node transceiver harness test). Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
chuangz0
force-pushed
the
perf_sanity_cache_transceiver_test
branch
from
July 24, 2026 07:12
d0cdb1e to
96f22ad
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #61529 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61529 [ run ] completed with state
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Description
Disaggregated perf-sanity stages occasionally fail or hang deep inside model
bring-up for reasons that are purely network/transport-level (UCX transport
fallback, NIXL wire-up stalls, cross-rack link issues). By the time the real
test surfaces the failure, an hour-scale allocation has been burned and the
logs point at the model rather than the network.
This PR adds a fast go/no-go cache-transceiver precheck that runs inside
every disaggregated perf-sanity stage, before the real test:
placement (
-wnode slices), UCX environment, andcache_transceiver_configas the real test — parity is by construction:the precheck commands are generated by
submit.pyfrom the same env/configstrings as the worker steps.
transceiver and verifies the received bytes. Data is seeded per
(request, global layer) so asymmetric layouts (e.g. ctx dep4 → gen dep16,
ctx pp8 → gen tp32) verify correctly under any TP/PP resharding.
real model's
config.json; unmodelable models fall back to a simplifiedpool so the network path is still exercised.
runtimes, resolved through the same
llm_utilscode paths serving uses.(
TIMEOUT/TRANSFER_ERROR/MISMATCH/INIT_ERROR), a consolesummary (per-instance verdicts, first-error lines + log tails, UCX
red-flag lines), a junit result, and full artifacts
(status/logs/per-request bandwidth CSVs) under
<testOutputDir>/cache_transceiver_precheck/. The expensive modelbring-up never starts.
fetchRemoteMDper rank pair, measured at 100–170 s cold cross-rack);later reps run under a tight per-wave timeout, which is what actually
catches hangs.
Enablement: on by default for all disaggregated perf-sanity tests.
Opt out per test yaml (
cache_transceiver_precheck: {enabled: false}) orglobally via
TRTLLM_DISAGG_CT_PRECHECK=0(kill switch;=1force-enables). Timeouts and request sizes are also overridable per yaml.
Also included:
slurm_env_setup.sh: the UCX env fixup extracted fromslurm_run.shsothe precheck and the real workers source identical environment setup.
fails, so failures are diagnosable without digging through Jenkins.
See
tests/scripts/perf-sanity/cache_transceiver_precheck/README.mdfor theparity table, timeout model, failure-output layout, and manual-run recipes.
Test Coverage
tests/unittest/others/test_cache_transceiver_precheck_config.py(yaml → precheck-config resolution, enable/override semantics, KV shape
modeling incl. MLA/GQA/MTP, V1/V2 + transceiver-runtime resolution,
asymmetric-layout expected-data generation).
(
jenkins/L0_Test.groovyperf-sanity Slurm stages), validated manually onSLURM across symmetric and asymmetric topologies (dep4→dep8, ctx2→dep16,
dep4→tep8, pp8→dep16, pp8→tep8, v2↔v2), with both C++ and Python
transceiver runtimes.
--dry-runmode allows GPU-free inspection of what a test yaml resolvesto.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.