[WIP][EPD] Nixl connector (Compatible with epd_mooncake_engine branch)

[herotai214]

Purpose

Test Plan

Test Result

---

[20260120 update]
Mean to be compatible with epd_mooncake_engine branch; based on commit 829b5752d5133c0ae0a39fa48c27597b5a0235e6

[WIP] -> nixl connector still have room to improve, by directly pulling cache from PD node using mm_hash;
In current implementation, PD node first use mm_hash to get the base_address (data pointer) of the cache in E node, through 1 nixl transfer. Then use the base_address to get the cache, through another nixl transfer. This can be simplify into 1 single action.

Can run both mooncake connector and nixl connector;

Verified correctness:
================================================================================
Metrics:
{
"explicit_prompt_relaxed_correctness": 0.812,
"anywhere_in_answer_relaxed_correctness": 0.814
}
================================================================================

Performance of nixl connector:
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Benchmark duration (s): 17.91
Total input tokens: 40000
Total generated tokens: 10000
Request throughput (req/s): 5.58
Output token throughput (tok/s): 558.44
Peak output token throughput (tok/s): 2716.00
Peak concurrent requests: 100.00
Total token throughput (tok/s): 2792.20
---------------Time to First Token----------------
Mean TTFT (ms): 11235.31
Median TTFT (ms): 11943.74
P99 TTFT (ms): 15502.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 39.72
Median TPOT (ms): 40.34
P99 TPOT (ms): 57.68
---------------Inter-token Latency----------------
Mean ITL (ms): 44.71
Median ITL (ms): 23.23
P99 ITL (ms): 262.29
==================================================

VS mooncake:
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Benchmark duration (s): 15.89
Total input tokens: 40000
Total generated tokens: 10000
Request throughput (req/s): 6.29
Output token throughput (tok/s): 629.44
Peak output token throughput (tok/s): 1921.00
Peak concurrent requests: 100.00
Total token throughput (tok/s): 3147.20
---------------Time to First Token----------------
Mean TTFT (ms): 8882.88
Median TTFT (ms): 8882.53
P99 TTFT (ms): 13471.40
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 41.31
Median TPOT (ms): 42.48
P99 TPOT (ms): 56.54
---------------Inter-token Latency----------------
Mean ITL (ms): 66.08
Median ITL (ms): 22.80
P99 ITL (ms): 476.82
==================================================

VS disk (example connector):
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Benchmark duration (s): 18.40
Total input tokens: 40000
Total generated tokens: 10000
Request throughput (req/s): 5.44
Output token throughput (tok/s): 543.60
Peak output token throughput (tok/s): 2282.00
Peak concurrent requests: 100.00
Total token throughput (tok/s): 2717.98
---------------Time to First Token----------------
Mean TTFT (ms): 11461.25
Median TTFT (ms): 12944.49
P99 TTFT (ms): 15861.16
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.82
Median TPOT (ms): 33.09
P99 TPOT (ms): 47.05
---------------Inter-token Latency----------------
Mean ITL (ms): 39.66
Median ITL (ms): 24.20
P99 ITL (ms): 270.28
==================================================

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[roytman]

Hello @fake0fan, @herotai214,
do you have an estimated timeline for when the Nixl EC_Connector will be available within the main vLLM branch?