Skip to content

[BUG] Crash in PD backend init due to NIXL change in LMCache v0.4.5 #264

Description

@marcobarlo

Describe the bug

Ascend PD prefiller crashes on first peer connect with
AttributeError: 'AscendPDBackend' object has no attribute '_nixl_agent_lock'.
AscendPDSenderMixin._ensure_peer_connection called
super()._ensure_peer_connection(...), which resolved (via MRO) to upstream
PDBackend._ensure_peer_connection. That path serializes with
self._nixl_agent_lock, created only in PDBackend.__init__.
AscendPDBackend.__init__ reimplements init for HCCL/NPU and never calls
PDBackend.__init__, so the lock is never created. Note:
super()._init_sender() only opens the proxy ZMQ PUSH socket and does
not create _nixl_agent_lock.

To Reproduce

Steps to reproduce the behavior:

  1. Launch Ascend PD disagg (prefiller kv_producer + decoder kv_consumer,
    transfer_channel: hccl), e.g. tests/run_pd_disagg_deepseek_v4.sh.
  2. Send a chat/completions request through the LMCache disagg proxy
    (e.g. tests/query3.sh).
  3. Prefiller hits batched_submit_put_task_ensure_peer_connection.
  4. See AttributeError: ... '_nixl_agent_lock'; proxy hangs waiting for
    KV-ready ZMQ notif.

Expected behavior

Peer connect should use the HCCL transfer channel
(lazy_init_peer_connection + alloc REQ socket) without touching NIXL
agent state or _nixl_agent_lock, then complete PD KV transfer.

Screenshots

N/A (log-only failure).

Platform & Software (please complete the following information):

  • OS: Ubuntu 22.04.5 LTS (container, aarch64)
  • CANN Version: 9.0.0
  • Ascend HDK Version: [fill from local Ascend install]
  • Hardware: Ascend 910B (TP8 prefiller + TP8 decoder)
  • Framework: PyTorch / vLLM-Ascend
  • vLLM Version: ~0.20.2 (Ascend build)
  • LMCache / LMCache-Ascend: v0.4.5-aligned Ascend plugin
  • Model: DeepSeek-V4-Flash (MLA)

Additional context

  • Upstream commit that introduced _nixl_agent_lock:
    7f60057ce37102bf7e7a519901930d6b9a874136
    (feat(pd_backend): add bidirectional NIXL cache probe / #2972,
    Ziwen Ning, 2026-04-26). First release tag containing it: v0.4.5
    (also v0.4.5-cu129 / v0.4.6+). Absent from v0.4.4.
  • Why upstream needed it: #2972 added a dedicated NIXL worker thread +
    queue so KV xfer does not block the vLLM worker (CUDA context /
    RPC timeouts on vLLM v0.19 multiprocess). _nixl_agent_lock serializes
    main-thread peer handshake vs worker-thread batched_write /
    batched_read on shared nixl_agent state.
  • Do we need it on Ascend? No _nixl_agent_lock. Ascend PD has no
    NIXL worker thread/queue and does not share a NIXL agent. HCCL channel
    already serializes peer/state mutations via transfer_channel._state_lock.
    Copying the NIXL lock would paper over a wrong super() call without
    matching the async NIXL design.
  • MRO: AscendPDBackend(AscendPDSenderMixin, AscendPDReceiverMixin, PDBackend).
    Mixin super()PDBackend NIXL peer-connect.
  • Root cause: Ascend stub (from Enable Direct rH2D / D2D KVtransfer in P2P and PD backend  #173, before #2972) delegated to upstream
    after LMCache ≥ v0.4.5; Ascend __init__ never constructs the lock.
  • Fix: override _ensure_peer_connection with HCCL peer init; do not call
    super()._ensure_peer_connection.
  • logs:
prefiller: AttributeError: 'AscendPDBackend' object has no attribute '_nixl_agent_lock'
  File ".../pd_backend.py", line ..., in _ensure_peer_connection
    with self._nixl_agent_lock:
prefiller: wait_for_save failed ... skipping save
proxy: waiting for KV ready (hang)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions