You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ascend PD prefiller crashes on first peer connect with AttributeError: 'AscendPDBackend' object has no attribute '_nixl_agent_lock'. AscendPDSenderMixin._ensure_peer_connection called super()._ensure_peer_connection(...), which resolved (via MRO) to upstream PDBackend._ensure_peer_connection. That path serializes with self._nixl_agent_lock, created only in PDBackend.__init__. AscendPDBackend.__init__ reimplements init for HCCL/NPU and never calls PDBackend.__init__, so the lock is never created. Note: super()._init_sender() only opens the proxy ZMQ PUSH socket and does not create _nixl_agent_lock.
See AttributeError: ... '_nixl_agent_lock'; proxy hangs waiting for
KV-ready ZMQ notif.
Expected behavior
Peer connect should use the HCCL transfer channel
(lazy_init_peer_connection + alloc REQ socket) without touching NIXL
agent state or _nixl_agent_lock, then complete PD KV transfer.
Screenshots
N/A (log-only failure).
Platform & Software (please complete the following information):
OS: Ubuntu 22.04.5 LTS (container, aarch64)
CANN Version: 9.0.0
Ascend HDK Version: [fill from local Ascend install]
Upstream commit that introduced _nixl_agent_lock: 7f60057ce37102bf7e7a519901930d6b9a874136
(feat(pd_backend): add bidirectional NIXL cache probe / #2972,
Ziwen Ning, 2026-04-26). First release tag containing it: v0.4.5
(also v0.4.5-cu129 / v0.4.6+). Absent from v0.4.4.
Why upstream needed it: #2972 added a dedicated NIXL worker thread +
queue so KV xfer does not block the vLLM worker (CUDA context /
RPC timeouts on vLLM v0.19 multiprocess). _nixl_agent_lock serializes
main-thread peer handshake vs worker-thread batched_write / batched_read on shared nixl_agent state.
Do we need it on Ascend? No _nixl_agent_lock. Ascend PD has no
NIXL worker thread/queue and does not share a NIXL agent. HCCL channel
already serializes peer/state mutations via transfer_channel._state_lock.
Copying the NIXL lock would paper over a wrong super() call without
matching the async NIXL design.
Fix: override _ensure_peer_connection with HCCL peer init; do not call super()._ensure_peer_connection.
logs:
prefiller: AttributeError: 'AscendPDBackend' object has no attribute '_nixl_agent_lock'
File ".../pd_backend.py", line ..., in _ensure_peer_connection
with self._nixl_agent_lock:
prefiller: wait_for_save failed ... skipping save
proxy: waiting for KV ready (hang)
Describe the bug
Ascend PD prefiller crashes on first peer connect with
AttributeError: 'AscendPDBackend' object has no attribute '_nixl_agent_lock'.AscendPDSenderMixin._ensure_peer_connectioncalledsuper()._ensure_peer_connection(...), which resolved (via MRO) to upstreamPDBackend._ensure_peer_connection. That path serializes withself._nixl_agent_lock, created only inPDBackend.__init__.AscendPDBackend.__init__reimplements init for HCCL/NPU and never callsPDBackend.__init__, so the lock is never created. Note:super()._init_sender()only opens the proxy ZMQ PUSH socket and doesnot create
_nixl_agent_lock.To Reproduce
Steps to reproduce the behavior:
kv_producer+ decoderkv_consumer,transfer_channel: hccl), e.g.tests/run_pd_disagg_deepseek_v4.sh.(e.g.
tests/query3.sh).batched_submit_put_task→_ensure_peer_connection.AttributeError: ... '_nixl_agent_lock'; proxy hangs waiting forKV-ready ZMQ notif.
Expected behavior
Peer connect should use the HCCL transfer channel
(
lazy_init_peer_connection+ alloc REQ socket) without touching NIXLagent state or
_nixl_agent_lock, then complete PD KV transfer.Screenshots
N/A (log-only failure).
Platform & Software (please complete the following information):
Additional context
_nixl_agent_lock:7f60057ce37102bf7e7a519901930d6b9a874136(
feat(pd_backend): add bidirectional NIXL cache probe/ #2972,Ziwen Ning, 2026-04-26). First release tag containing it: v0.4.5
(also v0.4.5-cu129 / v0.4.6+). Absent from v0.4.4.
queue so KV xfer does not block the vLLM worker (CUDA context /
RPC timeouts on vLLM v0.19 multiprocess).
_nixl_agent_lockserializesmain-thread peer handshake vs worker-thread
batched_write/batched_readon sharednixl_agentstate._nixl_agent_lock. Ascend PD has noNIXL worker thread/queue and does not share a NIXL agent. HCCL channel
already serializes peer/state mutations via
transfer_channel._state_lock.Copying the NIXL lock would paper over a wrong
super()call withoutmatching the async NIXL design.
AscendPDBackend(AscendPDSenderMixin, AscendPDReceiverMixin, PDBackend).Mixin
super()→PDBackendNIXL peer-connect.after LMCache ≥ v0.4.5; Ascend
__init__never constructs the lock._ensure_peer_connectionwith HCCL peer init; do not callsuper()._ensure_peer_connection.