[BUG] Crash due to race condition in HCCL peer handshake

[marcobarlo]

Describe the bug

HCCL peer handshake in HcclChannel races between the ZMQ init loop and the background accept() thread. HcclMemRegRequest can be handled before conn_handles_dict[peer_id] is published, so the decoder returns HcclErrorResponse and the prefiller fails with AttributeError: 'HcclErrorResponse' object has no attribute 'server_mem_handle_bytes'. PD KV transfer then fails and the disagg proxy hangs waiting for KV-ready.

To Reproduce

Steps to reproduce the behavior:

1. Launch Ascend PD disagg (prefiller kv_producer + decoder kv_consumer, transfer_channel: hccl).
2. Send a chat/completions request through the LMCache disagg proxy (e.g. tests/query3.sh).
3. Prefiller calls lazy_init_peer_connection → Init → connect() → MemReg.
4. See decoder Failed to process hccl initialization loop: '<peer_id>' and prefiller MemReg decode error.

Expected behavior

MemReg must run only after hccl_agent.accept() has stored the peer handle (same ordering as HcclReadyRequest / ready_event), then exchange buffer handles and complete PD transfer.

Platform & Software (please complete the following information):

• OS: Ubuntu 22.04.5 LTS (container, aarch64)
• CANN Version: 9.0.0
• Ascend HDK Version: [fill from local Ascend install]
• Hardware: Ascend 910B (TP8 prefiller + TP8 decoder)
• Framework: PyTorch / vLLM-Ascend
• vLLM Version: ~0.20.2 (Ascend build)
• LMCache / LMCache-Ascend: v0.4.5-aligned Ascend plugin
• Model: DeepSeek-V4-Flash (MLA)

Additional context

• Root cause: On HcclInitRequest, _handle_init_msg starts complete_handshake() (background accept()), waits only for accept_started_event, then replies Init. Client may finish connect() and send MemReg before T3 writes conn_handles_dict.
• Fix: In HcclMemRegRequest handling, ready_event.wait() before looking up conn_handles_dict.
• logs:

decoder: Processing HcclMemRegRequest
decoder: Failed to process hccl initialization loop: 'localhostXXXX'
decoder: Background: Connection established with localhostXXXX
prefiller: AttributeError: 'HcclErrorResponse' object has no attribute 'server_mem_handle_bytes'
prefiller: wait_for_save failed ... skipping save

• Threads: T1 lazy_init_peer_connection, T2 _init_loop → _handle_init_msg, T3 nested complete_handshake.