Skip to content

[Bug][Update] Bump HCOMM version to v9.0.0 to fix SEGFAULT and match CANN 9.0 in v0.4.5#267

Open
marcobarlo wants to merge 1 commit into
LMCache:mainfrom
marcobarlo:fix_262_issue
Open

[Bug][Update] Bump HCOMM version to v9.0.0 to fix SEGFAULT and match CANN 9.0 in v0.4.5#267
marcobarlo wants to merge 1 commit into
LMCache:mainfrom
marcobarlo:fix_262_issue

Conversation

@marcobarlo

@marcobarlo marcobarlo commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes #262

Bump third_party/hcomm from v8.5.0 (d3cae0c) to v9.0.0 (09a3047) so HCCL public headers match the CANN 9.0 toolkit and the shipped libhccl_plf on Ascend 910B. CANN v9.0 is the default shippend CANN for vLLM >= v0.20.

This PR is a submodule pointer only. It does not add CMake ABI / LTO overrides; the existing per-TU _GLIBCXX_USE_CXX11_ABI=0 guards in csrc/hccl/{hccl_utils,hccl_agent,bindings}.cpp already force the old libstdc++ ABI required by libhccl_plf.

Problem

hccl_npu_comms compiles against HCOMM headers (HcclIpAddress, hrtRaGetDeviceIP, …) and links the system libhccl_plf. On hosts upgraded to CANN 9.0, that shared library matches HCOMM v9 layout (including HcclIpAddress / Eid-related fields).

Keeping the submodule on v8.5 while linking CANN 9.0 libhccl_plf creates a silent struct / ABI mismatch on types that embed std::string and are passed in std::vector<hccl::HcclIpAddress>. At runtime this surfaces as a segfault inside GetLocalIpv4() when calling hrtRaGetDeviceIP during HCCL agent setup (PD buffer register / peer init). Prefiller and decoder then never establish a healthy PD path.

Fix

Point third_party/hcomm at tag v9.0.0 so include paths used by csrc/hccl agree with the toolkit binary. No other source changes are required for this crash.

After the bump, rebuild hccl_npu_comms so objects are compiled against the new headers before linking libhccl_plf.

Test plan

  • git submodule update --init third_party/hcommgit describe --tags reports v9.0.0
  • Clean rebuild: rm -rf build && pip install --no-build-isolation -e .
  • Launch PD disagg (PD_INSTANCE=both: prefiller + decoder serve /v1/models
  • Logs show Registered memory with HCCL on prefiller and decoder TP0
  • No Segmentation fault / SIGSEGV around HCCL agent init in prefiller or decoder logs

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the third_party/hcomm subproject commit from d3cae0c5402398bb86e3a591e7d19706c0a1e552 to 09a304742994c603879a57291579afd52c3126d9. There are no review comments, and I have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@marcobarlo marcobarlo changed the title [Bug][Update] Bump HCOMM version to v9.0.0 to match CANN 9.0 in v0.4.5 [Bug][Update] Bump HCOMM version to v9.0.0 to fix SEGFAULT and match CANN 9.0 in v0.4.5 Jul 13, 2026

@matthewygf matthewygf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm ! Let's merge once we create the release @chloroethylene

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Segfault in CANN 9.0.0 when using Ascend p2p

2 participants