fix(pcie): preserve default all-reduce output lifetime

[yatesdr]

Summary

Preserve normal out-of-place lifetime semantics for
PCIeDmaAllReduce.all_reduce() when out is omitted.

The persistent default-output workspace added by #76 returns the same storage
for every call. A transformer can retain one all-reduce result across the next
collective: the embedding result becomes the layer-0 residual, then the
attention output projection performs another all-reduce. Reusing one output
buffer silently overwrites that live residual.

This PR:

• restores torch.empty_like(inp) for the default-output form;
• preserves the explicit out= path;
• changes the GPU test to retain multiple results, require disjoint live
  storage, and revalidate earlier contents after later collectives;
• keeps CUDA graph replay on explicit caller-owned output storage;
• removes unit tests for the unsafe persistent-output helper.

This is intentionally based on
fix/pcie-dma-persistent-output-20260723, the head branch of #76, because
SparkInfer master does not currently contain the affected implementation.

Integration bug report and full model-level evidence:

local-inference-lab/vllm#181

Reproduction and evidence

Standalone four-rank production-geometry proof:

https://gist.github.com/yatesdr/a2e84aa3171ee0b355649704f04f96a8

Geometry: 2735 x 6144 BF16, 33,607,680 bytes/result, four ranks,
i8_ring.

Before:

pointer_alias=true
retained_mismatches=16,803,840 / 16,803,840
rank_consistent=true
status=FAIL

After:

pointer_alias=false
retained_mismatches=0 / 16,803,840
rank_consistent=true
status=PASS

The collective output hashes are unchanged.

In an end-to-end TP4/DCP4/MTP3 GLM-5.2 trace, the affected layer-0 residual
was bit-identical to the attention output and differed from the layer input in
6,141/6,144 BF16 values. With this fix it is bit-identical to the layer input
on all four ranks.

Validation

• python3 -m py_compile on both modified Python files
• git diff --check
• four-rank standalone retained-output proof: PASS
• matched v20 model boot and CUDA-graph capture: PASS
• complete 1,248-record layer trace: repaired residual invariant on 4/4 ranks

Scope

This fixes the confirmed storage-lifetime corruption. It is not presented as
the complete fix for the separately tracked GLM-5.2 long-context retrieval
regression: the frozen 350k request still missed after this corruption was
removed.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• dev/*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b8b6537e-0d91-4404-9ce0-debee4e78296

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}