[ci] add PPU unittest workflow

[tiankongdeguiji]

Summary

Adds a PPU unit-test lane (.github/workflows/unittest_ppu_ci.yml) for release/v1.1, targeting the self-hosted tzrec-ppu-runner with image mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/tzrec-test:1.1-ppu. Shaped after master's unittest_h20_ci.yml (inline docker run instead of container: to dodge the DSW authZ docker.sock injection), with PPU-specific adjustments: alixpu /dev device passthrough instead of --gpus all, --shm-size=256g, no CUDA_HOME env, 24h job timeout, and a dedicated scripts/ci/ci_test_ppu.sh that skips pip install (the image ships deps pre-installed).

Trigger policy: auto-fires on PRs targeting release/** (where stability matters) plus workflow_dispatch. PRs to master do not auto-trigger — PPU runner capacity stays opt-in for development branches.

Making the lane green

Initial run (26218011590) had 8 failures + 5 errors; follow-up runs surfaced 28 subprocess timeouts and an AOTI-load failure. Fixed by root cause:

• CUDA-only wheels missing on PPU image — hstu_attn_2_cuda (cutlass HSTU), torch_fx_tool (RTP export), dynamicemb. Gate the affected tests on @unittest.skipUnless(<dep>_available, ...) so they skip cleanly on any host missing the wheel (not a PPU-named carve-out).
• PPU Triton atomic_cas/atomic_xchg memory-ordering bug in _weighted_rms_norm_bwd_dx produced wrong dw for bf16 rms_norm backward (deterministic ~0.196 abs-diff at D=257 in CI; flaky ~0.5 abs-diff at N=3000 with contention via direct kernel probe). Route rms_norm's Kernel.TRITON to PyTorch eager on PPU at the dispatcher in tzrec/ops/layer_norm.py; the Triton kernel itself is left untouched.
• alixpu Triton fp32 matmul ULP drift vs PyTorch eager (~5e-6 rel, ~1e-4 abs) — wider than CUDA but mathematically correct. Add get_compare_tolerance(dtype) that returns (atol=3e-5, rtol=2e-5) for fp32 on PPU (~2.1x / ~4.4x headroom on observed worst) and PyTorch defaults elsewhere, so NVIDIA's tight regression guard stays intact.
• Latent NameError in dynamicemb_util.build_dynamicemb_constraints — referenced DynamicEmbScoreStrategy outside the try: import dynamicemb guard. Made it fail loudly with a clear RuntimeError when has_dynamicemb is False.
• NVIDIA-tuned subprocess timeouts too tight on PPU — train_eval at 600s and export at 1800s were over-running on PPU. Wrap every hardcoded timeout in tzrec/tests/utils.py with a _t(...) helper that scales by $TZREC_TEST_TIMEOUT_SCALE (default 1.0 — NVIDIA lanes unchanged); set the scale to 4 in the PPU workflow's docker env (export budget becomes 7200s; observed unified_aot needs ~5300s in-container).
• torch._inductor.codecache not auto-imported on PPU torch 2.10.0+ppu — import torch._inductor does NOT populate torch._inductor.codecache, and PyTorch's own _load_aoti (pt2_archive/_package.py:1019) accesses it as if pre-imported, raising AttributeError in the predict subprocess of every AOT integration test. Fix is one explicit import torch._inductor.codecache at the top of tzrec/acc/aot_utils.py.

New tzrec.ops.is_ppu_arch() cached helper detects PPU by device-name substring.

Test plan

• Re-ran each originally failing test on PPU devices 2,3 (devices 0,1 are CI-reserved). Verified on the actual CI container image — all pass or skip cleanly.
• PPU CI lane goes green on this PR.
• After merge, verify the workflow auto-fires for the next PR targeting release/v1.1 and does NOT auto-fire for master PRs.

🤖 Generated with Claude Code

[github-actions]

The (None, None) return path makes the contract awkward — every caller has to special-case it, and the docstring doesn't mention it. Two concrete issues this invites:

1. Partial-override blind spot at call sites. Both callsites added in this PR use if atol is None and rtol is None: (see jagged_tensors_test.py:389, mm_test.py:79). A future caller that passes atol=1e-6, rtol=None silently skips the widening and rtol falls back to torch's dtype defaults — which on PPU could mask or surface false regressions depending on direction. Widening each independently would be safer:

   ppu_atol, ppu_rtol = get_compare_tolerance(dtype)
   atol = atol if atol is not None else ppu_atol
   rtol = rtol if rtol is not None else ppu_rtol

2. Returning (None, None) to mean "use defaults" is unusual. Consider returning concrete defaults for non-PPU/non-fp32 (e.g., the dtype's standard torch.testing.assert_close defaults), or restructure as get_compare_tolerance(dtype, atol, rtol) so the helper owns the precedence rule and callsites stay one line.

At minimum, the docstring should call out the (None, None) return path.

[github-actions]

Two notes on the implementation vs. the docstring:

1. Scope is broader than the docstring suggests. The function returns True if any visible CUDA device's name contains the substring "PPU", and the result is cached for the process lifetime. The docstring reads as if it tests "a CUDA device" — worth tightening to e.g. "Return True if any visible CUDA device name contains 'PPU'. Cached after first call." This also surfaces the mixed-fleet edge case (PPU + non-PPU on one host both report True).
2. except Exception is wider than needed. The only realistic failure modes are RuntimeError from a broken driver and AssertionError from a stale device index. The broad clause will silently swallow future typos / API changes and quietly disable PPU code paths. Narrow to the specific exceptions, or at least leave a one-line comment naming the failure mode being defended against — otherwise the next reader has to guess.

[github-actions]

github.event.pull_request.number is empty for workflow_dispatch, so all manual invocations collapse into the group key unittest-ppu-ci- and cancel-in-progress: true will cancel each other. Suggest:

Suggested change

	concurrency:
	group: unittest-ppu-ci-${{ github.event.pull_request.number }}
	cancel-in-progress: true
	concurrency:
	group: unittest-ppu-ci-${{ github.event.pull_request.number || github.run_id }}
	cancel-in-progress: true

[github-actions]

Minor drift from unittest_h20_ci.yml: the H20 lane pins ref: ${{ github.event.pull_request.head.sha }}, this one doesn't. With pull_request, the default checkout is the merge ref — slightly different semantics (catches conflicts, but won't pin if the PR is rebased mid-run). If the intent was parity with H20, add the ref: line; if intentional, worth a one-line comment noting why.

[github-actions]

Review summary

Focused, well-scoped PR: a new release-branch-only PPU CI lane plus minimal source changes (capability gating + tolerance widening) to make the existing test suite green on alixpu. The PR description's failure-mode → root-cause walk is unusually clear. Posture comments below; specific issues are inline.

Test gating approach is right. Using ImportError-based flags (cutlass_hstu_unavailable, torch_fx_tool_unavailable) rather than a is_ppu_arch() carve-out means tests skip cleanly on any host missing the wheel — not just PPU. Tolerance widening, on the other hand, is correctly arch-named because it's a numerics issue, not a wheel-availability one. The asymmetry is worth one comment in test_util.py so a future contributor doesn't "fix" it for consistency.

num_warps=32 exclusion on PPU. Correct call — Triton autotune still has to compile each candidate; "let Triton find it's bad" requires compilation to succeed. Excluding the known-broken config is the right behavior, and it leaves the NVIDIA search space unchanged.

Security posture (informational, not a blocker). The lane runs on a self-hosted runner on pull_request, which means code from forked PRs targeting release/** will execute on tzrec-ppu-runner with raw /dev/alixpu* device passthrough and --ulimit memlock=-1. The H20 lane on master has the same posture, so this isn't a regression — but the PPU device-node attack surface (raw ioctls into the driver) is genuinely broader than --gpus all. Worth tracking as a repo-wide policy item (fork allowlist / label-gated trigger / required-reviewers environment) rather than blocking this PR.

Inline comments cover:

• test_util.py:67-75 — get_compare_tolerance (None, None) contract + partial-override blind spot
• tzrec/ops/__init__.py:30-41 — is_ppu_arch() docstring scope + over-broad except Exception
• unittest_ppu_ci.yml:9-11 — concurrency group collapses workflow_dispatch runs
• unittest_ppu_ci.yml:19-22 — drift from H20 lane's ref: head.sha pin