W2 finish (KERNEL-SSM-MAMBA): 9/9 CUDA mutations caught, compute-sanitizer clean, Debug arm green, spec 8.3 re-authored (#496) - #566
Closed
localai-bot wants to merge 12 commits into
Closed
Conversation
…ps (#496) W2 of .agents/specs/mamba2-ssd.md. These are the failing tests, committed before the kernels they gate, per the implementer contract. The three suites gain a `#ifdef VLLM_CPP_CUDA` section that runs the SAME inputs through the device arm. They fail for the intended reason: no NATIVE kernel is registered for kMamba2ChunkScan / kMamba2StateUpdate / kRmsNormGatedGroup on DeviceType::kCUDA. That reason is NOT "GetOp throws", and the difference is the whole point of one assertion in these suites. GB10 is `integrated && pageable_memory_access` (cuda_backend.cu Registrar), so `Backend::UnifiedMemory()` is TRUE and `ReferenceTierEligible(kCUDA)` with it. On a GetOp miss the provider seam does not throw: it installs the CPU HOST kernel as a `kReferenceProviderName` provider and runs THAT over the device pointers (op_provider.h, "portable reference tier"). Every numeric assertion in a device arm would then pass while nothing ran on the GPU -- the device arm gated by running the host arm twice. So every CUDA case calls `RequireNativeCudaProvider`, which reads `GetOpProviderStats(op, kCUDA).last_selected` and refuses `vt-cpu-ref`. These are EAGER dispatches, not a captured graph, so the counter is genuinely populated ([[graph-replay-does-no-host-dispatch-counters-read-zero]]). The declared equivalence contract is written down here BEFORE the kernel, in the head comment of the SSD suite's CUDA section: * the CUDA arm keeps f32 accumulation throughout and does NOT mirror the tile downcasts in upstream's Triton dots (ssd_chunk_state.py:283-285, ssd_chunk_scan.py:266-269, :359-363) -- those are the input-precision requirement of `tl.dot`, i.e. of a tensor-core MMA, and every one of those tiles is loaded `.to(tl.float32)` and computed in f32 right up to the MMA. The memory format is unchanged, so this is not a "too wide" dtype; * G1, the primary gate, is the device output against the SAME independent double-precision sequential reference at the SAME upstream-ported tolerances the host arm is held to; * G2, device-vs-host, is a DERIVED bound: `rtol(K) = 4*(K + 2)*2^-24` over a recurrence of length K. CUDA's `expf` is documented to <= 2 ulp and glibc's to <= 0.5, so a product of K decay factors carries <= 2.5*K*u of libm disagreement, and the length-K f32 summation adds the standard (K-1)*u -- 3.5*K*u, rounded up to integers. Everything else is held identical by construction: each device output element is accumulated in ONE thread over the host arm's index range in the host arm's direction, so summation order is not a second source. A BYTE COMPARE IS NOT REACHABLE, and the libm difference is exactly why. The slack actually used is REPORTED on every comparison, so a bar that stopped doing work would be visible rather than silently absorbing a defect. Also lands the mutation-proof §8.2 records as owed: the decode kernel's `CheckMamba2ANegative` at cpu_ops.cpp:1877 was pinned by NO test -- deleting it left test_ops_mamba2_state_update fully green while the same deletion on its chunk-scan twin reds. The new "A must be negative" SUBCASE mirrors test_ops_mamba2_ssd.cpp:900 and additionally pins that the guard is a SIGN test, not an accidental magnitude floor (A = -1e-30 is accepted). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…he gate host after a worktree loss (#496) FOLLOWING_AGENTS_PROTOCOL NOT AUTHORED BY THE COMMITTER. This is an operator recovery of a fresh implementer's work after the fourth external deletion of an isolation worktree this session. The implementer had built and run this green on both boxes and staged it; the worktree was removed before the commit. The bytes survived on the gate host and are restored here unchanged: md5 cbb1f928f4 for cuda_mamba2_ssd.cuh and 37a0404433 for cuda_gdn.cu, matching what the implementer reported before the loss. The declared equivalence contract, which the implementer decided BEFORE writing the kernel and recorded in the kernel header and all three test headers: The CUDA arm keeps f32 accumulation throughout and deliberately does NOT mirror upstream's tile downcasts. Those casts -- b.to(x_ptr.dtype.element_ty) at ssd_chunk_state.py:283-285, cb.to(...)/prev_states.to(...) at ssd_chunk_scan.py:266-269,359-363 -- are the input-precision requirement of tl.dot, a tensor-core MMA. Every one of those tiles is loaded .to(tl.float32) and computed in f32 right up to the MMA. These are scalar-FMA kernels with no MMA, so mirroring the downcast would copy a constraint we do not have. The inter-chunk `passed` buffer is allocated at state_dtype, NOT the host arm's f32 working width that spec 8.2 F9 warned W2 must not inherit. A byte compare against the host arm is NOT reachable, and the downcasts are not why: the two arms call different libms (CUDA expf <= 2 ulp, glibc <= 0.5). Everything else is identical by construction. So the primary gate is the device output against the same double-precision sequential reference at the same upstream-ported tolerances the host arm uses, on the same inputs -- which separates "device defect" from "wrong threshold". The derived device-vs-host bar is rtol(K) = 4*(K+2)*2^-24, derived from 2.5 ulp of libm disagreement per decay factor through a product of at most K plus (K-1)*u summation error. No number was tuned and no tolerance was widened; each comparison logs the fraction of budget actually used through MESSAGE rather than INFO, because doctest prints INFO only on failure and an unaudited bar would have been a false claim. Evidence already captured on the gate host: Release build for 121a with CUTLASS 4.5.0, fa2 ENABLED and Marlin NVFP4 enabled, 0 warnings; RED run SIGSEGV on all three binaries; GREEN run ssd 11 cases / 2069 assertions, state_update 10 / 5965, gated_norm 12 / 3723, all Status SUCCESS, exit 0, with zero reference-tier lines. That RED SIGSEGV is a real shared-seam defect, filed as #547 and deliberately not fixed in flow: GB10 reports Backend::UnifiedMemory() == true, so ReferenceTierEligible(kCUDA) is true, and with no native kernel GetOp installs the CPU host kernel as a vt-cpu-ref provider and runs it over cudaMalloc pointers. include/vt/backend.h already says a cudaMalloc pointer is not host-dereferenceable on GB10; op_provider.cpp:515-526 gates on UnifiedMemory() where it needs DeviceMemoryIsHostAddressable(). Every CUDA case here now calls RequireNativeCudaProvider, so a device arm can never be gated by running the host arm twice. STILL OWED, and this branch is NOT landable until a fresh implementer finishes it: the 8 scripted CUDA mutations, compute-sanitizer, the Debug arm, a full ctest on the gate host, the spec section 8.3 that records the contract above (its only copy was a staged blob and is presumed lost), and an origin/main re-merge. A fresh review follows that, not this commit. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…e $8.2 decode SUBCASE (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…aught, sanitizer clean, Debug arm green (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…tributed failure (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… -- a second job locks /tmp/gpu.lock (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…e attribution re-run is REMOTE_UNVERIFIED (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… reproduced standalone on an idle box (#496) FOLLOWING_AGENTS_PROTOCOL The last unattributed ctest failure is closed, and the contention hypothesis this spec had recorded as "plausible, not proven" is REFUTED rather than quietly kept: test_minimax_h3 reproduces STANDALONE, serially, under the lock, on a box that had just rebooted and had no CUDA process resident. It is #486 (open) with #516 as its root cause -- which is what row/pool-device-key is repairing, and therefore why that baseline branch passed a test main-based branches fail. The difference was the baseline carrying a FIX, not this branch carrying a defect. The re-run reproduces #486's recorded signature number for number: 38 | 36 passed | 2 failed | 41 skipped, 42724 assertions, the same two case names, the same `cudaFree: invalid argument`, the same SIGSEGV. All ten ctest failures are now pre-existing and tracked. None is W2's. Also fixed a live protocol defect found while doing it: the re-run had been relaunched as `flock -w 3600 $HOME/gpu.lock ./w2refail.sh`, an OUTER flock wrapping a script that takes the SAME lock on its own fd. flock locks an open file DESCRIPTION, so the inner acquisition blocked against its own parent -- a self-deadlock HOLDING the shared GPU mutex with three other agents' jobs queued behind it. Killed the stack, the lock passed straight to a waiter, and relaunched with a single acquisition. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…- deterministic, not starved (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…vidence; 32G build tree reclaimed (#496) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 13, 2026
… against the corrected bound (#496) (#675) FOLLOWING_AGENTS_PROTOCOL W2 of KERNEL-SSM-MAMBA (#496): the CUDA arm of vt::Mamba2ChunkScan, vt::Mamba2StateUpdate and vt::RmsNormGatedGroup. Supersedes #566 and #592, which had diverged -- one carried the evidence resolving the merge precondition, the other the F1/F2 repair, and neither contained the other. F1: the declared equivalence contract named the elementary functions as the ONLY admitted source of device-vs-host divergence. Host C++ is pinned -ffp-contract=off and nothing passes --fmad=false to nvcc, so acc += xv * bv contracts on device and not on host. Stated model 3.5*K*u, contraction adds ~1.0*K*u, bound was 4*(K+2)*u -- provable only for K <= 18 while the driver shapes run at K = 200. Repaired by carrying the term to 5*(K+2)*u with the arithmetic shown in the header, all three test comments and 8.3. -fmad=false was rejected deliberately: it is a per-TU flag on a header included by the hot GDN TU. Repo-wide gap filed as #591. F2: the header claimed the device kernels stay memory-safe under a contract violation, but the validator checks metadata shape and dtype only. Two dropped checks were memory-unsafe -- an out-of-bounds WRITE past a cudaMallocAsync allocation and an out-of-bounds READ of initial_states -- and the stated reason for omitting them (a D2H plus a stream sync) did not apply, since both values are already in registers and the decode kernel does exactly that clamp for free. Repaired with both the clamps and a narrowed claim enumerating all six dropped checks. A third hole the review did not name was found and closed: seq_idx[0] < 0 indexes passed at chunk -1. All four owed gates discharged by measurement, operator-run on the gate host, each stamping its own lock-acquire time, load and disk. nvcc compile for sm_121a clean with CUTLASS, FA2 and Marlin confirmed ENABLED in the configure log rather than assumed. The three CUDA arms 12/2095, 10/5965, 12/3723 all SUCCESS. compute-sanitizer memcheck ERROR SUMMARY 0 errors. And the 9-mutation re-sweep against the MOVED bound: 9 of 9 CAUGHT. The re-sweep is the item a reader would most likely have waved through, and M6 is why it could not be: it aborts at exit 134 while printing "assertions: 2577 | 2577 passed | 0 failed" -- a clean assertions line on a FAILING run -- and is caught only because the harness reads the exit code. Two of the original eight mutations did not COMPILE under -Werror=all-warnings and were being scored as caught; a mutation that will not build is a suite that never ran. test_minimax_h3 is attributed rather than waived. Reproduced standalone on an idle box under the lock at TEST_EXIT=139, it is #486 with root cause #516, signature-for-signature. The independent baseline that PASSED was row/pool-device-key, the branch that FIXES #516, so the baseline carried a fix and this branch does not carry a defect. CI is REMOTE_UNVERIFIED, not green: every run on the predecessor branches ended cancelled, including a repo-wide mass cancellation of 20 runs across 7 branches, and cuda-fat-build never completed -- which is why the compile was run directly on the gate host. Windows reds are the main baseline (#514, #584). The row stays INVENTORIED. No lifecycle move, no measurement claimed, no performance result: a host reference plus its device arm is not a speed number. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
|
Superseded by #675, which consolidates this branch with the F1/F2 repair branch and lands the row. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes W2 of
.agents/specs/mamba2-ssd.md— the CUDA arm of the three Mamba2 SSD ops. Issue: #496.The kernels themselves arrived on
row/KERNEL-SSM-MAMBA-SSD-W2atfcdb7d824(an operator recovery of a lost worktree, byte-exact). This branch adds only what that commit listed as still owed, plus theorigin/mainre-merge. Since the merge,git diffis.agents/specs/mamba2-ssd.mdalone.What is here
origin/mainre-merged (4 commits, clean; includes thetests/parity/hf_snapshot.hfix that had RED'd main).SUBCASEmarked CLOSED, re-proved here rather than taken on report (mutation M9 below).Evidence (gate host
promaxgb10-4ad8, GB10 / sm_121a)Configure log READ, not assumed:
cutlass-nvfp4: ENABLED,cutlass-fp8: ENABLED,marlin-nvfp4: ENABLED,fa2: ENABLED for [121a],CUTLASS found at ~/cutlass-4.5.0. 0 warnings, both arms.Release, post-merge — identical to the pre-merge counts, so the merge moved nothing:
test_ops_mamba2_ssdSUCCESS!test_ops_mamba2_state_updateSUCCESS!test_ops_mamba2_gated_normSUCCESS!Debug arm (
NDEBUGOFF, so everyassertis live; CUDA-g, deliberately not-G): same counts, allSUCCESS!, exit 0.compute-sanitizer: 8 runs, every oneERROR SUMMARY: 0 errors/EXIT=0—memcheck×4,initcheck×3,synccheckon the gated norm (the one kernel with a block reduction and__syncthreads).Mutation sweep: 9 of 9 CAUGHT. Each mutant runs under a doctest
-tcfilter, and the pristine binary runs under the identical filter first — a filter that selects no test case makes doctest printSUCCESS!, so an unverified filter would score a false catch. Sources restored byte-for-byte after each, md5 re-asserted (cuda_mamba2_ssd.cuhcbb1f928f4…,cpu_ops.cpp9ed9eb98…).SUCCESS!FAILURE!initial_statesin state passingSUCCESS!FAILURE!states[c]forstates[c-1]SUCCESS!FAILURE!Dskip connectionSUCCESS!FAILURE!state_indicesSUCCESS!FAILURE!SUCCESS!FAILURE!SUCCESS!FAILURE!SUCCESS!FAILURE!CheckMamba2ANegativeon decode (§8.2)SUCCESS!FAILURE!Two mutations had to be reformulated, which is itself a finding. The obvious M1 (
if (!prev_zero)→if (false)) and M7 (passing1, hidden) do not COMPILE: the arm is built-Werror=all-warningsand nvcc raises#550-D "prev_zero was set but never used"/#177-D "group_size was declared but never referenced"once the mutation dead-codes a read. A mutation that will not build is not a caught mutation and must not be scored as one. Both were rewritten to drop the same term with every variable still read (M1 multiplies the inter-chunk product by0.0f; M7 passes1, group_size * args.n_groups, which ishidden).The derived bar is audited, not asserted. Across the 55 device-vs-host comparisons in a green run the worst used 7.66% of
rtol(K) = 4·(K+2)·2⁻²⁴; the driver shapes used 0.32% and 0.18%. Under mutant M3 the sameMESSAGEline readsused 962173% of its derived budget.Full ctest — and one failure I have NOT attributed
All 392 test targets built (0 warnings),
ctest -j 1:98% tests passed, 10 tests failed out of 431,CTEST_EXIT=8, 53 min. None of the three mamba2 suites is among them.Nine of the ten match by name an independent same-box baseline — another agent's full ctest on
row/pool-device-key, finishing 80 min earlier:98% tests passed, 9 tests failed out of 437, the same nine (test_serve_low_tools,test_linear_method,test_glm4_moe_lite_paged_engine,test_capiSEGFAULT,test_ops_gdn,test_qwen3_apc_e2e,test_minicpm3_paged_engine,test_internlm2_paged_engine,test_llama_paged_engine). Two branches, two builds, the same nine.The tenth —
test_minimax_h3(SEGFAULT, 11.81 s) — passed on that baseline, so it was the one difference. It is now fully attributed, and it is not this brick.Re-run standalone, serially, under the lock, on a box that had just rebooted and was idle (
up 5 min, load 0.12, no CUDA process resident). It still fails — so the contention hypothesis I had recorded as "plausible, not proven" is REFUTED, not quietly kept. It is #486, already open: "test_minimax_h3 is RED on dgx (GB10): cudaFree invalid argument + SIGSEGV when two CUDA cases run in one process". The re-run reproduces that issue's recorded signature number for number —minimax_h3: the WHOLE t2va path composes end to endthrowsvt cuda: cudaFree: invalid argument(:3537),an NVFP4 checkpoint loads into a runnable DiTSIGSEGVs (:3977),38 | 36 passed | 2 failed | 41 skipped,42724assertions.Its root cause is #516 — "vllm::Pool() free list is keyed by size class with no DEVICE in the key" — which is exactly what
row/pool-device-keyrepairs. The baseline branch was carrying a FIX, not this branch a defect. That also independently corroborates the static argument above: no model code calls these ops at all.All ten failures reproduce STANDALONE (
CTEST_EXIT=8each,W2_REFAIL_DONE). That cuts both ways: none is a contention flake to be waved through, and all ten are deterministic reds that exist independently of this branch — which is what makes the same-name baseline comparison sound rather than two noisy runs agreeing by luck. All ten are pre-existing and tracked (#486/#516; #233 coverstest_glm4_moe_lite_paged_engine"plus 4 more pre-existing ctest failures"). None is W2's.Two environment findings, reported because they cost other agents time
flock -w 3600 $HOME/gpu.lock ./w2refail.sh— an OUTERflockwrapping a script that takes the same lock itself on its own fd.flocklocks an open file description, so the inner acquisition blocks against its own parent: it held the mutex while making no progress, with three other agents' jobs queued behind it. I killed the stack, the lock passed straight to a waiter, and relaunched with a single acquisition.Carried forward, not relitigated
src/vt/cuda/cuda_mamba2_ssd.cuhincluded bycuda_gdn.cu, not a new.cu— a new library TU must be listed in the rootCMakeLists.txt, whichcheck-doc-checkpointclassifiesuser_usage+landing_page(check-doc-checkpoint treats every root CMakeLists.txt edit as a usage change, so adding a source file demands a docs/USAGE.md edit that has nothing true to say #515). Same reason as W1'scpu_ops.cppplacement.A < 0orstate_indicesdistinctness; both operands are on-device and re-reading costs a D2H + stream sync per call and makes the op uncapturable in a CUDA graph, mirroring the policycuda_gdn.cu:8-13already states. Kernels stay memory safe. Owed (recorded in §8.3, not implemented): route both through the deferred error ring atcuda_ops.cu:790-940.Deliberately NOT fixed here
The W2 RED run SIGSEGV'd because GB10 reports
Backend::UnifiedMemory() == true, soReferenceTierEligible(kCUDA)is true andGetOpinstalls the CPU host kernel overcudaMallocpointers.op_provider.cpp:515-526gates onUnifiedMemory()where it needsDeviceMemoryIsHostAddressable()— shared-seam semantics across three backends, filed as #547 and fixed on its own row. Every CUDA case here callsRequireNativeCudaProvider, so a device arm can never be gated by running the host arm twice.Still owed on the row
W3 (the
MambaSpecproducer), a fresh scoped review of W2 — this branch has had none — therefail.logattribution above, and the two residuals.KERNEL-SSM-MAMBAstaysINVENTORIED: no lifecycle state moved and no performance claim is made, so noSTATUS.md/BENCHMARKS.mdprojection is owed (check-doc-checkpoint:OK: public documents match the claims this change makes).scripts/agent-preflight.sh --stagedandcheck-commit-trailers --range origin/main..HEADare green.🤖 Generated with Claude Code
CI — every job run to completion on
6a418e05716 pass, 2 fail, and both failures are the
mainbaseline.plan,cuda-arch-features,vulkan-spirv-freshness,pr-size,commit-protocol-tag,device-leakage,documentation-checkpoint,agent-record,build-test-vulkan,build-test-cpu-arm64,build-test-cpu(28m10s),cuda-fat-build(1h16m18s),sanitize-cpu (address,undefined),sanitize-cpu (thread),verify (cpu),verify (vulkan)— all pass.windows-msvc-cpuandwindows-msvc-vulkanfail, and the baseline was subtracted rather than assumed:row/ENG-ISSUE-TABLE-INTAKE, a records-only change) fails both, 22m15s / 22m6s.main, failed both, 22m6s / 21m33s.This branch's own diff since the merge is one markdown file, so it cannot reach an MSVC build. No Windows-arm claim is made or repaired here.
One CI caveat worth flagging to the operator: getting that complete tally took a manual
gh run rerun. Successive pushes, and even two workflow runs on the same SHA, cancel each other through the concurrency group —conclusion: cancelledpresenting as every job "failing" at an identical duration, including trivial ones that pass in 15s. That is #274 ("main is not verified by its own CI: every run is cancelled"), already open and demonstrated there on a docs-only PR. Re-running a single workflow in isolation produced the clean result above.