Skip to content

fix(tests): the two sanitizer defects actually red on main (#395, #396) - #399

Merged
mudler merged 3 commits into
mainfrom
row/SANITIZER-TRIAGE
Aug 11, 2026
Merged

fix(tests): the two sanitizer defects actually red on main (#395, #396)#399
mudler merged 3 commits into
mainfrom
row/SANITIZER-TRIAGE

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Summary

Triage of the sanitizer half of #274, plus the two defects that triage actually
found. Both sanitizer lanes are red on main; neither failure is one of the six
#274 lists.

The six in #274 were #301 (five misaligned BF16 reads from borrowed
safetensors mmap) and #321 (AsyncLLM output thread using a freed
PromRegistry). Both landed in PR #344 (ea4deb20). A full local run of the CI
recipe at main 0eb049f7 confirms all six are green and finds exactly one
failure per lane instead:

Lane Result at 0eb049f7 Issue
address,undefined 99% tests passed, 1 tests failed out of 384test_punica_cpu #395
thread 99% tests passed, 1 tests failed out of 384test_lmcache_client #396

Closes #395.
Closes #396.

#395test_punica_cpu reads past a_stacked

ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 4 at 0x50d0000001a0 ... 0 bytes after 144-byte region
    #0 RefShrink tests/vllm/lora/test_punica_cpu.cpp:63
    #1 DOCTEST_ANON_FUNC_13 tests/vllm/lora/test_punica_cpu.cpp:312

The case add_shrink leaves a -1 slot's buffer row UNTOUCHED, not zeroed feeds
idx = {1, -1, 0, num_slots} deliberately, to prove the kernel skips an
out-of-range slot. Production AddShrink does exactly that
(src/vllm/lora/punica_cpu.cpp:128). The test's own reference helper guarded
only s < 0, so it read a[36..53] out of a 36-element adapter to compute a
value the checking loop then discards. The reference now skips what the kernel
skips.

New since the last green main sanitizer run: 03b615eb (#278) added the case
and is not an ancestor of dbd0d51c, which was green (run 31493332021).

#396MockLmcacheServer races its own accept thread

WARNING: ThreadSanitizer: data race
  Write of size 4 by main thread:  ~MockLmcacheServer ...test_lmcache_client.cpp:76
  Previous read of size 4 by T1:   Run ...test_lmcache_client.cpp:206

stop_ was made atomic; listen_fd_ was not. The destructor closes and
reassigns it before joining, so that write races the accept thread's read of
the same plain int. The mock is duplicated in test_lmcache_client.cpp and
test_lmcache_connector.cpp and both carry it — which is why CI reported the
connector on #311 and the local suite reported the client. Both fixed: publish
the descriptor once the socket is ready, hand it over with exchange(-1), read
it through load().

Verification

RED, at 0eb049f7, CI recipe (-DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SANITIZE=…, VT_POOL_BYPASS=1, serial ctest):

  • ASan/UBSan full suite: 1/384 failed — test_punica_cpu, report byte-identical
    to hosted job 93821547189
  • TSan full suite: 1/384 failed — test_lmcache_client, same race hosted job
    93616755110 reported against the connector

GREEN, re-verified after merging origin/main 0b7fbff7:

  • ASan/UBSan focused, 3/3
  • TSan focused, 5 consecutive runs, 3/3 each (10/10 before the merge)
  • -Wall -Wextra clean in both lanes, so the plain lane's -Werror holds
  • scripts/agent-preflight.sh and --staged: all gates green

Mutation, to show the punica case still catches its defect: dropping
slot >= num_slots from production AddShrink moves the ASan report from the
test helper to punica_cpu.cpp:129 and the binary aborts. Tree restored
byte-for-byte.

No suppression file, no sanitizer flag change, no expected-to-fail marking.

Notes for #274

mudler added 2 commits August 11, 2026 18:02
Both lanes of `sanitize-cpu` are red on `main` at `0eb049f7`, and neither
failure is one of the six #274 lists -- those were #301 and #321 and both
landed. A full local run of the CI recipe at that SHA finds exactly one
failure per lane:

    address,undefined: 1/384  test_punica_cpu
    thread:            1/384  test_lmcache_client

#395 -- `test_punica_cpu` heap-buffer-overflow, ASan:

    READ of size 4 ... 0 bytes after 144-byte region
      #0 RefShrink tests/vllm/lora/test_punica_cpu.cpp:63
      #1 DOCTEST_ANON_FUNC_13 tests/vllm/lora/test_punica_cpu.cpp:312

The case feeds `idx = {1, -1, 0, num_slots}` on purpose, to prove the kernel
skips an out-of-range slot. Production `AddShrink` does exactly that
(`punica_cpu.cpp:128`, `slot < 0 || slot >= num_slots`). The test's own
reference helper only guarded `s < 0`, so it read `a[36..53]` out of a
36-element adapter while computing a value the checking loop then discards.
Guard the reference the same way the kernel guards, deriving `num_slots` from
`a`'s extent since it is not a parameter.

New since the last green main sanitizer run: `03b615eb` (#278) added the case
and is not an ancestor of `dbd0d51c`, which was green.

#396 -- `MockLmcacheServer` data race, TSan:

    Write of size 4 by main thread:  ~MockLmcacheServer ...:76
    Previous read of size 4 by T1:   Run ...:206

`stop_` was made atomic; `listen_fd_` was not. The destructor closes and
reassigns it BEFORE joining the accept thread, so the write races that
thread's read of the same plain `int`. Intermittent by construction, which is
why CI sees it on the connector on one PR and the client on another -- the
mock is duplicated in both TUs and both carry the defect. Publish the
descriptor once the socket is ready, hand it over with `exchange(-1)`, and
read it through `load()`; fixed at both sites.

RED evidence, at `0eb049f7`:

  - full ASan/UBSan ctest, CI recipe: `99% tests passed, 1 tests failed out of
    384` -- `test_punica_cpu`, report byte-identical to hosted job 93821547189
  - full TSan ctest: `99% tests passed, 1 tests failed out of 384` --
    `test_lmcache_client`, same race the hosted job 93616755110 reported
    against the connector

GREEN after:

  - ASan/UBSan focused: `test_punica_cpu`, `test_lmcache_client`,
    `test_lmcache_connector` -- 3/3
  - TSan focused, 10 consecutive runs: 3/3 each time
  - `-Wall -Wextra` clean in both lanes, so the plain lane's `-Werror` holds

Mutation, to prove the punica case still catches what it exists to catch:
dropping `slot >= num_slots` from production `AddShrink` moves the ASan report
from the test helper to `punica_cpu.cpp:129` and the binary aborts. Tree
restored byte-for-byte.

No suppression file, no sanitizer flag change, no expected-to-fail marking.

Refs #274.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…epted fd residual, and #400

Fresh review at `da543d83` returned PASS with three low, non-blocking findings.
Two of them are rules this branch was actually breaking, so they are answered
here rather than deferred; the third is a code comment.

1. AGENTS.md wants an issue linked in three places that agree -- the roadmap
   issue table, the PR body, and the ROW'S SPEC. #395 and #396 had the first two
   and not the third. Added to `specs/lora-adapter.md` and
   `specs/lmcache-cpp-client-connector.md`, each stating what the bug actually
   was, which is what `.agents/bugfixing.md` asks the spec to carry.

2. Mutating `BgmvShrink`'s `s >= num_slots` guard away leaves `test_punica_cpu`
   GREEN -- 8 cases / 149 assertions / rc=0. So does the same guard in
   `BgmvExpandSlice`. The `bgmv_shrink` case feeds `idx = {0,1,2,-1,1,0}` with
   `num_slots = 3`: no index is ever out of range, so the upper half of those
   guards has no test at all. PRE-EXISTING since `1d37f152`, not introduced
   here, and a different fixture change from this branch's scope -- filed as
   #400 with the one-line close, and recorded in the roadmap table.

3. The reviewer asked whether `exchange(-1)` then `close(fd)` can race a stale
   `load()` in `Run()`. It cannot reach a reused descriptor in this mock: only
   two threads exist, the destructor opens nothing and joins before returning,
   so the worst case is `EBADF`, which is the loop's intended exit. Named as an
   accepted residual in a comment at both sites, including why the obvious
   alternative is wrong -- deferring the `close()` until after the join does not
   wake `accept()` on Darwin, which these files explicitly target.

Re-verified after the edits: ASan/UBSan 2/2, TSan 5 consecutive runs 2/2 each,
zero `-Wall -Wextra` warnings in both lanes, `agent-preflight.sh --staged` all
gates green.

Refs #274. Refs #395. Refs #396. Refs #400.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Fresh review at da543d83: PASS, three low non-blocking findings. All three answered in 75cbc7b9, which is records + comments only — no behaviour change to the fix.

Reviewer evidence worth keeping in the PR record (their runs, not mine):

  • Mutation A — reverting only the RefShrink guard reproduces the ASan heap-buffer-overflow at that line. The fix is motivated.
  • Mutation B — dropping slot >= num_slots from production AddShrink moves the report to punica_cpu.cpp:129 and the test still fails, so the change did not weaken the case. They went further and built a plain -O2 non-sanitized driver on the same inputs: 6 REQUIRE(got == kSentinel) violations, so the assertion catches a dropped kernel guard even where ASan is not running.
  • Mutation C — reverting listen_fd_ to a plain int in test_lmcache_client.cpp reproduces the race at 7/25 runs (28%). After the fix: 0/50 on the client and 0/25 on the connector. Note for future CI readers: doctest prints 3 passed / Status: SUCCESS! on a racing run — the only signal is TSan exit 66.
  • num_slots = a.size() / a_stride checked at all three RefShrink call sites: exact at each, and fail-safe by construction (integer division floors, so any accepted s satisfies (s+1)*a_stride <= a.size()). The narrower-rank case at :340 does not call RefShrink — it carries its own inline reference.
  • Roadmap issue table is main's version plus exactly the new rows, every other key byte-identical.
  • Zero warning: lines across both full sanitizer builds.

Findings and what 75cbc7b9 does about them:

  1. main is RED on sanitize-cpu (address,undefined): test_punica_cpu RefShrink reads past a_stacked for an out-of-range slot #395/test_lmcache_connector data race under TSan: MockLmcacheServer writes non-atomic listen_fd_ before joining its accept thread #396 were linked from the roadmap table and this PR but not from the row specs, which AGENTS.md requires to agree. Added to specs/lora-adapter.md and specs/lmcache-cpp-client-connector.md, each stating what the bug actually was.
  2. Mutating BgmvShrink's s >= num_slots guard away leaves test_punica_cpu green — its fixture never feeds an out-of-range index. Same for BgmvExpandSlice. Pre-existing since 1d37f152, not this PR, and a different fixture change from this scope: filed as test_punica_cpu does not cover the out-of-range slot guard in BgmvShrink or BgmvExpandSlice: dropping either leaves the suite green #400 with the one-line close, and added to the roadmap table.
  3. The exchange(-1)-then-close() vs a stale load() question: not reachable here (two threads, the destructor opens nothing and joins before returning, worst case EBADF which is the loop's intended exit). Named as an accepted residual in a comment at both sites, including why deferring the close() past the join is not the fix — on Darwin shutdown() alone does not wake accept().

Re-verified after the edits: ASan/UBSan 2/2, TSan 5 consecutive runs 2/2, zero warnings, agent-preflight.sh --staged all green.

@localai-bot

Copy link
Copy Markdown
Collaborator Author

Hosted CI green at 75cbc7b9both sanitizer lanes pass, which is the first fully green sanitize-cpu pair through this window:

sanitize-cpu (address,undefined)   pass   35m14s   job 93887124466
sanitize-cpu (thread)              pass   23m53s   job 93887124490

Everything else on the run is green too: build-test-cpu (26m30s), build-test-cpu-arm64, build-test-vulkan, verify (cpu), verify (vulkan), agent-record, commit-protocol-tag, documentation-checkpoint, device-leakage, cuda-arch-features, vulkan-spirv-freshness, pr-size, plan. Only cuda-fat-build is still running.

That closes the loop on #274 finding #1: the lane was red on main for four distinct defects across this window (#301, #321, #395, #396), it is green here, and it can now tell a regression from the floor again.

Not merging — merge authority is not recorded in developer preferences and this is a helper claim. Ready for an operator to rerun the gate and land.

@mudler
mudler merged commit b8293c8 into main Aug 11, 2026
19 of 20 checks passed
@localai-bot
localai-bot deleted the row/SANITIZER-TRIAGE branch August 11, 2026 19:39
localai-bot pushed a commit that referenced this pull request Aug 11, 2026
Brings in #407 (Marlin gencode), #399 (sanitizer defects), #391 (CPU decode
barrier) and the SPEC-DSPARK correction. No overlap with this branch: none of
them touch .github/workflows/, scripts/main-baseline.py or
scripts/agent-preflight.sh.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants