From 6e4da480a61ed8d771aa340419c3c7a0c3cc310a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 18:29:50 +0000 Subject: [PATCH 01/35] spec(ENG-RELEASE-WINDOWS): bind empty-argv repair (#512) The exact merged-SHA release run compiles Windows, then PowerShell rejects every checked test that takes no arguments. Bind the repair to the real helper contract and require both Windows release lanes before the pre-alpha tag. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 46 ++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index d938f39cc..774fe8bb0 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -74,6 +74,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#475](https://github.com/mudler/vllm.cpp/issues/475) | `ENG-RELEASE-WINDOWS` | PR #446 release benchmark projection exceeds its page budget after merging current main | bug | | [#499](https://github.com/mudler/vllm.cpp/issues/499) | `ENG-RELEASE-WINDOWS` | Release binaries report project version instead of prerelease identity | bug | | [#500](https://github.com/mudler/vllm.cpp/issues/500) | `ENG-RELEASE-WINDOWS` | MSVC release build lacks ostream definition for string_view doctest diagnostics | bug | +| [#512](https://github.com/mudler/vllm.cpp/issues/512) | `ENG-RELEASE-WINDOWS` | Windows release runner rejects empty process arguments | bug | | [#170](https://github.com/mudler/vllm.cpp/issues/170) | `ENG-RELEASE-BINARIES` | Publish container images to GHCR (cuda, vulkan, cpu) | feature | | [#322](https://github.com/mudler/vllm.cpp/issues/322) | `ENG-RELEASE-BINARIES` | Release handoff collides with tracked checkout `assets` directory | bug | | [#406](https://github.com/mudler/vllm.cpp/issues/406) | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate fails on how commits LAND: GitHub's Co-authored-by displaces the trailer block | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index b34a5b0b7..34b915361 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -4,7 +4,8 @@ Identity: `ENG-RELEASE-WINDOWS` Issues: [#499](https://github.com/mudler/vllm.cpp/issues/499) and -[#500](https://github.com/mudler/vllm.cpp/issues/500) +[#500](https://github.com/mudler/vllm.cpp/issues/500), with post-merge follow-up +[#512](https://github.com/mudler/vllm.cpp/issues/512) Parent specifications: [release-binary-matrix.md](release-binary-matrix.md) and @@ -251,6 +252,49 @@ Native Windows compilation, the complete ten-tuple non-publishing workflow, and the tag-run publication/audit remain post-merge acceptance gates. No tag or release is authorized by the local evidence alone. +### Follow-up outcome and repair contract: issue #512 + +PR #508 merged as `2bc4be070a3883f0f7115682469a289f42d86d1a`. +Exact-SHA dry run +[`31625581156`](https://github.com/mudler/vllm.cpp/actions/runs/31625581156) +proved the prior fixes far enough for Windows CPU to compile +`test_cpu_isa_x86.cpp` successfully. Job `94211117810` then failed before its +first no-argument test executed: + +```text +Cannot bind argument to parameter 'Arguments' because it is an empty array. +``` + +`scripts/build-windows-release.ps1:20-26` declares `Invoke-Checked` with a +mandatory `string[] Arguments` parameter. The script deliberately passes +`@()` when executing tests that take no arguments at lines 226-231 and 264-266. +PowerShell parameter binding rejects that explicit empty collection before the +helper invokes the executable. The build, strict MSVC warning gate, and +`test_cpu_isa_x86` link all succeeded; this is not a recurrence of #500. + +The #512 repair is limited to making `Invoke-Checked` accept an explicitly empty +argument array while preserving argument splatting and the non-zero exit-status +failure contract. It must not remove `Invoke-Checked`, bypass any test, add a +dummy argument, weaken the Windows release gate, change a release tuple, or +publish a tag. + +RED-first evidence must execute the real PowerShell helper contract with both +zero and nonzero argument counts. Before the fix, the zero-argument arm must +fail with the observed parameter-binding error. After the fix, it must prove +that the target runs exactly once with zero arguments, that nonempty arguments +arrive unchanged, and that a nonzero child exit remains rejected. Mutating the +helper back to a mandatory non-empty array must make the focused contract red. + +Focused acceptance is the PowerShell contract test plus +`tests.scripts.test_release_windows_metadata`, +`tests.scripts.test_release_pipeline`, the direct Windows portability and +release-binary checkers, and the full repository preflight. Hosted acceptance +requires both Windows CPU and Vulkan jobs in a new exact-merged-SHA ten-tuple +dry run to execute, package, and validate their archives. All other required +tuples, aggregate `build`, and `verify` must also pass on that same SHA before +`v0.0.3-pre.1` is tagged. The tag run must then pass all 15 required jobs and +the authenticated post-publication audit over exactly 32 assets. + Fresh review of immutable implementation `e0b17eb9` found that the compiled version test derived its default expectation from the same cache value under test. Mutating the cache default from `${PROJECT_VERSION}` to `9.9.9` therefore From 1f520cc3052e040a74245af0c5c354a416805ca0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 18:33:15 +0000 Subject: [PATCH 02/35] spec(ENG-RELEASE-WINDOWS): bind Vulkan env repair (#514) The exact merged-SHA Vulkan build reaches a second failure: its cross-device test calls POSIX environment APIs under MSVC. Bind the repair to a test-local platform seam and preserve both fused-chain tiers plus environment restoration. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 26 ++++++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 774fe8bb0..3a4b1405c 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -75,6 +75,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#499](https://github.com/mudler/vllm.cpp/issues/499) | `ENG-RELEASE-WINDOWS` | Release binaries report project version instead of prerelease identity | bug | | [#500](https://github.com/mudler/vllm.cpp/issues/500) | `ENG-RELEASE-WINDOWS` | MSVC release build lacks ostream definition for string_view doctest diagnostics | bug | | [#512](https://github.com/mudler/vllm.cpp/issues/512) | `ENG-RELEASE-WINDOWS` | Windows release runner rejects empty process arguments | bug | +| [#514](https://github.com/mudler/vllm.cpp/issues/514) | `ENG-RELEASE-WINDOWS` | Windows Vulkan release test uses POSIX environment APIs | bug | | [#170](https://github.com/mudler/vllm.cpp/issues/170) | `ENG-RELEASE-BINARIES` | Publish container images to GHCR (cuda, vulkan, cpu) | feature | | [#322](https://github.com/mudler/vllm.cpp/issues/322) | `ENG-RELEASE-BINARIES` | Release handoff collides with tracked checkout `assets` directory | bug | | [#406](https://github.com/mudler/vllm.cpp/issues/406) | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate fails on how commits LAND: GitHub's Co-authored-by displaces the trailer block | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 34b915361..43a999e35 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -5,7 +5,8 @@ Identity: `ENG-RELEASE-WINDOWS` Issues: [#499](https://github.com/mudler/vllm.cpp/issues/499) and [#500](https://github.com/mudler/vllm.cpp/issues/500), with post-merge follow-up -[#512](https://github.com/mudler/vllm.cpp/issues/512) +[#512](https://github.com/mudler/vllm.cpp/issues/512) and +[#514](https://github.com/mudler/vllm.cpp/issues/514) Parent specifications: [release-binary-matrix.md](release-binary-matrix.md) and @@ -295,6 +296,29 @@ tuples, aggregate `build`, and `verify` must also pass on that same SHA before `v0.0.3-pre.1` is tagged. The tag run must then pass all 15 required jobs and the authenticated post-publication audit over exactly 32 assets. +Issue #514 is a separate Windows Vulkan compile defect from the same dry run. +Job `94211117906` compiled and linked `test_cpu_isa_x86`, then MSVC rejected +`setenv` at `tests/vt/test_backend_cross_device.cpp:1004,1048` and `unsetenv` +at line 1050 with C3861. Those POSIX-only calls set, then restore, +`VT_FUSED_TIER` around the cross-device fused-chain test. Linux accepts them; +MSVC exposes `_putenv_s` instead. + +The #514 repair is limited to a test-local cross-platform environment seam. +On Windows it must use checked `_putenv_s`, with an empty value removing the +variable. On POSIX it must preserve checked `setenv(..., 1)` and `unsetenv`. +The fused-chain case must still execute tiers 0 and 1, assert the selected tier, +and restore the caller's prior environment state. Do not disable the case, +change fused-chain production behavior, add a global compatibility macro, or +weaken `/W4` or any release gate. + +RED-first structural coverage must reject the three live unguarded POSIX calls +and require both platform arms in the real translation unit. Mutation must +remove or bypass the Windows arm and make the focused suite fail. Focused green +is the relevant Windows portability suite and direct checker, a clean local +CPU compile/execution of `test_backend_cross_device`, and the full preflight. +Hosted acceptance remains the same exact-merged-SHA dry run: Windows Vulkan +must compile, execute, package, and validate before any tag is authorized. + Fresh review of immutable implementation `e0b17eb9` found that the compiled version test derived its default expectation from the same cache value under test. Mutating the cache default from `${PROJECT_VERSION}` to `9.9.9` therefore From 9f8f03b699670d80d7a0d638d3557c0f790efefe Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 18:43:40 +0000 Subject: [PATCH 03/35] fix(release): accept empty Windows argv (#512) The Windows release driver passes @() to tests that take no arguments. PowerShell rejects the mandatory array before it invokes each test. Allow the empty collection and exercise zero, exact nonempty, and failed child invocations in the hosted Windows contract. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 15 +++++ scripts/build-windows-release.ps1 | 64 ++++++++++++++++++- .../scripts/test_release_windows_metadata.py | 23 +++++++ 3 files changed, 101 insertions(+), 1 deletion(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 43a999e35..88cc030b8 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -328,3 +328,18 @@ declaration. With the `9.9.9` mutation applied, that assertion failed exactly with `9.9.9 != ${PROJECT_VERSION}`; after restoration it and the complete 42-test release pipeline suite passed. Production CMake and runtime code remain unchanged by this follow-up. + +Issue #512 implementation makes the existing mandatory `Arguments` parameter +explicitly accept an empty collection; argument splatting and the checked +nonzero exit path are unchanged. The Windows `-ContractTest` path now invokes a +temporary recording target once with zero arguments, invokes it with three +literal nonempty arguments and checks each value, and requires an exit-23 child +to be rejected. Before the production edit, the new focused contract failed on +the missing `AllowEmptyCollection` declaration; the hosted exact-SHA failure +above is the matching real-PowerShell RED execution. The local host has no +PowerShell runtime, so execution of the new live process contract remains a +required native Windows PR gate rather than being inferred from local Python. +After the fix, Windows metadata passed 8/8, release pipeline passed 42/42, and +both direct Windows portability and release-binary checkers passed. Removing +`AllowEmptyCollection` made the focused contract red again, and both changed +files were restored to their exact pre-mutation SHA-256 values. diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index b8725cce1..2167d1a35 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -19,13 +19,74 @@ if ($ArtifactId -ne "windows-x86_64-msvc-$Backend") { } function Invoke-Checked { param([Parameter(Mandatory)][string]$Program, - [Parameter(Mandatory)][string[]]$Arguments) + [Parameter(Mandatory)][AllowEmptyCollection()][string[]]$Arguments) & $Program @Arguments if ($LASTEXITCODE -ne 0) { throw "$Program exited with status $LASTEXITCODE" } } +function Invoke-CheckedContractTests { + $temporaryDir = Join-Path ([System.IO.Path]::GetTempPath()) ` + "vllm-invoke-checked-$([guid]::NewGuid().ToString('N'))" + $recordingTarget = Join-Path $temporaryDir "record-arguments.cmd" + $failingTarget = Join-Path $temporaryDir "fail.cmd" + $callLog = Join-Path $temporaryDir "calls.txt" + $savedCallLog = $env:VLLM_INVOKE_CHECKED_LOG + try { + New-Item -ItemType Directory -Path $temporaryDir | Out-Null + @' +@echo off +>>"%VLLM_INVOKE_CHECKED_LOG%" echo CALL +>>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG1=[%~1] +>>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG2=[%~2] +>>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG3=[%~3] +>>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG4=[%~4] +exit /b 0 +'@ | Set-Content -LiteralPath $recordingTarget -Encoding ascii + @' +@echo off +exit /b 23 +'@ | Set-Content -LiteralPath $failingTarget -Encoding ascii + $env:VLLM_INVOKE_CHECKED_LOG = $callLog + + Invoke-Checked $recordingTarget @() + $zeroArgumentCalls = @(Get-Content -LiteralPath $callLog) + if (($zeroArgumentCalls -join "`n") -ne + "CALL`nARG1=[]`nARG2=[]`nARG3=[]`nARG4=[]") { + throw "zero-argument target was not invoked exactly once without arguments" + } + + Remove-Item -LiteralPath $callLog + Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value") + $nonemptyCalls = @(Get-Content -LiteralPath $callLog) + if (($nonemptyCalls -join "`n") -ne + "CALL`nARG1=[alpha]`nARG2=[two words]`nARG3=[--flag=value]`nARG4=[]") { + throw "nonempty arguments did not arrive unchanged" + } + + $rejected = $false + try { + Invoke-Checked $failingTarget @() + } catch { + if ($_.Exception.Message -notmatch 'exited with status 23') { + throw + } + $rejected = $true + } + if (-not $rejected) { + throw "nonzero child exit was accepted" + } + } finally { + if ($null -eq $savedCallLog) { + Remove-Item Env:VLLM_INVOKE_CHECKED_LOG -ErrorAction SilentlyContinue + } else { + $env:VLLM_INVOKE_CHECKED_LOG = $savedCallLog + } + Remove-Item -Recurse -Force $temporaryDir -ErrorAction SilentlyContinue + } +} + function Assert-CrtPolicy { param([Parameter(Mandatory)][string[]]$DirectiveOutput, [Parameter(Mandatory)][string[]]$ImportOutput) @@ -157,6 +218,7 @@ function Invoke-UnsupportedTierContractTests { } if ($ContractTest) { + Invoke-CheckedContractTests Invoke-CrtContractTests Invoke-UnsupportedTierContractTests Write-Host "Windows PowerShell/CRT contract tests OK" diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 4ed2cfe30..3e756cb86 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -99,6 +99,29 @@ def test_contract_test_precedes_required_release_environment(self) -> None: self.assertLess(contract, contract_exit) self.assertLess(contract_exit, environment) + def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text(encoding="utf-8") + helper_start = script.index("function Invoke-Checked {") + helper_end = script.index("\n}\n", helper_start) + len("\n}\n") + helper = script[helper_start:helper_end] + self.assertIn("[AllowEmptyCollection()][string[]]$Arguments", helper) + self.assertIn("& $Program @Arguments", helper) + self.assertIn("if ($LASTEXITCODE -ne 0)", helper) + + contract_start = script.index("function Invoke-CheckedContractTests {") + contract_end = script.index("\n}\n", contract_start) + len("\n}\n") + contract = script[contract_start:contract_end] + for behavior in ( + "zero-argument target", + "nonempty arguments", + "nonzero child exit", + ): + with self.subTest(behavior=behavior): + self.assertIn(behavior, contract) + + dispatch = script[script.index("if ($ContractTest)"):] + self.assertEqual(dispatch.count("Invoke-CheckedContractTests"), 1) + def test_pe_report_rejects_msys_debug_crt_and_developer_paths(self) -> None: with tempfile.TemporaryDirectory() as temporary: args = self.fixture(Path(temporary), "cpu") From 0a70da1dfaca9f29bfe7be9ef9396e7f386991d8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 18:53:44 +0000 Subject: [PATCH 04/35] fix(tests): make fused tier environment portable (#514) Use checked native environment APIs in the cross-device fused-chain test while preserving tier selection and prior-state restoration on Windows and POSIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 14 +++++ .../scripts/test_check_windows_portability.py | 52 +++++++++++++++++++ tests/vt/test_backend_cross_device.cpp | 18 +++++-- 3 files changed, 81 insertions(+), 3 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 88cc030b8..57ab71a78 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -319,6 +319,20 @@ CPU compile/execution of `test_backend_cross_device`, and the full preflight. Hosted acceptance remains the same exact-merged-SHA dry run: Windows Vulkan must compile, execute, package, and validate before any tag is authorized. +The #514 implementation keeps the environment change local to the cross-device +test. A single `SetTestEnvironment` helper uses checked `_putenv_s` on Windows, +passing an empty value for removal, and retains checked `setenv(..., 1)` and +`unsetenv` on POSIX. The fused-chain case still selects tiers 0 and 1, asserts +the active tier, and restores either the saved value or the prior absence. + +RED-first structural coverage failed on the pinned candidate because the helper +and Windows arm were absent. After the repair, the 72-case Windows portability +suite and direct checker passed. A scratch mutation deleting the `_putenv_s` +call failed the focused contract, and the candidate files were restored +byte-for-byte. A clean Release CPU configuration compiled and executed +`test_backend_cross_device`: 19 cases and 6 assertions passed. Native MSVC and +the exact-merged-SHA ten-tuple dry run remain hosted acceptance gates. + Fresh review of immutable implementation `e0b17eb9` found that the compiled version test derived its default expectation from the same cache value under test. Mutating the cache default from `${PROJECT_VERSION}` to `9.9.9` therefore diff --git a/tests/scripts/test_check_windows_portability.py b/tests/scripts/test_check_windows_portability.py index a5202a936..4619cc530 100644 --- a/tests/scripts/test_check_windows_portability.py +++ b/tests/scripts/test_check_windows_portability.py @@ -562,6 +562,58 @@ def test_cpu_isa_test_owns_ostream_for_string_view_diagnostics(self) -> None: ) self.assertIn("std::string_view::npos", source) + def test_cross_device_fused_tier_environment_is_windows_portable(self) -> None: + source = (REPO / "tests/vt/test_backend_cross_device.cpp").read_text( + encoding="utf-8" + ) + helper_span = checker._cpp_function_body_span( + source, r"\bSetTestEnvironment\s*\(" + ) + self.assertIsNotNone(helper_span, "SetTestEnvironment helper") + assert helper_span is not None + helper = source[helper_span[1]:helper_span[2]] + + directives = checker._cpp_directive_view(helper) + self.assertRegex(directives, r"(?m)^\s*#\s*ifdef\s+_WIN32\s*$") + self.assertRegex(directives, r"(?m)^\s*#\s*else\s*$") + self.assertRegex(directives, r"(?m)^\s*#\s*endif\s*$") + + active = checker._cpp_structural_view(helper) + self.assertRegex( + directives, + r"REQUIRE\s*\(\s*::_putenv_s\s*\(\s*name\s*,\s*" + r'value\s*!=\s*nullptr\s*\?\s*value\s*:\s*""\s*\)\s*' + r"==\s*0\s*\)", + ) + self.assertRegex( + active, + r"REQUIRE\s*\(\s*setenv\s*\(\s*name\s*,\s*value\s*,\s*1\s*\)\s*" + r"==\s*0\s*\)", + ) + self.assertRegex( + active, + r"REQUIRE\s*\(\s*unsetenv\s*\(\s*name\s*\)\s*==\s*0\s*\)", + ) + + windows_lines = "\n".join( + line for _, line in checker.windows_possible_lines( + checker.without_cpp_comments(source) + ) + ) + self.assertIn("::_putenv_s", windows_lines) + self.assertNotRegex(windows_lines, r"(? None: source = "src/vt/cuda/nvfp4_persistent_cache.cpp" for condition, expected in (("NOT WIN32", {source}), ("WIN32", set()), ("NOT APPLE", set())): diff --git a/tests/vt/test_backend_cross_device.cpp b/tests/vt/test_backend_cross_device.cpp index a56116eb4..bcfabfafe 100644 --- a/tests/vt/test_backend_cross_device.cpp +++ b/tests/vt/test_backend_cross_device.cpp @@ -96,6 +96,18 @@ std::vector RegisteredDevices() { bool OpAvailable(vt::OpId op, DeviceType t) { return vt::OpRegistered(op, t); } +void SetTestEnvironment(const char* name, const char* value) { +#ifdef _WIN32 + REQUIRE(::_putenv_s(name, value != nullptr ? value : "") == 0); +#else + if (value != nullptr) { + REQUIRE(setenv(name, value, 1) == 0); + } else { + REQUIRE(unsetenv(name) == 0); + } +#endif +} + // A device-resident f32 buffer with host staging, so one body serves a unified // backend (Metal, GB10) and a discrete one identically: every transfer goes // through Backend::Copy rather than assuming the host can dereference the @@ -1001,7 +1013,7 @@ TEST_CASE("FusedChain matches the CPU oracle within NMSE <= 5e-4 (both tiers)") for (int tier : {0, 1}) { CAPTURE(tier); - setenv("VT_FUSED_TIER", tier == 0 ? "0" : "1", 1); + SetTestEnvironment("VT_FUSED_TIER", tier == 0 ? "0" : "1"); // ASSERT the tier actually took effect rather than trusting the log: doctest // CAPTURE is lazily stringified, so a mis-set environment would silently // run the same path twice and still look like two-tier coverage. @@ -1045,9 +1057,9 @@ TEST_CASE("FusedChain matches the CPU oracle within NMSE <= 5e-4 (both tiers)") } if (had_prev) { - setenv("VT_FUSED_TIER", saved.c_str(), 1); + SetTestEnvironment("VT_FUSED_TIER", saved.c_str()); } else { - unsetenv("VT_FUSED_TIER"); + SetTestEnvironment("VT_FUSED_TIER", nullptr); } } From 472f77d509f6c1ca6a9efd240e4c4eaa418f0b1f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 19:27:14 +0000 Subject: [PATCH 05/35] test(release): bind Windows repair proofs (#512 #514) Reject a dead Invoke-Checked contract path and require both fused-tier restoration branches to target the exact environment key. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 23 +++++++++++++ .../scripts/test_check_windows_portability.py | 14 ++++++-- .../scripts/test_release_windows_metadata.py | 34 ++++++++++++++++--- 3 files changed, 64 insertions(+), 7 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 57ab71a78..d1d1d5660 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -357,3 +357,26 @@ After the fix, Windows metadata passed 8/8, release pipeline passed 42/42, and both direct Windows portability and release-binary checkers passed. Removing `AllowEmptyCollection` made the focused contract red again, and both changed files were restored to their exact pre-mutation SHA-256 values. + +Fresh review of combined candidate `0a70da1d` found two test-strength gaps. +Adding an unconditional `return` as the first statement of +`Invoke-CheckedContractTests` left all eight Windows metadata tests green even +though the hosted `-ContractTest` path would silently skip its runtime proofs. +Changing both fused-tier restoration calls to target +`VT_FUSED_TIER_WRONG` likewise left the focused portability test and direct +checker green. The test-only repair now binds the three exact `Invoke-Checked` +calls and their ordered literal outcomes, rejects an executable early control +exit in the contract-test function, and pins both restoration branches to the +exact `VT_FUSED_TIER` key with `saved.c_str()` or `nullptr` respectively. + +After the repair, the early-return mutation and separate wrong-key mutations +in the prior-present and prior-absent restoration branches each failed their +focused test. The existing `AllowEmptyCollection`-removal and `_putenv_s`- +removal mutations also remained red. Production files were restored to their +pre-mutation SHA-256 values. The combined Windows metadata, release pipeline, +and Windows portability suites passed 122/122, followed by both direct Windows +portability and release-binary checkers. Native PowerShell execution and MSVC +compilation remain the required external Windows PR gates. The full unstaged +repository preflight passed every gate except one transient tempfile failure in +`test_check_release_binary_contract`; an immediate isolated retry of that exact +suite passed all 30 tests. diff --git a/tests/scripts/test_check_windows_portability.py b/tests/scripts/test_check_windows_portability.py index 4619cc530..9138ff44e 100644 --- a/tests/scripts/test_check_windows_portability.py +++ b/tests/scripts/test_check_windows_portability.py @@ -608,11 +608,19 @@ def test_cross_device_fused_tier_environment_is_windows_portable(self) -> None: '(both tiers)")' ) case_end = source.index("\nTEST_CASE(", case_start + 1) - case = checker._cpp_structural_view(source[case_start:case_end]) + case_source = checker.without_cpp_comments(source[case_start:case_end]) + case = checker._cpp_structural_view(case_source) self.assertEqual(len(re.findall(r"\bSetTestEnvironment\s*\(", case)), 3) self.assertRegex(case, r"REQUIRE\s*\(\s*vt::FusedTier\s*\(\s*\)\s*==\s*tier") - self.assertIn("had_prev", case) - self.assertIn("saved", case) + self.assertRegex( + case_source, + r'if\s*\(\s*had_prev\s*\)\s*\{\s*' + r'SetTestEnvironment\s*\(\s*"VT_FUSED_TIER"\s*,\s*' + r'saved\.c_str\s*\(\s*\)\s*\)\s*;\s*' + r'\}\s*else\s*\{\s*' + r'SetTestEnvironment\s*\(\s*"VT_FUSED_TIER"\s*,\s*' + r'nullptr\s*\)\s*;\s*\}', + ) def test_posix_cache_source_requires_exact_not_win32_cmake_guard(self) -> None: source = "src/vt/cuda/nvfp4_persistent_cache.cpp" diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 3e756cb86..b7d35265d 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -6,6 +6,7 @@ import argparse import importlib.util import json +import re import tempfile import unittest from pathlib import Path @@ -111,13 +112,38 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) contract_start = script.index("function Invoke-CheckedContractTests {") contract_end = script.index("\n}\n", contract_start) + len("\n}\n") contract = script[contract_start:contract_end] + required_statements = ( + "Invoke-Checked $recordingTarget @()", + '"CALL`nARG1=[]`nARG2=[]`nARG3=[]`nARG4=[]"', + 'Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value")', + '"CALL`nARG1=[alpha]`nARG2=[two words]`nARG3=[--flag=value]`nARG4=[]"', + "Invoke-Checked $failingTarget @()", + "if (-not $rejected)", + ) + cursor = 0 + for statement in required_statements: + with self.subTest(statement=statement): + self.assertEqual(contract.count(statement), 1) + offset = contract.index(statement) + self.assertGreaterEqual(offset, cursor) + cursor = offset + len(statement) + executable_contract = re.sub( + r"(?ms)^[ \t]*@'\s*$.*?^[ \t]*'@[^\n]*$", + "", + contract, + ) + self.assertNotRegex( + executable_contract, + r"(?m)^\s*(?:return|exit|break|continue)(?:\s|$)", + "Invoke-Checked contract proofs must not be bypassable", + ) for behavior in ( - "zero-argument target", - "nonempty arguments", - "nonzero child exit", + "zero-argument target was not invoked exactly once without arguments", + "nonempty arguments did not arrive unchanged", + "nonzero child exit was accepted", ): with self.subTest(behavior=behavior): - self.assertIn(behavior, contract) + self.assertEqual(contract.count(behavior), 1) dispatch = script[script.index("if ($ContractTest)"):] self.assertEqual(dispatch.count("Invoke-CheckedContractTests"), 1) From 1731f9fb1d5e0a2e3adddd269f52e0db3871a859 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 20:31:17 +0000 Subject: [PATCH 06/35] spec(release): record hosted argv contract failure (#525) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 39 ++++++++++++++++++- 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 4201873f1..8c68b7986 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -80,6 +80,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#500](https://github.com/mudler/vllm.cpp/issues/500) | `ENG-RELEASE-WINDOWS` | MSVC release build lacks ostream definition for string_view doctest diagnostics | bug | | [#512](https://github.com/mudler/vllm.cpp/issues/512) | `ENG-RELEASE-WINDOWS` | Windows release runner rejects empty process arguments | bug | | [#514](https://github.com/mudler/vllm.cpp/issues/514) | `ENG-RELEASE-WINDOWS` | Windows Vulkan release test uses POSIX environment APIs | bug | +| [#525](https://github.com/mudler/vllm.cpp/issues/525) | `ENG-RELEASE-WINDOWS` | Windows release contract recorder misreports non-empty argv | bug | | [#170](https://github.com/mudler/vllm.cpp/issues/170) | `ENG-RELEASE-BINARIES` | Publish container images to GHCR (cuda, vulkan, cpu) | feature | | [#322](https://github.com/mudler/vllm.cpp/issues/322) | `ENG-RELEASE-BINARIES` | Release handoff collides with tracked checkout `assets` directory | bug | | [#406](https://github.com/mudler/vllm.cpp/issues/406) | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate fails on how commits LAND: GitHub's Co-authored-by displaces the trailer block | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index d1d1d5660..5ae83e449 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -6,7 +6,8 @@ Issues: [#499](https://github.com/mudler/vllm.cpp/issues/499) and [#500](https://github.com/mudler/vllm.cpp/issues/500), with post-merge follow-up [#512](https://github.com/mudler/vllm.cpp/issues/512) and -[#514](https://github.com/mudler/vllm.cpp/issues/514) +[#514](https://github.com/mudler/vllm.cpp/issues/514), and hosted-contract follow-up +[#525](https://github.com/mudler/vllm.cpp/issues/525) Parent specifications: [release-binary-matrix.md](release-binary-matrix.md) and @@ -380,3 +381,39 @@ compilation remain the required external Windows PR gates. The full unstaged repository preflight passed every gate except one transient tempfile failure in `test_check_release_binary_contract`; an immediate isolated retry of that exact suite passed all 30 tests. + +### Hosted contract follow-up: issue #525 + +PR #524 candidate `a4d61bbddbc1a0aa744aea72c1cff3c4ed165a72` +executed the new #512 contract on both hosted Windows lanes. CPU job +`94242642198` and Vulkan job `94242642222` both accepted the explicit empty +argument array, then failed at the exact non-empty record comparison with +`nonempty arguments did not arrive unchanged`. Both stopped in +`build-windows-release.ps1 -ContractTest` before configuration. The shared +diagnostic isolates the failure to the contract harness's temporary `.cmd` +argument recorder; it does not reopen the original empty-array binding defect +or establish a product build failure. + +The #525 repair replaces only the batch recorder with a native PowerShell +target whose parameter declaration accepts all remaining arguments and writes +a structured exact record. The real `Invoke-Checked` function must still invoke +that target for both the explicit empty array and the three literals `alpha`, +`two words`, and `--flag=value`. The exit-23 target and rejection assertion +remain live. Do not weaken an equality, normalize values, remove a behavior, +or infer a hosted pass from structural Linux coverage. + +RED-first coverage must reject the current `.cmd` recorder and require the +PowerShell recorder's remaining-arguments binding plus a structured exact +empty/non-empty record. Mutations that remove the remaining-arguments binding, +drop `two words`, bypass the real helper, or accept the nonzero child must make +the focused contract red. Focused green remains Windows metadata, release +pipeline, both direct Windows checkers, and full preflight. Hosted acceptance +requires both native Windows PR jobs to execute the complete contract and +continue into their MSVC build/runtime/archive gates. + +After #525 is integrated, current `origin/main` must be merged into the task +branch before the operator gate and plain push so the PR-size checker receives +an ancestor base. The HTTP 503 downloading glslang in job `94242642545` is an +external retry condition, not authorization to change the freshness gate. No +tag is authorized until PR #524 merges and a new exact-merged-SHA dry run has +all ten tuples plus aggregate handoff and verify green. From 732a1a26dcc033e9231acbd8eaf4de1b228d6253 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 20:31:17 +0000 Subject: [PATCH 07/35] fix(release): record Windows argv in PowerShell (#525) Replace the lossy batch recorder with a native remaining-arguments JSON fixture while preserving the real checked empty, nonempty, and exit-23 invocations. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 17 +++++++++ scripts/build-windows-release.ps1 | 36 +++++++++++-------- .../scripts/test_release_windows_metadata.py | 23 ++++++++++-- 3 files changed, 59 insertions(+), 17 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 5ae83e449..a3b60521c 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -417,3 +417,20 @@ an ancestor base. The HTTP 503 downloading glslang in job `94242642545` is an external retry condition, not authorization to change the freshness gate. No tag is authorized until PR #524 merges and a new exact-merged-SHA dry run has all ten tuples plus aggregate handoff and verify green. + +The #525 implementation replaces the batch argument recorder with a temporary +PowerShell script. Its `ValueFromRemainingArguments` string-array parameter +defaults to an empty array, and it serializes the exact count and argument array +as compact JSON. The unchanged real `Invoke-Checked` helper executes that script +once with `@()` and once with `alpha`, `two words`, and `--flag=value`; the +contract parses each record and compares its exact count and values. The +separate exit-23 batch target and rejection assertion remain unchanged. + +RED-first coverage rejected the prior `record-arguments.cmd` implementation. +After the repair, mutations removing the remaining-arguments binding, dropping +`two words`, bypassing `Invoke-Checked`, or disabling the nonzero rejection each +failed the focused metadata contract, and the script was restored byte-for-byte. +The Windows metadata and release pipeline suites passed 50/50, followed by the +direct Windows portability and release-binary checkers and the full unstaged +repository preflight. The local host has no PowerShell runtime, so native +execution remains a required hosted Windows gate. diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 2167d1a35..5fc8e34bb 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -29,21 +29,24 @@ function Invoke-Checked { function Invoke-CheckedContractTests { $temporaryDir = Join-Path ([System.IO.Path]::GetTempPath()) ` "vllm-invoke-checked-$([guid]::NewGuid().ToString('N'))" - $recordingTarget = Join-Path $temporaryDir "record-arguments.cmd" + $recordingTarget = Join-Path $temporaryDir "record-arguments.ps1" $failingTarget = Join-Path $temporaryDir "fail.cmd" $callLog = Join-Path $temporaryDir "calls.txt" $savedCallLog = $env:VLLM_INVOKE_CHECKED_LOG try { New-Item -ItemType Directory -Path $temporaryDir | Out-Null @' -@echo off ->>"%VLLM_INVOKE_CHECKED_LOG%" echo CALL ->>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG1=[%~1] ->>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG2=[%~2] ->>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG3=[%~3] ->>"%VLLM_INVOKE_CHECKED_LOG%" echo ARG4=[%~4] -exit /b 0 -'@ | Set-Content -LiteralPath $recordingTarget -Encoding ascii +param( + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$RemainingArguments = @() +) +[pscustomobject]@{ + Count = @($RemainingArguments).Count + Arguments = @($RemainingArguments) +} | ConvertTo-Json -Compress | Set-Content ` + -LiteralPath $env:VLLM_INVOKE_CHECKED_LOG -Encoding utf8 +exit 0 +'@ | Set-Content -LiteralPath $recordingTarget -Encoding utf8 @' @echo off exit /b 23 @@ -51,17 +54,20 @@ exit /b 23 $env:VLLM_INVOKE_CHECKED_LOG = $callLog Invoke-Checked $recordingTarget @() - $zeroArgumentCalls = @(Get-Content -LiteralPath $callLog) - if (($zeroArgumentCalls -join "`n") -ne - "CALL`nARG1=[]`nARG2=[]`nARG3=[]`nARG4=[]") { + $zeroArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json + if ([int]$zeroArgumentRecord.Count -ne 0 -or + @($zeroArgumentRecord.Arguments).Count -ne 0) { throw "zero-argument target was not invoked exactly once without arguments" } Remove-Item -LiteralPath $callLog Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value") - $nonemptyCalls = @(Get-Content -LiteralPath $callLog) - if (($nonemptyCalls -join "`n") -ne - "CALL`nARG1=[alpha]`nARG2=[two words]`nARG3=[--flag=value]`nARG4=[]") { + $nonemptyArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json + if ([int]$nonemptyArgumentRecord.Count -ne 3 -or + @($nonemptyArgumentRecord.Arguments).Count -ne 3 -or + $nonemptyArgumentRecord.Arguments[0] -cne "alpha" -or + $nonemptyArgumentRecord.Arguments[1] -cne "two words" -or + $nonemptyArgumentRecord.Arguments[2] -cne "--flag=value") { throw "nonempty arguments did not arrive unchanged" } diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index b7d35265d..5e5124c72 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -112,11 +112,30 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) contract_start = script.index("function Invoke-CheckedContractTests {") contract_end = script.index("\n}\n", contract_start) + len("\n}\n") contract = script[contract_start:contract_end] + self.assertNotIn("record-arguments.cmd", contract) + self.assertEqual(contract.count('"record-arguments.ps1"'), 1) + for recorder_statement in ( + "[Parameter(ValueFromRemainingArguments = $true)]", + "[string[]]$RemainingArguments = @()", + "Count = @($RemainingArguments).Count", + "Arguments = @($RemainingArguments)", + "ConvertTo-Json -Compress", + ): + with self.subTest(recorder_statement=recorder_statement): + self.assertEqual(contract.count(recorder_statement), 1) + required_statements = ( "Invoke-Checked $recordingTarget @()", - '"CALL`nARG1=[]`nARG2=[]`nARG3=[]`nARG4=[]"', + "$zeroArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json", + "[int]$zeroArgumentRecord.Count -ne 0", + "@($zeroArgumentRecord.Arguments).Count -ne 0", 'Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value")', - '"CALL`nARG1=[alpha]`nARG2=[two words]`nARG3=[--flag=value]`nARG4=[]"', + "$nonemptyArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json", + "[int]$nonemptyArgumentRecord.Count -ne 3", + "@($nonemptyArgumentRecord.Arguments).Count -ne 3", + '$nonemptyArgumentRecord.Arguments[0] -cne "alpha"', + '$nonemptyArgumentRecord.Arguments[1] -cne "two words"', + '$nonemptyArgumentRecord.Arguments[2] -cne "--flag=value"', "Invoke-Checked $failingTarget @()", "if (-not $rejected)", ) From 469ed2e456aea907c84a89b5f2ac8ad6cc105417 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 20:48:01 +0000 Subject: [PATCH 08/35] spec(cpu): bind paired proc stat samples (#530) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .../cpu-llamacpp-floor-x86-2026-08-11.md | 32 ++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 8c68b7986..6909f9b42 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -40,6 +40,7 @@ issue is not yet placed. Keyed record: update in place, never append. |---:|---|---|---| | [#168](https://github.com/mudler/vllm.cpp/issues/168) | `BACKEND-CUDA-SM110` | Jetson AGX Thor (sm_110): 32B NVFP4, Tekken tokenizer blocker | feature | | [#433](https://github.com/mudler/vllm.cpp/issues/433) | `BACKEND-GATE-CPU-LLAMACPP` | No x86_64 arm: the closed CPU floor is Arm/i8mm-only and every lever that closed it is Arm-specific | perf | +| [#530](https://github.com/mudler/vllm.cpp/issues/530) | `BACKEND-GATE-CPU-LLAMACPP` | CPU floor smoke gate samples `/proc/stat` incoherently at zero window | bug | | [#199](https://github.com/mudler/vllm.cpp/issues/199) | `BACKEND-METAL-MLX` | macOS MLX build fails on `-Werror` in MLX headers | bug | | [#41](https://github.com/mudler/vllm.cpp/issues/41) | `BACKEND-ROCM` | ROCm (AMD GPU) backend | feature | | [#132](https://github.com/mudler/vllm.cpp/issues/132) | `BACKEND-ROCM` | ROCm `-O0` RmsNorm CLR HostcallListener teardown deadlock | bug | diff --git a/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md b/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md index ea8caef59..4db9fd545 100644 --- a/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md +++ b/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md @@ -5,7 +5,8 @@ matrix](../backend-matrix.md)) · `QUANT-GGUF-COMPUTE` ([quantization matrix](../quantization-matrix.md)) · `ROAD-V1-D1` punch-list item 13, CPU half ([roadmap-v1-completion](roadmap-v1-completion.md)) · -**issue:** [#433](https://github.com/mudler/vllm.cpp/issues/433) · +**issues:** [#433](https://github.com/mudler/vllm.cpp/issues/433), +[#530](https://github.com/mudler/vllm.cpp/issues/530) · **claim:** `CLAIM-CPU-X86-FLOOR-1` · **base:** `31a2b493`. ## Scope @@ -289,3 +290,32 @@ presented G4's 3.38x/8.20x/2.29x as the live position. Both now point at **Out of scope and stated as such.** The Metal/MLX half of `ROAD-V1-D1` punch-list item 13 needs an Apple M4, which this host is not, and was not touched. + +### Follow-up repair contract: issue #530 + +The exact release-repair candidate +`6f5b98afd4019d6dca492e8ee1edc135163b9d5e` passed the full operator gate, +then exposed a nondeterministic defect while the required handoff gate was +being chained to its push. The smoke harness sets `BUSY_WINDOW=0` and +`QUIET_BUSY=100`; `busy_pct` separately opens `/proc/stat` for `stat_busy` and +`stat_total` at each side of that zero-length interval. On a contended host the +counter boundaries disagree, producing impossible readings including 110%. +`test_a_contended_leg_is_discarded_and_never_summarised` then exits 4 for no +quiet window instead of reaching its intended foreign-work discard and exit 2. +The same immutable tree passed when scheduling happened not to expose the split +snapshot, and failed twice when chained to the push. + +Repair each sample, not the threshold. One read of the aggregate `cpu` line +must derive both busy and total counters, and both `busy_pct` and `run_leg` must +consume paired samples. Preserve which fields count as busy, the fresh-window +policy, builder detection, `QUIET_BUSY`, `FOREIGN_MAX`, discard semantics, +evidence format, and every accepted benchmark value. Do not clamp the result, +raise a ceiling, add a retry to the unit test, or make contention disappear. + +RED-first coverage must reject the current split `stat_busy`/`stat_total` +reads and require both call sites to use one paired snapshot. Mutation must +restore a split sample independently in `busy_pct` and `run_leg`; each must +make the focused contract red. Focused green is the complete +`test_cpu_x86_llamacpp_floor.py` suite under current host contention. The fresh +reviewer mutates both call sites, the operator runs the full preflight, and an +unchanged exact SHA must pass that gate immediately before the plain push. From 41c00ac80f06742b6cbffbc1961967d2f32ac926 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 20:58:31 +0000 Subject: [PATCH 09/35] fix(gate): pair proc stat samples (#530) Derive busy and total jiffies from one aggregate CPU-line read for each quiet-window and benchmark-leg snapshot. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .../cpu-llamacpp-floor-x86-2026-08-11.md | 16 ++++++++++ scripts/cpu-x86-llamacpp-floor.sh | 13 ++++---- tests/scripts/test_cpu_x86_llamacpp_floor.py | 30 +++++++++++++++++++ 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md b/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md index 4db9fd545..a190db78f 100644 --- a/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md +++ b/.agents/specs/cpu-llamacpp-floor-x86-2026-08-11.md @@ -319,3 +319,19 @@ make the focused contract red. Focused green is the complete `test_cpu_x86_llamacpp_floor.py` suite under current host contention. The fresh reviewer mutates both call sites, the operator runs the full preflight, and an unchanged exact SHA must pass that gate immediately before the plain push. + +The #530 implementation replaces the separate `stat_busy` and `stat_total` +readers with one `stat_sample` reader. A single `awk` invocation derives the +unchanged busy-field sum and total-field sum from the same aggregate `cpu` line, +and both `busy_pct` and `run_leg` read paired before/after samples. No threshold, +clamp, retry, discard rule, evidence format, or benchmark value changed. + +RED-first structural coverage rejected the split reader functions and call +sites. The pre-edit full preflight also reproduced the operational failure: +`test_a_contended_leg_is_discarded_and_never_summarised` observed impossible +`busy=110%` with `BUSY_WINDOW=0` and returned 4 instead of its intended 2. +After the repair, the complete focused suite passed 11/11 under the same host +contention. Independently splitting the initial snapshot in `busy_pct` and in +`run_leg` made the focused structural contract fail for the mutated consumer; +the script was restored byte-for-byte after both mutations. The full unstaged +repository preflight then passed, including the complete CPU floor suite. diff --git a/scripts/cpu-x86-llamacpp-floor.sh b/scripts/cpu-x86-llamacpp-floor.sh index bd7a51925..029cdd4c0 100755 --- a/scripts/cpu-x86-llamacpp-floor.sh +++ b/scripts/cpu-x86-llamacpp-floor.sh @@ -97,17 +97,18 @@ builders() { # "busy" and "total" jiffies across all cpus. iowait is NOT counted as busy: a # neighbour waiting on disk does not steal our cores. steal IS counted -- on a # KVM guest that is exactly the co-tenant this box keeps losing series to. -stat_busy() { awk '/^cpu /{print $2+$3+$4+$7+$8+$9}' /proc/stat; } -stat_total() { awk '/^cpu /{s=0; for(i=2;i<=NF;i++)s+=$i; print s}' /proc/stat; } +stat_sample() { + awk '/^cpu /{busy=$2+$3+$4+$7+$8+$9; for(i=2;i<=NF;i++) total+=$i; print busy, total}' /proc/stat +} # Percent of the whole machine busy over a fresh BUSY_WINDOW. Nothing here is # decayed and nothing here counts a process that has already exited, so the # harness cannot gate on its own previous leg. busy_pct() { local b0 t0 b1 t1 db dt - b0=$(stat_busy); t0=$(stat_total) + read -r b0 t0 < <(stat_sample) sleep "$BUSY_WINDOW" - b1=$(stat_busy); t1=$(stat_total) + read -r b1 t1 < <(stat_sample) db=$((b1 - b0)); dt=$((t1 - t0)) if [ "$dt" -le 0 ]; then echo 100; return; fi echo $((100 * db / dt)) @@ -154,7 +155,7 @@ run_leg() { # engine rep -> 0 accepted, 1 discard echo "before builders: $(builders)" } > "$stem.load" echo "$eng rep=$rep START load=$(loadall) builders=$(builders)" - b0=$(stat_busy); t0=$(stat_total) + read -r b0 t0 < <(stat_sample) if [ "$eng" = ours ]; then # shellcheck disable=SC2086 $TIMEV $TASKSET env VLLM_CPP_CPU_THREADS=$T \ @@ -171,7 +172,7 @@ run_leg() { # engine rep -> 0 accepted, 1 discard rc=$? own=$(own_cpu_jiffies "$OUT/llama-bench-$rep.time") fi - b1=$(stat_busy); t1=$(stat_total) + read -r b1 t1 < <(stat_sample) bafter=$(builders) # Everything the machine burned while our leg ran, minus what our leg burned, # as a share of the machine. This is the check the old post-leg test did not diff --git a/tests/scripts/test_cpu_x86_llamacpp_floor.py b/tests/scripts/test_cpu_x86_llamacpp_floor.py index 358927d17..f7283ae47 100644 --- a/tests/scripts/test_cpu_x86_llamacpp_floor.py +++ b/tests/scripts/test_cpu_x86_llamacpp_floor.py @@ -118,6 +118,36 @@ def run_harness( timeout=300, ) + def test_proc_stat_samples_pair_busy_and_total_from_one_read(self) -> None: + script = SCRIPT.read_text() + + def function(name: str) -> str: + match = re.search(rf"(?ms)^{re.escape(name)}\(\).*?^\}}$", script) + self.assertIsNotNone(match, f"{name} function is missing") + assert match is not None + return match.group(0) + + self.assertNotRegex(script, r"(?m)^stat_(?:busy|total)\(\)") + sample = function("stat_sample") + self.assertEqual(sample.count("/proc/stat"), 1) + self.assertIn("busy=$2+$3+$4+$7+$8+$9", sample) + self.assertIn("for(i=2;i<=NF;i++) total+=$i", sample) + self.assertIn("print busy, total", sample) + + for consumer in ("busy_pct", "run_leg"): + body = function(consumer) + with self.subTest(consumer=consumer): + self.assertEqual( + len( + re.findall( + r"(?m)^\s*read -r b[01] t[01] < <\(stat_sample\)$", + body, + ) + ), + 2, + ) + self.assertNotRegex(body, r"\bstat_(?:busy|total)\b") + def test_runs_to_completion_and_creates_its_own_output_dir(self) -> None: out = self.tmp / "nested" / "evi" # does not exist: the shipped bug got = self.run_harness(out) From 74cc0c480d13c83bb31caca34ff64eb87966068d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 21:49:35 +0000 Subject: [PATCH 10/35] spec(windows): bind socket reset repair (#537) Record the hosted fast-fail, the upstream accepted-socket drain fix, scoped server-thread teardown, RED-first mutations, and immutable release acceptance gates before implementation. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 74 +++++++++++++++++-- 2 files changed, 70 insertions(+), 5 deletions(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 4ccdf1d02..9cc94b7ab 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -82,6 +82,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#512](https://github.com/mudler/vllm.cpp/issues/512) | `ENG-RELEASE-WINDOWS` | Windows release runner rejects empty process arguments | bug | | [#514](https://github.com/mudler/vllm.cpp/issues/514) | `ENG-RELEASE-WINDOWS` | Windows Vulkan release test uses POSIX environment APIs | bug | | [#525](https://github.com/mudler/vllm.cpp/issues/525) | `ENG-RELEASE-WINDOWS` | Windows release contract recorder misreports non-empty argv | bug | +| [#537](https://github.com/mudler/vllm.cpp/issues/537) | `ENG-RELEASE-WINDOWS` | Windows OpenAI socket smoke fast-fails instead of reporting a reset response | bug | | [#170](https://github.com/mudler/vllm.cpp/issues/170) | `ENG-RELEASE-BINARIES` | Publish container images to GHCR (cuda, vulkan, cpu) | feature | | [#322](https://github.com/mudler/vllm.cpp/issues/322) | `ENG-RELEASE-BINARIES` | Release handoff collides with tracked checkout `assets` directory | bug | | [#406](https://github.com/mudler/vllm.cpp/issues/406) | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate fails on how commits LAND: GitHub's Co-authored-by displaces the trailer block | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index a3b60521c..71ce9516c 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -6,8 +6,9 @@ Issues: [#499](https://github.com/mudler/vllm.cpp/issues/499) and [#500](https://github.com/mudler/vllm.cpp/issues/500), with post-merge follow-up [#512](https://github.com/mudler/vllm.cpp/issues/512) and -[#514](https://github.com/mudler/vllm.cpp/issues/514), and hosted-contract follow-up -[#525](https://github.com/mudler/vllm.cpp/issues/525) +[#514](https://github.com/mudler/vllm.cpp/issues/514), hosted-contract follow-up +[#525](https://github.com/mudler/vllm.cpp/issues/525), and native socket-runtime +follow-up [#537](https://github.com/mudler/vllm.cpp/issues/537) Parent specifications: [release-binary-matrix.md](release-binary-matrix.md) and @@ -16,10 +17,10 @@ Parent specifications: Related native compiler contract: [windows-msvc-strict-build.md](windows-msvc-strict-build.md) -Status: `READY`. The developer approved this bounded design on 2026-08-12. It +Status: `ACTIVE`. The developer approved this bounded design on 2026-08-12. It starts from exact main commit `e1087a8812c9b7d96fca5a813981f378fcace638` -after PR #446 merged. Implementation, a fresh review, the operator gate, and a -new non-publishing ten-tuple dry run remain required. +after PR #446 merged. The #537 implementation, a fresh review, the operator +gate, and a new non-publishing ten-tuple dry run remain required. ## Scope @@ -434,3 +435,66 @@ The Windows metadata and release pipeline suites passed 50/50, followed by the direct Windows portability and release-binary checkers and the full unstaged repository preflight. The local host has no PowerShell runtime, so native execution remains a required hosted Windows gate. + +### Native socket-runtime follow-up: issue #537 + +PR #524 candidate `d47d8408ab4dc2639e47ddfc7c7a997fa45d9981` +executed the complete #525 PowerShell contract successfully on hosted Windows. +CPU job +[`94265239385`](https://github.com/mudler/vllm.cpp/actions/runs/31641616323/job/94265239385) +then built `test_openai_api_server.exe` under unchanged `/W4 /WX`, served the +final request in the first real-socket smoke test, and terminated with signed +status `-1073740791` (`0xC0000409`) before doctest printed an assertion or +summary. + +The failure has two independently grounded causes. First, vendored +cpp-httplib v0.49.0 ends `process_and_close_socket` with an immediate +shutdown/close. Upstream commit `8e702d3837b2164765ca1d98cb6d180ae4711e70` +records that on Windows this can send an abortive RST when bytes are still in +flight, making a fully written response appear to the client as a failed read. +Its accepted repair half-closes the write side, drains at most 100 ms or 1 MiB, +then performs final shutdown/close. Second, each local real-socket test owns a +raw joinable `std::thread`, contains aborting `REQUIRE` assertions after that +thread starts, and stops/joins only on the normal path. When a Windows client +read is reset, doctest unwinds through the joinable thread and `std::terminate` +turns the useful assertion into the observed process-wide fast-fail. Upstream +commit `ae8356d86eabfd3ad4a969b55266fb3ecc2aa834` documents and repairs that exact +test-lifetime pattern with scoped teardown. + +The #537 implementation is limited to both complete repairs. Backport the +upstream accepted-socket drain helper and its single production call site with +the upstream 100 ms and 1 MiB bounds unchanged. Add one test-local scoped +server-thread owner and route every real-socket OpenAI test through it so +server stop and thread join happen both normally and while an assertion +unwinds. Preserve every existing request, status, body, concurrency, capacity, +route-gating, and shutdown assertion. Do not skip the Windows runtime target, +weaken `REQUIRE`, catch and discard a test failure, lengthen a timeout, detach a +thread, change release topology, or vendor unrelated upstream cpp-httplib +changes. + +RED-first evidence must demonstrate both defects against the pinned candidate. +A focused behavioral test must make the accepted-socket peer leave unread +request data and prove the old immediate close presents as a reset/failure on +Windows while the bounded drain preserves the completed response. A separate +test must intentionally fail after a server thread starts and prove scoped +teardown lets doctest report the assertion instead of terminating the process. +Structural coverage over the real vendored header and socket tests supplements, +but does not replace, those native behaviors. Mutations that restore the old +immediate accepted-socket close, remove either drain bound, or remove the scoped +stop/join path must make the focused gates red, with the tree restored +byte-for-byte afterwards. + +Focused green requires the OpenAI API server test, the cpp-httplib regression +test, the Windows release metadata/pipeline contracts, both direct Windows +checkers, and full unstaged/staged/post-commit preflight. Hosted acceptance +requires both Windows CPU and Vulkan lanes to execute the full OpenAI API test, +all other PR jobs to pass at one immutable head, and a fresh reviewer to mutate +both guarantees. After merge, the exact merged SHA must pass all ten release +tuples plus aggregate build and verify before `v0.0.3-pre.1` is tagged. The tag +run must then pass all 15 required jobs and the authenticated audit over exactly +32 assets. + +Stop with `NEEDS_DECISION` if the bounded upstream backport changes the public +HTTP API, broadens beyond accepted-socket close, or requires any release-gate +waiver. Stop with `NEEDS_CONTEXT` if Vulkan reports a different native failure; +file and specify that defect separately rather than folding it into #537. From 3b83dfe32b88d654533309d5f2849b0310ee4ad4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 21:53:51 +0000 Subject: [PATCH 11/35] spec(windows): bind shadow warning repair (#540) Record the hosted C4456 failures, test-only rename scope, six independent mutations, and native MSVC acceptance before implementation. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 44 ++++++++++++++++++- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 9cc94b7ab..16867955d 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -83,6 +83,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#514](https://github.com/mudler/vllm.cpp/issues/514) | `ENG-RELEASE-WINDOWS` | Windows Vulkan release test uses POSIX environment APIs | bug | | [#525](https://github.com/mudler/vllm.cpp/issues/525) | `ENG-RELEASE-WINDOWS` | Windows release contract recorder misreports non-empty argv | bug | | [#537](https://github.com/mudler/vllm.cpp/issues/537) | `ENG-RELEASE-WINDOWS` | Windows OpenAI socket smoke fast-fails instead of reporting a reset response | bug | +| [#540](https://github.com/mudler/vllm.cpp/issues/540) | `ENG-RELEASE-WINDOWS` | Windows Vulkan strict build rejects shadowed KV-cache test locals | bug | | [#170](https://github.com/mudler/vllm.cpp/issues/170) | `ENG-RELEASE-BINARIES` | Publish container images to GHCR (cuda, vulkan, cpu) | feature | | [#322](https://github.com/mudler/vllm.cpp/issues/322) | `ENG-RELEASE-BINARIES` | Release handoff collides with tracked checkout `assets` directory | bug | | [#406](https://github.com/mudler/vllm.cpp/issues/406) | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate fails on how commits LAND: GitHub's Co-authored-by displaces the trailer block | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 71ce9516c..66a28d79f 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -7,8 +7,9 @@ Issues: [#500](https://github.com/mudler/vllm.cpp/issues/500), with post-merge follow-up [#512](https://github.com/mudler/vllm.cpp/issues/512) and [#514](https://github.com/mudler/vllm.cpp/issues/514), hosted-contract follow-up -[#525](https://github.com/mudler/vllm.cpp/issues/525), and native socket-runtime -follow-up [#537](https://github.com/mudler/vllm.cpp/issues/537) +[#525](https://github.com/mudler/vllm.cpp/issues/525), native socket-runtime +follow-up [#537](https://github.com/mudler/vllm.cpp/issues/537), and strict +Vulkan-test follow-up [#540](https://github.com/mudler/vllm.cpp/issues/540) Parent specifications: [release-binary-matrix.md](release-binary-matrix.md) and @@ -498,3 +499,42 @@ Stop with `NEEDS_DECISION` if the bounded upstream backport changes the public HTTP API, broadens beyond accepted-socket close, or requires any release-gate waiver. Stop with `NEEDS_CONTEXT` if Vulkan reports a different native failure; file and specify that defect separately rather than folding it into #537. + +### Strict Vulkan-test follow-up: issue #540 + +The same immutable PR #524 candidate reached a different boundary in hosted +Windows Vulkan job +[`94265239433`](https://github.com/mudler/vllm.cpp/actions/runs/31641616323/job/94265239433). +The PowerShell contract and prior Windows portability fixes passed; MSVC then +rejected `tests/vt/test_backend_cross_device.cpp:525-529` under unchanged +`/W4 /WX`. The unbound-flash-layout CPU-oracle block redeclares `cpu`, `cq`, +`cd`, `ck`, `cv`, and `cslots`, shadowing names at lines 462-466 in the +enclosing `ReshapeAndCache` test. MSVC C4456 diagnoses all six. Git history +grounds the collision: `2c86f79ec` added the enclosing oracle and `822b3a2e15` +later added the nested oracle with the same short names. + +The #540 repair is test-only. Rename exactly those six inner declarations to +role-specific unbound-layout names and update only their uses in that nested +oracle. Preserve the data, types, tensor shapes and strides, CPU operation, +device loop, memcmp assertions, `/W4 /WX`, build targets, release topology, and +all production files. Do not suppress C4456, relax the warning gate, introduce +a compiler conditional, remove the nested oracle, or conflate this diagnostic +with #537. + +RED-first evidence is the hosted MSVC diagnostic above plus a focused local +contract over the real `ReshapeAndCache` test that rejects each of the six +shadowing inner declarations. Each independent mutation restoring one old name +must fail the focused contract, and the source must be restored byte-for-byte. +Focused green requires that contract, the direct Windows portability checker, +a clean local CPU compile/execution of `test_backend_cross_device`, and full +unstaged/staged/post-commit preflight. Hosted acceptance requires both native +Windows lanes to compile their complete unchanged targets; Vulkan must continue +through runtime, package, and archive validation. + +After both #537 and #540 pass fresh review and operator gates, PR #524 must be +plain-pushed at one exact SHA and all required PR checks must pass. The merged +SHA must then pass the complete ten-tuple non-publishing workflow before the +prerelease tag. Tag-run publication and the authenticated exactly-32-asset +audit remain mandatory. Stop with `NEEDS_DECISION` if a production edit or +warning-policy change is required; otherwise this is a bounded test hygiene +repair. From 17f78a01e6264af2c67c094ab2d474825ce95c5f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 22:20:44 +0000 Subject: [PATCH 12/35] fix(windows): drain accepted sockets (#537) Windows can reset a completed response when unread peer bytes remain. The reset then unwinds through joinable test threads and aborts the suite. Drain accepted sockets within upstream bounds. Give each real-socket test scoped server ownership so doctest can report fatal assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 27 ++++++++ tests/CMakeLists.txt | 22 +++++++ tests/scripts/run_expected_doctest_failure.py | 36 +++++++++++ .../scripts/test_check_windows_portability.py | 57 +++++++++++++++++ .../entrypoints/openai/scoped_server_thread.h | 32 ++++++++++ .../entrypoints/openai/test_api_server.cpp | 62 ++++++++----------- .../test_httplib_accepted_socket_close.cpp | 57 +++++++++++++++++ .../test_server_thread_failure_fixture.cpp | 36 +++++++++++ third_party/httplib/httplib.h | 36 ++++++++++- 9 files changed, 327 insertions(+), 38 deletions(-) create mode 100644 tests/scripts/run_expected_doctest_failure.py create mode 100644 tests/vllm/entrypoints/openai/scoped_server_thread.h create mode 100644 tests/vllm/entrypoints/openai/test_httplib_accepted_socket_close.cpp create mode 100644 tests/vllm/entrypoints/openai/test_server_thread_failure_fixture.cpp diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 66a28d79f..da4abc643 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -538,3 +538,30 @@ prerelease tag. Tag-run publication and the authenticated exactly-32-asset audit remain mandatory. Stop with `NEEDS_DECISION` if a production edit or warning-policy change is required; otherwise this is a bounded test hygiene repair. + +#### #537 implementation outcome + +The implementation backports only cpp-httplib +`8e702d3837b2164765ca1d98cb6d180ae4711e70`: accepted plain HTTP sockets now +half-close writes, drain for at most 100 ms or 1 MiB, and then perform the +existing final shutdown and close. The helper remains internal to the vendored +header and has exactly one production call site. No public HTTP API, TLS close +path, timeout, or release topology changed. + +All 12 real-socket cases in `test_openai_api_server.cpp` now use one test-local +scoped owner that stops the server and joins its thread on normal return and +assertion unwind. A deliberately failing doctest fixture must exit with +doctest's normal status 1, print the named assertion, and emit its failure +summary; a subprocess harness rejects termination or any other exit status. +The accepted-socket regression leaves 64 KiB of unread request data after a +complete close-delimited request and requires the complete response followed by +an orderly EOF. + +The focused transport tests, the full OpenAI API server test, Windows release +metadata, both direct Windows checkers, and the 74-test portability suite pass +locally. Mutations restoring immediate close, changing either drain bound, +removing scoped stop, and removing scoped join each made the focused gates red; +the last two respectively timed out and reproduced `SIGABRT`. The mutated files +were restored byte-for-byte. Linux also passes the socket behavior, but the +native reset RED and final acceptance remain correctly pending the hosted +Windows CPU and Vulkan lanes; this local result is not substituted for them. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 172a25903..6666556d1 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -798,6 +798,16 @@ target_compile_definitions(test_chat_template PRIVATE VLLM_TEST_FIXTURES_DIR="${CMAKE_SOURCE_DIR}/tests/fixtures") # The HTTP api_server smoke test needs the vendored cpp-httplib (gated). if(VLLM_CPP_SERVER) + add_executable(test_httplib_accepted_socket_close + vllm/entrypoints/openai/test_httplib_accepted_socket_close.cpp) + target_link_libraries(test_httplib_accepted_socket_close PRIVATE + vllm_test_main Threads::Threads) + if(WIN32) + target_link_libraries(test_httplib_accepted_socket_close PRIVATE ws2_32) + endif() + vllm_cpp_set_warnings(test_httplib_accepted_socket_close) + add_test(NAME test_httplib_accepted_socket_close + COMMAND test_httplib_accepted_socket_close) vllm_cpp_add_test(test_openai_api_server vllm/entrypoints/openai/test_api_server.cpp) # /v1/audio/transcriptions dispatch + socket smoke run against the REAL # library transcription seam on the committed parakeet_e2e fixture @@ -809,6 +819,18 @@ if(VLLM_CPP_SERVER) # (ARCH-ONE-SURFACE ROW 6). LLAMA_EMBED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/llama_embed_e2e") target_include_directories(test_openai_api_server PRIVATE ${CMAKE_SOURCE_DIR}/src) + add_executable(test_openai_server_thread_failure_fixture + vllm/entrypoints/openai/test_server_thread_failure_fixture.cpp) + target_link_libraries(test_openai_server_thread_failure_fixture PRIVATE + vllm_test_main Threads::Threads) + vllm_cpp_set_warnings(test_openai_server_thread_failure_fixture) + add_test(NAME test_openai_server_thread_failure_fixture + COMMAND ${Python3_EXECUTABLE} + "${CMAKE_SOURCE_DIR}/tests/scripts/run_expected_doctest_failure.py" + $ + "scoped teardown fixture assertion") + set_tests_properties(test_openai_server_thread_failure_fixture PROPERTIES + TIMEOUT 10) # M3.6: the OpenAI server CONFORMANCE suite — the full API contract exercised # end to end over the REAL cpp-httplib server on an ephemeral port. vllm_cpp_add_test(test_openai_conformance vllm/entrypoints/openai/test_conformance.cpp) diff --git a/tests/scripts/run_expected_doctest_failure.py b/tests/scripts/run_expected_doctest_failure.py new file mode 100644 index 000000000..7ae546228 --- /dev/null +++ b/tests/scripts/run_expected_doctest_failure.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""Require one intentional doctest assertion failure without process abort.""" + +from __future__ import annotations + +import subprocess +import sys + + +def main() -> int: + if len(sys.argv) != 3: + print("usage: run_expected_doctest_failure.py TEST EXPECTED_TEXT") + return 2 + + completed = subprocess.run( + [sys.argv[1]], + check=False, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + ) + print(completed.stdout, end="") + if completed.returncode != 1: + print(f"expected doctest exit 1, got {completed.returncode}") + return 1 + if sys.argv[2] not in completed.stdout: + print(f"missing expected doctest assertion: {sys.argv[2]}") + return 1 + if "[doctest] Status: FAILURE!" not in completed.stdout: + print("doctest failure summary was not emitted") + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/scripts/test_check_windows_portability.py b/tests/scripts/test_check_windows_portability.py index 9138ff44e..efe5d07a1 100644 --- a/tests/scripts/test_check_windows_portability.py +++ b/tests/scripts/test_check_windows_portability.py @@ -547,6 +547,63 @@ def test_windows_command_line_expectation_is_not_stringized(self) -> None: r"\(\s*argv\s*\)\s*==\s*expected\s*\)\s*;", ) + def test_accepted_server_socket_close_is_bounded_and_drained(self) -> None: + source = (REPO / "third_party/httplib/httplib.h").read_text( + encoding="utf-8" + ) + helper = re.search( + r"inline\s+void\s+drain_and_close_socket\s*\(.*?\n\}", + source, + re.DOTALL, + ) + self.assertIsNotNone(helper, "accepted-socket drain helper") + assert helper is not None + body = helper.group(0) + self.assertIn("shutdown(sock, SD_SEND)", body) + self.assertIn("shutdown(sock, SHUT_WR)", body) + self.assertRegex(body, r"milliseconds\s*\(\s*100\s*\)") + self.assertRegex(body, r"1024u\s*\*\s*1024u") + self.assertLess(body.index("shutdown_socket(sock)"), + body.index("close_socket(sock)")) + + process_close = re.search( + r"inline\s+bool\s+Server::process_and_close_socket\s*\(.*?\n\}", + source, + re.DOTALL, + ) + self.assertIsNotNone(process_close) + assert process_close is not None + self.assertEqual( + process_close.group(0).count( + "detail::drain_and_close_socket(sock);" + ), + 1, + ) + self.assertEqual( + source.count("detail::drain_and_close_socket(sock);"), 1 + ) + + def test_real_socket_tests_use_scoped_server_threads(self) -> None: + source = ( + REPO / "tests/vllm/entrypoints/openai/test_api_server.cpp" + ).read_text(encoding="utf-8") + owner = ( + REPO + / "tests/vllm/entrypoints/openai/scoped_server_thread.h" + ).read_text(encoding="utf-8") + + bound_servers = len( + re.findall(r"\.server\.bind_to_any_port\s*\(", source) + ) + scoped_threads = len( + re.findall(r"ScopedServerThread\s+server_thread\s*\(", source) + ) + self.assertGreater(bound_servers, 0) + self.assertEqual(scoped_threads, bound_servers) + self.assertNotRegex(source, r"std::thread\s+server_thread\b") + self.assertIn("stop_();", owner) + self.assertIn("if (thread_.joinable()) thread_.join();", owner) + def test_cpu_isa_test_owns_ostream_for_string_view_diagnostics(self) -> None: source = (REPO / "tests/vt/test_cpu_isa_x86.cpp").read_text( encoding="utf-8" diff --git a/tests/vllm/entrypoints/openai/scoped_server_thread.h b/tests/vllm/entrypoints/openai/scoped_server_thread.h new file mode 100644 index 000000000..950d97c92 --- /dev/null +++ b/tests/vllm/entrypoints/openai/scoped_server_thread.h @@ -0,0 +1,32 @@ +#pragma once + +#include +#include +#include + +namespace vllm::test { + +// Owns a test server's background thread. Fatal doctest assertions unwind the +// test body, so teardown must stop the server and join before std::thread's +// destructor observes a joinable thread. +class ScopedServerThread { + public: + template + ScopedServerThread(Start&& start, Stop&& stop) + : stop_(std::forward(stop)), + thread_(std::forward(start)) {} + + ~ScopedServerThread() { + stop_(); + if (thread_.joinable()) thread_.join(); + } + + ScopedServerThread(const ScopedServerThread&) = delete; + ScopedServerThread& operator=(const ScopedServerThread&) = delete; + + private: + std::function stop_; + std::thread thread_; +}; + +} // namespace vllm::test diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 8015de896..e2dfd6dff 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -13,6 +13,7 @@ #include "vllm/entrypoints/openai/api_server.h" #include "vllm/entrypoints/openai/video_api.h" #include "vllm/multimodal/parakeet_transcription.h" +#include "scoped_server_thread.h" #include @@ -110,6 +111,7 @@ using vllm::v1::OutputProcessor; using vllm::v1::Scheduler; using vllm::v1::sha256_cbor; using vt::DType; +using vllm::test::ScopedServerThread; namespace { @@ -1238,7 +1240,8 @@ TEST_CASE("api_server: socket smoke — real HTTP requests over an ephemeral por const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); // Wait until the accept loop is up. for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); @@ -1297,8 +1300,6 @@ TEST_CASE("api_server: socket smoke — real HTTP requests over an ephemeral por CHECK(j.at("choices").at(0).at("message").at("role") == "assistant"); } - h.server.stop(); - server_thread.join(); } // Route-registration gate over a real socket: /tokenizer_info is ABSENT (404) @@ -1317,7 +1318,8 @@ TEST_CASE("api_server: /tokenizer_info + /abort_requests are opt-in routes") { h.server.set_tokenizer(&Fixture(), kMaxModelLen); const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -1332,8 +1334,6 @@ TEST_CASE("api_server: /tokenizer_info + /abort_requests are opt-in routes") { REQUIRE(abort); CHECK(abort->status == 404); // no callback → route not registered - h.server.stop(); - server_thread.join(); } SUBCASE("backings attached → routes serve (200)") { @@ -1348,7 +1348,8 @@ TEST_CASE("api_server: /tokenizer_info + /abort_requests are opt-in routes") { }); const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -1367,8 +1368,6 @@ TEST_CASE("api_server: /tokenizer_info + /abort_requests are opt-in routes") { CHECK(json::parse(abort->body).at("aborted") == 2); CHECK(aborted_calls == 1); - h.server.stop(); - server_thread.join(); } } @@ -1387,15 +1386,14 @@ TEST_CASE("api_server: ConfigureUtilityEndpoints wires the production C8 surface auto with_server = [](ServerHarness& h, auto&& body) { const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); httplib::Client client("127.0.0.1", port); client.set_read_timeout(5, 0); body(client); - h.server.stop(); - server_thread.join(); }; // RED: a default production server WITHOUT the wiring seam 404s every C8 route, @@ -1529,7 +1527,8 @@ TEST_CASE("api_server: concurrent requests share AsyncLLM without state races") const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -1569,8 +1568,6 @@ TEST_CASE("api_server: concurrent requests share AsyncLLM without state races") for (int i = 1; i < kClients; ++i) CHECK(texts[static_cast(i)] == texts[0]); - h.server.stop(); - server_thread.join(); } TEST_CASE("api_server: configured persistent-stream capacity remains readable") { @@ -1584,7 +1581,8 @@ TEST_CASE("api_server: configured persistent-stream capacity remains readable") kStreamCapacity + ApiServer::kControlWorkerHeadroom); const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -1616,8 +1614,6 @@ TEST_CASE("api_server: configured persistent-stream capacity remains readable") CHECK(response->status == 200); parked.clear(); - h.server.stop(); - server_thread.join(); } TEST_CASE("api_server: stream capacity must be positive") { @@ -1657,7 +1653,8 @@ TEST_CASE( const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -1695,8 +1692,6 @@ TEST_CASE( CHECK(nodelay == 1); // RED until ApiServer calls set_tcp_nodelay(true) ::close(client_fd); - h.server.stop(); - server_thread.join(); #endif // defined(__linux__) } @@ -2096,15 +2091,14 @@ TEST_CASE("api_server: the /v1/videos routes do not exist without a runner") { auto with_server = [](ServerHarness& h, auto&& body) { const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); httplib::Client client("127.0.0.1", port); client.set_read_timeout(5, 0); body(client); - h.server.stop(); - server_thread.join(); }; SUBCASE("no runner: every video route 404s, and the core routes are unaffected") { @@ -2262,7 +2256,8 @@ TEST_CASE("api_server: transcriptions socket smoke (multipart), generate routes AsrHarness h; const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -2309,8 +2304,6 @@ TEST_CASE("api_server: transcriptions socket smoke (multipart), generate routes "parakeet-fixture"); } - h.server.stop(); - server_thread.join(); } TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { @@ -2328,7 +2321,8 @@ TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -2361,8 +2355,6 @@ TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { CHECK(health->status == 200); } - h.server.stop(); - server_thread.join(); } // ─── ARCH-ONE-SURFACE ROW 8: the server's --device seam ────────────────────── @@ -2516,7 +2508,8 @@ TEST_CASE("api_server: embeddings socket smoke; generate routes 404 on the " EmbedHarness h; const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -2552,8 +2545,6 @@ TEST_CASE("api_server: embeddings socket smoke; generate routes 404 on the " "llama-embed-fixture"); } - h.server.stop(); - server_thread.join(); } TEST_CASE("api_server: /v1/embeddings does not exist on a TEXT server") { @@ -2569,7 +2560,8 @@ TEST_CASE("api_server: /v1/embeddings does not exist on a TEXT server") { const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); - std::thread server_thread([&h]() { h.server.serve(); }); + ScopedServerThread server_thread([&h]() { h.server.serve(); }, + [&h]() { h.server.stop(); }); for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); @@ -2584,8 +2576,6 @@ TEST_CASE("api_server: /v1/embeddings does not exist on a TEXT server") { CHECK(res->status == 404); } - h.server.stop(); - server_thread.join(); } TEST_CASE("platform process: Windows command line preserves every argv byte") { diff --git a/tests/vllm/entrypoints/openai/test_httplib_accepted_socket_close.cpp b/tests/vllm/entrypoints/openai/test_httplib_accepted_socket_close.cpp new file mode 100644 index 000000000..3e53d4cda --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_httplib_accepted_socket_close.cpp @@ -0,0 +1,57 @@ +#include + +#include +#include +#include + +#include + +TEST_CASE("cpp-httplib: accepted socket drains unread peer bytes before close") { + httplib::Server server; + server.Get("/complete", [](const httplib::Request&, httplib::Response& res) { + res.set_content("completed-response", "text/plain"); + }); + + const int port = server.bind_to_any_port("127.0.0.1"); + REQUIRE(port > 0); + std::thread server_thread([&server]() { server.listen_after_bind(); }); + auto cleanup = httplib::detail::scope_exit([&]() { + server.stop(); + if (server_thread.joinable()) server_thread.join(); + }); + server.wait_until_ready(); + + httplib::Error error = httplib::Error::Success; + const socket_t sock = httplib::detail::create_client_socket( + "127.0.0.1", "", port, AF_UNSPEC, false, false, nullptr, + /*connection_timeout_sec=*/5, /*connection_timeout_usec=*/0, + /*read_timeout_sec=*/5, /*read_timeout_usec=*/0, + /*write_timeout_sec=*/5, /*write_timeout_usec=*/0, "", error); + REQUIRE(sock != INVALID_SOCKET); + auto close_client = httplib::detail::scope_exit( + [&]() { httplib::detail::close_socket(sock); }); + + std::string request = + "GET /complete HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n"; + request.append(64 * 1024, 'x'); + size_t sent = 0; + while (sent < request.size()) { + const auto n = httplib::detail::send_socket( + sock, request.data() + sent, request.size() - sent, + CPPHTTPLIB_SEND_FLAGS); + REQUIRE(n > 0); + sent += static_cast(n); + } + + std::string response; + char buffer[4096]; + ssize_t received = 0; + while ((received = httplib::detail::read_socket( + sock, buffer, sizeof(buffer), CPPHTTPLIB_RECV_FLAGS)) > 0) { + response.append(buffer, static_cast(received)); + } + + CHECK(response.find("completed-response") != std::string::npos); + CHECK_MESSAGE(received == 0, + "accepted-socket close reset the completed response"); +} diff --git a/tests/vllm/entrypoints/openai/test_server_thread_failure_fixture.cpp b/tests/vllm/entrypoints/openai/test_server_thread_failure_fixture.cpp new file mode 100644 index 000000000..1b6c9625c --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_server_thread_failure_fixture.cpp @@ -0,0 +1,36 @@ +#include "scoped_server_thread.h" + +#include + +#include +#include +#include + +using vllm::test::ScopedServerThread; + +TEST_CASE("scoped server thread reports assertion failures without terminating") { + std::mutex mutex; + std::condition_variable changed; + bool running = false; + bool stopped = false; + + ScopedServerThread server_thread( + [&]() { + std::unique_lock lock(mutex); + running = true; + changed.notify_all(); + changed.wait(lock, [&]() { return stopped; }); + }, + [&]() { + std::lock_guard lock(mutex); + stopped = true; + changed.notify_all(); + }); + + { + std::unique_lock lock(mutex); + REQUIRE(changed.wait_for(lock, std::chrono::seconds(5), + [&]() { return running; })); + } + REQUIRE_MESSAGE(false, "scoped teardown fixture assertion"); +} diff --git a/third_party/httplib/httplib.h b/third_party/httplib/httplib.h index 90498af2b..9eb3162c0 100644 --- a/third_party/httplib/httplib.h +++ b/third_party/httplib/httplib.h @@ -5820,6 +5820,39 @@ inline int shutdown_socket(socket_t sock) noexcept { #endif } +// Half-close the write side and drain any in-flight/queued bytes before the +// final shutdown+close. Closing with unread data in the receive queue, or with +// bytes arriving after the receive side is closed, can make the stack send an +// abortive RST and make a fully written response appear to have failed. +// Backported from cpp-httplib 8e702d3837b2164765ca1d98cb6d180ae4711e70. +inline void drain_and_close_socket(socket_t sock) noexcept { +#ifdef _WIN32 + shutdown(sock, SD_SEND); +#else + shutdown(sock, SHUT_WR); +#endif + + char buf[CPPHTTPLIB_RECV_BUFSIZ]; + size_t total = 0; + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(100); // bound #1 + + while (total < size_t(1024u * 1024u)) { // bound #2 + const auto remaining = + std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()) + .count(); + if (remaining <= 0) { break; } + if (select_read(sock, 0, static_cast(remaining)) <= 0) { break; } + const auto n = read_socket(sock, buf, sizeof(buf), CPPHTTPLIB_RECV_FLAGS); + if (n <= 0) { break; } + total += static_cast(n); + } + + shutdown_socket(sock); + close_socket(sock); +} + inline std::string escape_abstract_namespace_unix_domain(const std::string &s) { if (s.size() > 1 && s[0] == '\0') { auto ret = s; @@ -12686,8 +12719,7 @@ inline bool Server::process_and_close_socket(socket_t sock) { nullptr, &websocket_upgraded); }); - detail::shutdown_socket(sock); - detail::close_socket(sock); + detail::drain_and_close_socket(sock); return ret; } From 439d35b6abec6339655c7aeef27f0d679b235836 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 22:18:30 +0000 Subject: [PATCH 13/35] fix(tests): name unbound CPU oracle locals (#540) MSVC /W4 /WX rejects the nested unbound-flash CPU oracle because its short local names shadow the enclosing oracle. Give those six locals role-specific names and lock the contract with a focused portability test. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 12 +++++++ .../scripts/test_check_windows_portability.py | 32 +++++++++++++++++++ tests/vt/test_backend_cross_device.cpp | 26 ++++++++------- 3 files changed, 59 insertions(+), 11 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index da4abc643..81344fa1e 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -565,3 +565,15 @@ the last two respectively timed out and reproduced `SIGABRT`. The mutated files were restored byte-for-byte. Linux also passes the socket behavior, but the native reset RED and final acceptance remain correctly pending the hosted Windows CPU and Vulkan lanes; this local result is not substituted for them. + +#### #540 implementation outcome + +Implementation evidence: the focused contract was red with all six old +declarations present (12 subtest failures), then green after the nested locals +were renamed to `unbound_cpu`, `unbound_queue`, `unbound_device`, `unbound_k`, +`unbound_v`, and `unbound_slots`. Restoring each old declaration independently +made its named contract checks red, with both source files restored to their +recorded hashes afterwards. The 73-test Windows portability suite, direct +Windows portability checker, and a clean CPU-only Release build and execution +of `test_backend_cross_device` (19 cases, 6 assertions) passed. Native MSVC +acceptance remains pending the hosted Windows CPU and Vulkan lanes. diff --git a/tests/scripts/test_check_windows_portability.py b/tests/scripts/test_check_windows_portability.py index efe5d07a1..7b0dae9e8 100644 --- a/tests/scripts/test_check_windows_portability.py +++ b/tests/scripts/test_check_windows_portability.py @@ -619,6 +619,38 @@ def test_cpu_isa_test_owns_ostream_for_string_view_diagnostics(self) -> None: ) self.assertIn("std::string_view::npos", source) + def test_unbound_flash_oracle_names_do_not_shadow_enclosing_oracle(self) -> None: + source = (REPO / "tests/vt/test_backend_cross_device.cpp").read_text( + encoding="utf-8" + ) + start = source.index("// --- Unbind flash layout:") + end = source.index("for (DeviceType dt : RegisteredDevices())", start) + oracle = checker._cpp_structural_view(source[start:end]) + + declarations = { + "unbound_cpu": r"vt::Backend\s*&\s*unbound_cpu\s*=", + "unbound_queue": r"\bQueue\s+unbound_queue\s*=", + "unbound_device": r"\bDevice\s+unbound_device\s*\{", + "unbound_k": r"std::vector\s+unbound_k\s*=", + "unbound_v": r"\bunbound_v\s*=", + "unbound_slots": r"std::vector\s+unbound_slots\s*=", + } + for name, declaration in declarations.items(): + with self.subTest(name=name): + self.assertEqual(len(re.findall(declaration, oracle)), 1) + + old_declarations = ( + r"vt::Backend\s*&\s*cpu\s*=", + r"\bQueue\s+cq\s*=", + r"\bDevice\s+cd\s*\{", + r"std::vector\s+ck\s*=", + r"\bcv\s*=", + r"std::vector\s+cslots\s*=", + ) + for declaration in old_declarations: + with self.subTest(old_declaration=declaration): + self.assertNotRegex(oracle, declaration) + def test_cross_device_fused_tier_environment_is_windows_portable(self) -> None: source = (REPO / "tests/vt/test_backend_cross_device.cpp").read_text( encoding="utf-8" diff --git a/tests/vt/test_backend_cross_device.cpp b/tests/vt/test_backend_cross_device.cpp index bcfabfafe..f25eec088 100644 --- a/tests/vt/test_backend_cross_device.cpp +++ b/tests/vt/test_backend_cross_device.cpp @@ -522,15 +522,18 @@ TEST_CASE("ReshapeAndCache scatters into the KV cache BIT-EXACTLY") { } std::vector ref_comb = combined; { - vt::Backend& cpu = vt::GetBackend(DeviceType::kCPU); - Queue cq = cpu.CreateQueue(); - const Device cd{DeviceType::kCPU, 0}; - std::vector ck = knew, cv = vnew, cslots_f; - std::vector cslots = slots; - Tensor tk = Tensor::Contiguous(ck.data(), DType::kF32, cd, {kTokens, kHk, kD}); - Tensor tv = Tensor::Contiguous(cv.data(), DType::kF32, cd, {kTokens, kHk, kD}); + vt::Backend& unbound_cpu = vt::GetBackend(DeviceType::kCPU); + Queue unbound_queue = unbound_cpu.CreateQueue(); + const Device unbound_device{DeviceType::kCPU, 0}; + std::vector unbound_k = knew, unbound_v = vnew, cslots_f; + std::vector unbound_slots = slots; + Tensor tk = Tensor::Contiguous(unbound_k.data(), DType::kF32, unbound_device, + {kTokens, kHk, kD}); + Tensor tv = Tensor::Contiguous(unbound_v.data(), DType::kF32, unbound_device, + {kTokens, kHk, kD}); Tensor tcomb = - Tensor::Contiguous(ref_comb.data(), DType::kF32, cd, {kBlocks * 2 * within}); + Tensor::Contiguous(ref_comb.data(), DType::kF32, unbound_device, + {kBlocks * 2 * within}); auto slice = [&](int which) { Tensor t = tcomb; t.data = static_cast(t.data) + @@ -546,10 +549,11 @@ TEST_CASE("ReshapeAndCache scatters into the KV cache BIT-EXACTLY") { t.stride[3] = 1; return t; }; - Tensor tsm = Tensor::Contiguous(cslots.data(), DType::kI64, cd, {kTokens}); + Tensor tsm = Tensor::Contiguous(unbound_slots.data(), DType::kI64, unbound_device, + {kTokens}); Tensor tkc = slice(0), tvc = slice(1); - vt::ReshapeAndCache(cq, tk, tv, tkc, tvc, tsm); - cpu.DestroyQueue(cq); + vt::ReshapeAndCache(unbound_queue, tk, tv, tkc, tvc, tsm); + unbound_cpu.DestroyQueue(unbound_queue); } for (DeviceType dt : RegisteredDevices()) { From a0fb338386c06d66cf5d52780249ad7c881128f5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 22:43:10 +0000 Subject: [PATCH 14/35] fix(tests): parse unbound oracle declarators (#540) The prior contract matched six exact declaration spellings. Equivalent comma declarators could restore an enclosing-oracle name without making the contract fail. Derive names from the scoped declaration list while respecting initializer nesting. This keeps each forbidden name observable across assignment, brace, and parenthesis forms. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 9 +++ .../scripts/test_check_windows_portability.py | 68 ++++++++++++++++--- 2 files changed, 66 insertions(+), 11 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 81344fa1e..62bbc8960 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -577,3 +577,12 @@ recorded hashes afterwards. The 73-test Windows portability suite, direct Windows portability checker, and a clean CPU-only Release build and execution of `test_backend_cross_device` (19 cases, 6 assertions) passed. Native MSVC acceptance remains pending the hosted Windows CPU and Vulkan lanes. + +Fresh review found that the negative checks recognized only six exact +declaration spellings. The equivalent comma declarator +`const Device unbound_device{...}, cd(DeviceType::kCPU, 0);` therefore passed +the old contract. The follow-up contract derives declarator names within the +exact unbound oracle while respecting comma, brace, and parenthesis nesting. +That bypass and independent mutations for all six forbidden names now fail; +the restored tree passes all 74 Windows portability tests and the direct +checker. diff --git a/tests/scripts/test_check_windows_portability.py b/tests/scripts/test_check_windows_portability.py index 7b0dae9e8..ed3f432c3 100644 --- a/tests/scripts/test_check_windows_portability.py +++ b/tests/scripts/test_check_windows_portability.py @@ -33,6 +33,41 @@ ) +_ORACLE_DECLARATION_TYPE = re.compile( + r"\b(?:vt::Backend|Queue|(?:const\s+)?Device|" + r"std::vector\s*<\s*(?:float|int64_t)\s*>)\s*(?:[&*]\s*)?" +) + + +def _oracle_declarator_names(source: str) -> set[str]: + """Return names declared with the unbound oracle's relevant types.""" + names: set[str] = set() + for type_match in _ORACLE_DECLARATION_TYPE.finditer(source): + declarators = [] + start = type_match.end() + depths = {"(": 0, "[": 0, "{": 0} + closing = {")": "(", "]": "[", "}": "{"} + cursor = start + for index in range(start, len(source)): + char = source[index] + if char in depths: + depths[char] += 1 + elif char in closing: + opener = closing[char] + if depths[opener] > 0: + depths[opener] -= 1 + elif char in {",", ";"} and not any(depths.values()): + declarators.append(source[cursor:index]) + cursor = index + 1 + if char == ";": + break + for declarator in declarators: + match = re.match(r"\s*(?:[&*]\s*)*([A-Za-z_]\w*)", declarator) + if match is not None: + names.add(match.group(1)) + return names + + SAFE_FILES = { "CMakeLists.txt": """ set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$:Debug>") @@ -639,17 +674,28 @@ def test_unbound_flash_oracle_names_do_not_shadow_enclosing_oracle(self) -> None with self.subTest(name=name): self.assertEqual(len(re.findall(declaration, oracle)), 1) - old_declarations = ( - r"vt::Backend\s*&\s*cpu\s*=", - r"\bQueue\s+cq\s*=", - r"\bDevice\s+cd\s*\{", - r"std::vector\s+ck\s*=", - r"\bcv\s*=", - r"std::vector\s+cslots\s*=", - ) - for declaration in old_declarations: - with self.subTest(old_declaration=declaration): - self.assertNotRegex(oracle, declaration) + forbidden = {"cpu", "cq", "cd", "ck", "cv", "cslots"} + self.assertTrue( + forbidden.isdisjoint(_oracle_declarator_names(oracle)), + "unbound oracle redeclares an enclosing-oracle name", + ) + + def test_unbound_flash_oracle_declarator_scan_handles_initializer_forms( + self) -> None: + declarations = checker._cpp_structural_view(""" + vt::Backend& unbound_cpu = backend, &cpu(get_backend()); + Queue unbound_queue = make_queue(), cq{make_queue()}; + const Device unbound_device{DeviceType::kCPU, 0}, + cd(DeviceType::kCPU, 0); + std::vector unbound_k = input, ck{1.0f, 2.0f}, + unbound_v(input), cv = input; + std::vector unbound_slots = slots, cslots{0, 1}; + """) + self.assertEqual( + {"cpu", "cq", "cd", "ck", "cv", "cslots"}, + _oracle_declarator_names(declarations) + & {"cpu", "cq", "cd", "ck", "cv", "cslots"}, + ) def test_cross_device_fused_tier_environment_is_windows_portable(self) -> None: source = (REPO / "tests/vt/test_backend_cross_device.cpp").read_text( From 32d301a230bae870757ad1917b5d6ef418f9c054 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 07:34:20 +0000 Subject: [PATCH 15/35] docs(release): record Windows teardown falsification (#537) Record the immutable hosted evidence that the accepted-socket drain and scoped server thread did not eliminate the native fast-fail. Narrow the next probe to the three destruction boundaries without weakening the full gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 62bbc8960..375f2ae96 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -566,6 +566,35 @@ were restored byte-for-byte. Linux also passes the socket behavior, but the native reset RED and final acceptance remain correctly pending the hosted Windows CPU and Vulkan lanes; this local result is not substituted for them. +#### #537 hosted falsification and next diagnostic + +Hosted CPU job +[`94287249909`](https://github.com/mudler/vllm.cpp/actions/runs/31648432555/job/94287249909) +and Vulkan job +[`94287249981`](https://github.com/mudler/vllm.cpp/actions/runs/31648432555/job/94287249981) +executed candidate `f52b547d44439e7cfb005f5697142b755ff65586` with both +repairs above and still terminated with `0xC0000409`. The log and doctest's +source-order execution place the failure in teardown of +`api_server: socket smoke — real HTTP requests over an ephemeral port`: its +final chat response completes, but doctest never prints that test's duration or +summary. Therefore the accepted-socket reset plus unscoped server thread was a +real defect, but it was not the complete root cause of this native fast-fail. +That earlier causal claim is rejected; its implementation remains required by +its focused regressions. + +The next hosted probe must preserve the full unchanged test and release gate, +add phase evidence around destruction of the `httplib::Client`, scoped server +thread, and `ServerHarness`, and install a diagnostic `std::terminate` marker in +the test process. It must run the isolated socket-smoke case first with doctest +success/duration output and then retain the normal full-suite invocation if the +probe survives. The probe may only add diagnostics; it must not skip an +assertion, detach a thread, catch a failure, alter production lifetime, or be +accepted as the fix. Use the first missing teardown marker plus the terminate +marker to identify one owner, then remove the probe and write a RED regression +for that exact lifetime defect before implementing a repair. If the isolated +case passes but the full process fails, bisect test-case prefixes in fresh +processes to prove the cross-test state dependency rather than guessing. + #### #540 implementation outcome Implementation evidence: the focused contract was red with all six old From 5d15082c36c2f76d19bf74fb9700d3b72873d955 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 08:08:57 +0000 Subject: [PATCH 16/35] docs(release): specify portable PowerShell contract (#599) Track the direct-checker failure exposed by fresh review and bound the repair to a cross-platform real-process failure fixture with exact exit-status evidence. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 32 +++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 814064117..5654977be 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -104,6 +104,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#525](https://github.com/mudler/vllm.cpp/issues/525) | `ENG-RELEASE-WINDOWS` | Windows release contract recorder misreports non-empty argv | bug | | [#537](https://github.com/mudler/vllm.cpp/issues/537) | `ENG-RELEASE-WINDOWS` | Windows OpenAI socket smoke fast-fails instead of reporting a reset response | bug | | [#540](https://github.com/mudler/vllm.cpp/issues/540) | `ENG-RELEASE-WINDOWS` | Windows Vulkan strict build rejects shadowed KV-cache test locals | bug | +| [#599](https://github.com/mudler/vllm.cpp/issues/599) | `ENG-RELEASE-WINDOWS` | Windows release contract failure fixture is not portable under PowerShell on Linux | bug | | [#515](https://github.com/mudler/vllm.cpp/issues/515) | `ENG-RECORD-CONFLICT-SURFACES` | check-doc-checkpoint treats every root `CMakeLists.txt` edit as a usage change, so adding a source file demands a `docs/USAGE.md` edit with nothing true to say | bug | | [#547](https://github.com/mudler/vllm.cpp/issues/547) | — | GB10 reports `UnifiedMemory()` true, so `ReferenceTierEligible(kCUDA)` runs the CPU host kernel over `cudaMalloc` pointers; it needs `DeviceMemoryIsHostAddressable()` | bug | | [#569](https://github.com/mudler/vllm.cpp/issues/569) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | The Nemotron revision pin was existence-only: a substituted checkpoint still resolved, and W6's token gate cannot see that | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 375f2ae96..0a341a139 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -595,6 +595,38 @@ for that exact lifetime defect before implementing a repair. If the isolated case passes but the full process fails, bisect test-case prefixes in fresh processes to prove the cross-test state dependency rather than guessing. +### Cross-platform PowerShell contract follow-up: issue #599 + +Fresh review of diagnostic candidate `5a845c28a7a9afe5addf94771e59a71cecd31e81` +ran the required direct Windows portability checker on a Linux host with +PowerShell Core installed. `Invoke-CheckedContractTests` created `fail.cmd` and +invoked that path directly. Non-Windows PowerShell handed the batch file to +`gio` instead of a Windows command interpreter; `gio` reported the unusable +path but returned success, so the contract failed at its own guard with +`nonzero child exit was accepted`. The repository preflight did not substitute +for this direct checker and its green result is not evidence for this boundary. + +The #599 repair is contract-test only. Replace the platform-specific failing +child fixture with a script directly executable by both Windows PowerShell and +PowerShell Core on non-Windows hosts. Preserve the real-process invocation, +exact exit status 23, zero- and non-empty-argument forwarding proofs, production +`Invoke-Checked` behavior, native release gate, and release topology. Do not +mock this boundary, special-case the checker host, skip the nonzero child, or +accept a desktop-opener status. + +RED-first evidence is the direct checker failure above. Focused coverage must +pin the portable fixture and its exact exit 23, then run the actual +`-ContractTest` path under installed PowerShell Core. Mutating the fixture back +to `.cmd`, changing its exit status, or bypassing the real child invocation must +make the focused gate red. Focused green requires Windows release metadata and +pipeline tests, the complete Windows portability unit suite, the direct Windows +portability and release-binary checkers, and full unstaged/staged/post-commit +preflight. Hosted CPU and Vulkan execution remains required because local +PowerShell Core does not prove Windows process semantics. + +Stop with `NEEDS_DECISION` if the repair would change production invocation or +the native release gate rather than only its contract fixture. + #### #540 implementation outcome Implementation evidence: the focused contract was red with all six old From 9119c9bb7a2ac6843ab54e25f837ea03ac76eec4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 07:52:33 +0000 Subject: [PATCH 17/35] test(windows): probe OpenAI teardown fast-fail (#537) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- scripts/build-windows-release.ps1 | 7 ++ tests/doctest_main.cpp | 40 +++++++++++- .../scripts/test_release_windows_metadata.py | 65 +++++++++++++++++++ .../entrypoints/openai/test_api_server.cpp | 27 ++++++++ 4 files changed, 138 insertions(+), 1 deletion(-) diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 5fc8e34bb..005de80ee 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -285,6 +285,13 @@ if ($Backend -eq "vulkan") { } Invoke-Checked cmake (@("--build", $BuildDir, "--config", "Release", "--target") + $targets) +$openaiApiServerTest = Join-Path $BuildDir "tests/Release/test_openai_api_server.exe" +Invoke-Checked $openaiApiServerTest @( + "--test-case=api_server: socket smoke *real HTTP requests over an ephemeral port", + "--success=true", + "--duration=true" +) + foreach ($test in @( "test_openai_api_server.exe", "test_lmcache_client.exe", diff --git a/tests/doctest_main.cpp b/tests/doctest_main.cpp index 0a3f254ea..d4c4ef00c 100644 --- a/tests/doctest_main.cpp +++ b/tests/doctest_main.cpp @@ -1,2 +1,40 @@ -#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN +#define DOCTEST_CONFIG_IMPLEMENT #include + +#include +#include +#include + +namespace { + +[[noreturn]] void DiagnosticTerminate() noexcept { + const std::exception_ptr exception = std::current_exception(); + if (exception == nullptr) { + std::fputs( + "[vllm-test-probe] std::terminate current_exception=none\n", stderr); + } else { + std::fputs( + "[vllm-test-probe] std::terminate current_exception=present\n", + stderr); + try { + std::rethrow_exception(exception); + } catch (const std::exception& error) { + std::fprintf(stderr, + "[vllm-test-probe] std::terminate std::exception what=%s\n", + error.what()); + } catch (...) { + std::fputs( + "[vllm-test-probe] std::terminate exception=non-std-exception\n", + stderr); + } + } + std::fflush(stderr); + std::abort(); +} + +} // namespace + +int main(int argc, char** argv) { + std::set_terminate(DiagnosticTerminate); + return doctest::Context(argc, argv).run(); +} diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 5e5124c72..4e4aff876 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -100,6 +100,71 @@ def test_contract_test_precedes_required_release_environment(self) -> None: self.assertLess(contract, contract_exit) self.assertLess(contract_exit, environment) + def test_openai_socket_teardown_probe_precedes_unchanged_full_suite(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text( + encoding="utf-8" + ) + test_path = ( + '$openaiApiServerTest = Join-Path $BuildDir ' + '"tests/Release/test_openai_api_server.exe"' + ) + probe = ( + "Invoke-Checked $openaiApiServerTest @(\n" + ' "--test-case=api_server: socket smoke *real HTTP requests over an ephemeral port",\n' + ' "--success=true",\n' + ' "--duration=true"\n' + ")" + ) + full_suite = ( + 'foreach ($test in @(\n' + ' "test_openai_api_server.exe",' + ) + self.assertEqual(script.count(test_path), 1) + self.assertEqual(script.count(probe), 1) + self.assertEqual(script.count(full_suite), 1) + self.assertLess(script.index(test_path), script.index(probe)) + self.assertLess(script.index(probe), script.index(full_suite)) + + def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: + main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8") + for diagnostic in ( + "std::set_terminate", + "std::current_exception()", + "std::rethrow_exception", + "catch (const std::exception& error)", + "error.what()", + "std::abort()", + ): + with self.subTest(diagnostic=diagnostic): + self.assertEqual(main.count(diagnostic), 1) + + source = ( + ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" + ).read_text(encoding="utf-8") + start = source.index( + 'TEST_CASE("api_server: socket smoke — real HTTP requests over an ephemeral port")' + ) + end = source.index( + "// Route-registration gate over a real socket", start + ) + socket_smoke = source[start:end] + owners = ( + ("ServerHarness", "h"), + ("ScopedServerThread", "server_thread"), + ("httplib::Client", "client"), + ) + for owner, variable in owners: + with self.subTest(owner=owner): + completed = f'"[vllm-test-probe] {owner} destruction complete\\n"' + started = f'"[vllm-test-probe] {owner} destruction start\\n"' + owner_declaration = f"{owner} {variable}" + self.assertEqual(socket_smoke.count(completed), 1) + self.assertEqual(socket_smoke.count(started), 1) + self.assertLess(socket_smoke.index(completed), + socket_smoke.index(owner_declaration)) + self.assertLess(socket_smoke.index(owner_declaration), + socket_smoke.index(started)) + def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) -> None: script = (ROOT / "scripts/build-windows-release.ps1").read_text(encoding="utf-8") helper_start = script.index("function Invoke-Checked {") diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index e2dfd6dff..408669293 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -115,6 +115,21 @@ using vllm::test::ScopedServerThread; namespace { +class TeardownProbeMarker { + public: + explicit TeardownProbeMarker(const char* message) : message_(message) {} + ~TeardownProbeMarker() { + std::fputs(message_, stderr); + std::fflush(stderr); + } + + TeardownProbeMarker(const TeardownProbeMarker&) = delete; + TeardownProbeMarker& operator=(const TeardownProbeMarker&) = delete; + + private: + const char* message_; +}; + // ─── Synthetic weights (mirrors test_serving.cpp) ──────────────────────────── uint64_t Mix(uint64_t x) { x += 0x9E3779B97F4A7C15ULL; @@ -1236,18 +1251,30 @@ TEST_CASE("api_server: /abort_requests aborts an in-flight engine request") { TEST_CASE("api_server: socket smoke — real HTTP requests over an ephemeral port") { const HfConfig c = MakeConfig(); const Qwen3_5MoeWeights w = MakeWeights(c); + [[maybe_unused]] TeardownProbeMarker harness_destroyed( + "[vllm-test-probe] ServerHarness destruction complete\n"); ServerHarness h(c, w, Fixture()); + [[maybe_unused]] TeardownProbeMarker harness_destroying( + "[vllm-test-probe] ServerHarness destruction start\n"); const int port = h.server.bind_to_any_port("127.0.0.1"); REQUIRE(port > 0); + [[maybe_unused]] TeardownProbeMarker server_thread_destroyed( + "[vllm-test-probe] ScopedServerThread destruction complete\n"); ScopedServerThread server_thread([&h]() { h.server.serve(); }, [&h]() { h.server.stop(); }); + [[maybe_unused]] TeardownProbeMarker server_thread_destroying( + "[vllm-test-probe] ScopedServerThread destruction start\n"); // Wait until the accept loop is up. for (int i = 0; i < 500 && !h.server.is_running(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(2)); REQUIRE(h.server.is_running()); + [[maybe_unused]] TeardownProbeMarker client_destroyed( + "[vllm-test-probe] httplib::Client destruction complete\n"); httplib::Client client("127.0.0.1", port); + [[maybe_unused]] TeardownProbeMarker client_destroying( + "[vllm-test-probe] httplib::Client destruction start\n"); client.set_connection_timeout(5, 0); client.set_read_timeout(15, 0); From 31bca7a944f54f9ec336566e3b10fe36ede21281 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 08:12:37 +0000 Subject: [PATCH 18/35] test(windows): bind teardown probe metadata (#537) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .../scripts/test_release_windows_metadata.py | 50 ++++++++++++++++++- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 4e4aff876..23ebf81f1 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -119,16 +119,36 @@ def test_openai_socket_teardown_probe_precedes_unchanged_full_suite(self) -> Non 'foreach ($test in @(\n' ' "test_openai_api_server.exe",' ) + full_suite_invoke = ( + 'Invoke-Checked (Join-Path $BuildDir ' + '"tests/Release/$test") @()' + ) self.assertEqual(script.count(test_path), 1) self.assertEqual(script.count(probe), 1) self.assertEqual(script.count(full_suite), 1) self.assertLess(script.index(test_path), script.index(probe)) self.assertLess(script.index(probe), script.index(full_suite)) + suite_start = script.index(full_suite) + suite_end = script.index("\n}\n", suite_start) + 2 + suite = script[suite_start:suite_end] + body_start = suite.index(")) {") + len(")) {") + invoke_start = suite.index(full_suite_invoke, body_start) + self.assertEqual(suite.count(full_suite_invoke), 1) + self.assertEqual( + suite[body_start:invoke_start].strip(), + "", + "the unchanged full suite must run before any loop-body control flow", + ) + def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8") + handler_start = main.index( + "[[noreturn]] void DiagnosticTerminate() noexcept {" + ) + handler_end = main.index("\n}\n\n} // namespace", handler_start) + 2 + handler = main[handler_start:handler_end] for diagnostic in ( - "std::set_terminate", "std::current_exception()", "std::rethrow_exception", "catch (const std::exception& error)", @@ -136,11 +156,37 @@ def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> No "std::abort()", ): with self.subTest(diagnostic=diagnostic): - self.assertEqual(main.count(diagnostic), 1) + self.assertEqual(handler.count(diagnostic), 1) + self.assertLess( + handler.index("std::fflush(stderr)"), handler.index("std::abort()") + ) + + entry_start = main.index("int main(int argc, char** argv) {") + entry_end = main.index("\n}", entry_start) + 2 + entry = main[entry_start:entry_end] + terminate_install = "std::set_terminate(DiagnosticTerminate);" + self.assertEqual(entry.count("std::set_terminate("), 1) + self.assertEqual(entry.count(terminate_install), 1) + self.assertLess( + entry.index(terminate_install), + entry.index("doctest::Context(argc, argv).run()"), + ) source = ( ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" ).read_text(encoding="utf-8") + marker_start = source.index("class TeardownProbeMarker {") + marker_end = source.index("\n};", marker_start) + 3 + marker = source[marker_start:marker_end] + destructor_start = marker.index("~TeardownProbeMarker() {") + destructor_end = marker.index("\n }", destructor_start) + 4 + destructor = marker[destructor_start:destructor_end] + emit = "std::fputs(message_, stderr);" + flush = "std::fflush(stderr);" + self.assertEqual(destructor.count(emit), 1) + self.assertEqual(destructor.count(flush), 1) + self.assertLess(destructor.index(emit), destructor.index(flush)) + start = source.index( 'TEST_CASE("api_server: socket smoke — real HTTP requests over an ephemeral port")' ) From d79fa283d0a514b131c6a04853c55b629caf5c0b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 08:18:25 +0000 Subject: [PATCH 19/35] fix(windows): make failure contract portable (#599) Use a PowerShell child for the live exit-23 contract so Windows PowerShell and cross-platform PowerShell execute the same real process boundary. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 21 ++++++++++++ scripts/build-windows-release.ps1 | 7 ++-- .../scripts/test_release_windows_metadata.py | 33 +++++++++++++++++++ 3 files changed, 57 insertions(+), 4 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 0a341a139..5b338f26d 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -627,6 +627,27 @@ PowerShell Core does not prove Windows process semantics. Stop with `NEEDS_DECISION` if the repair would change production invocation or the native release gate rather than only its contract fixture. +#### #599 implementation outcome + +The failure was confined to the contract fixture: PowerShell Core 7.6.4 on +Linux dispatched the temporary `fail.cmd` through `gio`, whose success status +made the real `Invoke-Checked` helper accept the supposed failure child. The +fixture is now a temporary PowerShell script containing only `exit 23`; both +Windows PowerShell and cross-platform PowerShell execute that script directly. +Production `Invoke-Checked`, its argument splatting, the release gate, and the +workflow topology are unchanged. + +Before the repair, the focused metadata contract failed on `fail.cmd` and the +live `-ContractTest` failed with `nonzero child exit was accepted`. After the +repair, the live PowerShell contract, all nine Windows metadata tests, all 42 +release-pipeline tests, all 76 Windows-portability tests, and both direct +Windows checkers passed. Independently restoring the batch fixture, changing +the child to exit 24, and bypassing the real child invocation each made both +the focused structural and live-process tests fail; restoration returned the +script and test to SHA-256 values `bc98e85ec59b076e04c490b23d00c41b9a8fe57277d2ba051f4fc0d3f65c569e` +and `cb81cc1b9914306081b0861929e907cf6d50310b8a6dde7f4d27aeae40700df8`. +Native Windows CPU and Vulkan acceptance remains pending the hosted PR jobs. + #### #540 implementation outcome Implementation evidence: the focused contract was red with all six old diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 005de80ee..17de124ab 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -30,7 +30,7 @@ function Invoke-CheckedContractTests { $temporaryDir = Join-Path ([System.IO.Path]::GetTempPath()) ` "vllm-invoke-checked-$([guid]::NewGuid().ToString('N'))" $recordingTarget = Join-Path $temporaryDir "record-arguments.ps1" - $failingTarget = Join-Path $temporaryDir "fail.cmd" + $failingTarget = Join-Path $temporaryDir "fail.ps1" $callLog = Join-Path $temporaryDir "calls.txt" $savedCallLog = $env:VLLM_INVOKE_CHECKED_LOG try { @@ -48,9 +48,8 @@ param( exit 0 '@ | Set-Content -LiteralPath $recordingTarget -Encoding utf8 @' -@echo off -exit /b 23 -'@ | Set-Content -LiteralPath $failingTarget -Encoding ascii +exit 23 +'@ | Set-Content -LiteralPath $failingTarget -Encoding utf8 $env:VLLM_INVOKE_CHECKED_LOG = $callLog Invoke-Checked $recordingTarget @() diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 23ebf81f1..7b4499272 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -7,6 +7,8 @@ import importlib.util import json import re +import shutil +import subprocess import tempfile import unittest from pathlib import Path @@ -225,6 +227,9 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) contract = script[contract_start:contract_end] self.assertNotIn("record-arguments.cmd", contract) self.assertEqual(contract.count('"record-arguments.ps1"'), 1) + self.assertNotIn('"fail.cmd"', contract) + self.assertEqual(contract.count('"fail.ps1"'), 1) + self.assertEqual(contract.count("exit 23"), 1) for recorder_statement in ( "[Parameter(ValueFromRemainingArguments = $true)]", "[string[]]$RemainingArguments = @()", @@ -278,6 +283,34 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) dispatch = script[script.index("if ($ContractTest)"):] self.assertEqual(dispatch.count("Invoke-CheckedContractTests"), 1) + def test_contract_test_executes_real_children_under_powershell_core(self) -> None: + pwsh = shutil.which("pwsh") + if pwsh is None: + snap_pwsh = Path("/snap/bin/pwsh") + if snap_pwsh.is_file(): + pwsh = str(snap_pwsh) + if pwsh is None: + self.skipTest("PowerShell Core is not installed") + + result = subprocess.run( + [ + pwsh, + "-NoProfile", + "-NonInteractive", + "-File", + str(ROOT / "scripts/build-windows-release.ps1"), + "-SourceDir", + str(ROOT), + "-ContractTest", + ], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("Windows PowerShell/CRT contract tests OK", result.stdout) + def test_pe_report_rejects_msys_debug_crt_and_developer_paths(self) -> None: with tempfile.TemporaryDirectory() as temporary: args = self.fixture(Path(temporary), "cpu") From 41d740fff747f9140586e057cdad0fe44171c945 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 08:43:47 +0000 Subject: [PATCH 20/35] fix(windows): preserve child process contract (#599) Run the portable PowerShell fixtures through the current host executable so the contract exercises a real process boundary. Pin the boundary with a child PID assertion while retaining exact exit and argv checks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 20 ++++++++++++++++++ scripts/build-windows-release.ps1 | 18 +++++++++++++--- .../scripts/test_release_windows_metadata.py | 21 ++++++++++++++++--- 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 5b338f26d..e570879a4 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -648,6 +648,26 @@ script and test to SHA-256 values `bc98e85ec59b076e04c490b23d00c41b9a8fe57277d2b and `cb81cc1b9914306081b0861929e907cf6d50310b8a6dde7f4d27aeae40700df8`. Native Windows CPU and Vulkan acceptance remains pending the hosted PR jobs. +Fresh review of `bb3a4ef409d372b7f22698b8b530b4cb7f953cb9` disproved the +outcome's claim that direct invocation of the portable `.ps1` fixture preserved +a real-process boundary: the parent and invoked fixture both recorded PID +`3526837`. Direct `.ps1` invocation creates another PowerShell scope in the +same host process, so the earlier process-boundary claim is rejected. + +The follow-up resolves the running host through +`[System.Diagnostics.Process]::GetCurrentProcess().MainModule.FileName`, which +is available to Windows PowerShell 5.1 as well as PowerShell Core, and passes +that executable plus `-NoProfile -NonInteractive -File` through the unchanged +`Invoke-Checked` helper for every fixture. The failing fixture records its PID +before exact exit 23, and the live contract rejects equality with the parent +PID. Before the repair that assertion failed with `failure target did not +execute in a child process`; after it, the live contract, all 9 Windows metadata +tests, all 42 release-pipeline tests, all 76 Windows-portability tests, and the +four direct Windows release checkers pass. Mutating the failure launch back to +direct `.ps1`, changing exit 23 to exit 24, or bypassing `Invoke-Checked` makes +both structural and live-process coverage red. Native hosted Windows remains +the final authority for Windows process semantics. + #### #540 implementation outcome Implementation evidence: the focused contract was red with all six old diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 17de124ab..4fec4016a 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -33,6 +33,7 @@ function Invoke-CheckedContractTests { $failingTarget = Join-Path $temporaryDir "fail.ps1" $callLog = Join-Path $temporaryDir "calls.txt" $savedCallLog = $env:VLLM_INVOKE_CHECKED_LOG + $powerShellExecutable = [System.Diagnostics.Process]::GetCurrentProcess().MainModule.FileName try { New-Item -ItemType Directory -Path $temporaryDir | Out-Null @' @@ -48,11 +49,15 @@ param( exit 0 '@ | Set-Content -LiteralPath $recordingTarget -Encoding utf8 @' +[string]$PID | Set-Content ` + -LiteralPath $env:VLLM_INVOKE_CHECKED_LOG -Encoding ascii exit 23 '@ | Set-Content -LiteralPath $failingTarget -Encoding utf8 $env:VLLM_INVOKE_CHECKED_LOG = $callLog + $parentProcessId = $PID - Invoke-Checked $recordingTarget @() + Invoke-Checked $powerShellExecutable @( + "-NoProfile", "-NonInteractive", "-File", $recordingTarget) $zeroArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json if ([int]$zeroArgumentRecord.Count -ne 0 -or @($zeroArgumentRecord.Arguments).Count -ne 0) { @@ -60,7 +65,9 @@ exit 23 } Remove-Item -LiteralPath $callLog - Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value") + Invoke-Checked $powerShellExecutable @( + "-NoProfile", "-NonInteractive", "-File", $recordingTarget, + "alpha", "two words", "--flag=value") $nonemptyArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json if ([int]$nonemptyArgumentRecord.Count -ne 3 -or @($nonemptyArgumentRecord.Arguments).Count -ne 3 -or @@ -72,7 +79,8 @@ exit 23 $rejected = $false try { - Invoke-Checked $failingTarget @() + Invoke-Checked $powerShellExecutable @( + "-NoProfile", "-NonInteractive", "-File", $failingTarget) } catch { if ($_.Exception.Message -notmatch 'exited with status 23') { throw @@ -82,6 +90,10 @@ exit 23 if (-not $rejected) { throw "nonzero child exit was accepted" } + $failingChildProcessId = [int](Get-Content -LiteralPath $callLog -Raw) + if ($failingChildProcessId -eq $parentProcessId) { + throw "failure target did not execute in a child process" + } } finally { if ($null -eq $savedCallLog) { Remove-Item Env:VLLM_INVOKE_CHECKED_LOG -ErrorAction SilentlyContinue diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 7b4499272..be957530d 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -240,20 +240,34 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) with self.subTest(recorder_statement=recorder_statement): self.assertEqual(contract.count(recorder_statement), 1) + self.assertEqual( + contract.count( + "$powerShellExecutable = " + "[System.Diagnostics.Process]::GetCurrentProcess().MainModule.FileName" + ), + 1, + ) + self.assertEqual(contract.count("Invoke-Checked $powerShellExecutable @("), 3) + self.assertNotIn("Invoke-Checked $recordingTarget", contract) + self.assertNotIn("Invoke-Checked $failingTarget", contract) + required_statements = ( - "Invoke-Checked $recordingTarget @()", + '"-NoProfile", "-NonInteractive", "-File", $recordingTarget)', "$zeroArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json", "[int]$zeroArgumentRecord.Count -ne 0", "@($zeroArgumentRecord.Arguments).Count -ne 0", - 'Invoke-Checked $recordingTarget @("alpha", "two words", "--flag=value")', + '"-NoProfile", "-NonInteractive", "-File", $recordingTarget,', + '"alpha", "two words", "--flag=value")', "$nonemptyArgumentRecord = Get-Content -LiteralPath $callLog -Raw | ConvertFrom-Json", "[int]$nonemptyArgumentRecord.Count -ne 3", "@($nonemptyArgumentRecord.Arguments).Count -ne 3", '$nonemptyArgumentRecord.Arguments[0] -cne "alpha"', '$nonemptyArgumentRecord.Arguments[1] -cne "two words"', '$nonemptyArgumentRecord.Arguments[2] -cne "--flag=value"', - "Invoke-Checked $failingTarget @()", + '"-NoProfile", "-NonInteractive", "-File", $failingTarget)', "if (-not $rejected)", + "$failingChildProcessId = [int](Get-Content -LiteralPath $callLog -Raw)", + "if ($failingChildProcessId -eq $parentProcessId)", ) cursor = 0 for statement in required_statements: @@ -276,6 +290,7 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) "zero-argument target was not invoked exactly once without arguments", "nonempty arguments did not arrive unchanged", "nonzero child exit was accepted", + "failure target did not execute in a child process", ): with self.subTest(behavior=behavior): self.assertEqual(contract.count(behavior), 1) From 244afc0a5dc455b33abaed10c8f44edf2bdd7405 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 09:07:45 +0000 Subject: [PATCH 21/35] test(windows): verify child PID externally (#599) Review proved the live Python test trusted the script's own PID guard. Emit both PIDs so the harness verifies the process boundary independently. Pin the runtime guard to keep the script fail-closed check. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 20 +++++++++++++++++ scripts/build-windows-release.ps1 | 1 + .../scripts/test_release_windows_metadata.py | 22 +++++++++++++++++++ 3 files changed, 43 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index e570879a4..a7dd5fd1d 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -668,6 +668,26 @@ direct `.ps1`, changing exit 23 to exit 24, or bypassing `Invoke-Checked` makes both structural and live-process coverage red. Native hosted Windows remains the final authority for Windows process semantics. +Fresh review of `8dfddfed2b27f6830768acbe32803c1bb7399459` found that the +external live Python test still trusted the contract's in-script PID guard. +Deleting that guard left the live method green, so the claimed child-process +evidence was not independently checked outside the script. The strengthened +contract emits one stable diagnostic containing only the parent and failing +child integer PIDs. The live Python method parses both values, requires each to +be positive, and independently rejects equality; it also pins the exact runtime +equality guard so removing either layer is red. + +Before the test change, deleting the runtime guard reproduced the false green. +The strengthened test was then RED because the diagnostic did not yet exist. +After adding the diagnostic only to `Invoke-CheckedContractTests`, the direct +PowerShell contract and live Python method pass with distinct PIDs. Removing +the runtime guard, launching the failing `.ps1` directly, changing exit 23 to +exit 24, and bypassing `Invoke-Checked` each made the focused live method red; +each mutation was restored before the next. All 9 Windows metadata tests, all +42 release-pipeline tests, all 76 Windows-portability tests, and the four direct +Windows release checkers pass. Production `Invoke-Checked`, the release gate, +and release topology remain unchanged; native hosted Windows is still binding. + #### #540 implementation outcome Implementation evidence: the focused contract was red with all six old diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 4fec4016a..d0e5c0464 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -91,6 +91,7 @@ exit 23 throw "nonzero child exit was accepted" } $failingChildProcessId = [int](Get-Content -LiteralPath $callLog -Raw) + Write-Host "Invoke-Checked PID contract: parent=$parentProcessId failure_child=$failingChildProcessId" if ($failingChildProcessId -eq $parentProcessId) { throw "failure target did not execute in a child process" } diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index be957530d..faab78e52 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -299,6 +299,16 @@ def test_invoke_checked_contract_covers_empty_exact_and_failing_arguments(self) self.assertEqual(dispatch.count("Invoke-CheckedContractTests"), 1) def test_contract_test_executes_real_children_under_powershell_core(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text( + encoding="utf-8" + ) + runtime_pid_guard = ( + " if ($failingChildProcessId -eq $parentProcessId) {\n" + ' throw "failure target did not execute in a child process"\n' + " }\n" + ) + self.assertEqual(script.count(runtime_pid_guard), 1) + pwsh = shutil.which("pwsh") if pwsh is None: snap_pwsh = Path("/snap/bin/pwsh") @@ -325,6 +335,18 @@ def test_contract_test_executes_real_children_under_powershell_core(self) -> Non ) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn("Windows PowerShell/CRT contract tests OK", result.stdout) + pid_diagnostic = re.search( + r"(?m)^Invoke-Checked PID contract: parent=(?P[0-9]+) " + r"failure_child=(?P[0-9]+)\r?$", + result.stdout, + ) + if pid_diagnostic is None: + self.fail(f"missing Invoke-Checked PID diagnostic:\n{result.stdout}") + parent_process_id = int(pid_diagnostic.group("parent")) + failure_child_process_id = int(pid_diagnostic.group("child")) + self.assertGreater(parent_process_id, 0) + self.assertGreater(failure_child_process_id, 0) + self.assertNotEqual(parent_process_id, failure_child_process_id) def test_pe_report_rejects_msys_debug_crt_and_developer_paths(self) -> None: with tempfile.TemporaryDirectory() as temporary: From fac27ce7535c63e62eaba7303cecd31246afa98d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 15:55:52 +0000 Subject: [PATCH 22/35] spec(windows): bind adaptive prefix bisect (#537) Hosted Windows now completes every teardown marker before the exact fast-fail. The prefix probe isolates the earliest contaminating boundary. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index a7dd5fd1d..c1e7364e1 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -595,6 +595,28 @@ for that exact lifetime defect before implementing a repair. If the isolated case passes but the full process fails, bisect test-case prefixes in fresh processes to prove the cross-test state dependency rather than guessing. +Candidate `f06c77fe4af502a9934e525112008e1a02bdc1ff` executed that probe in +[run `31688115193`](https://github.com/mudler/vllm.cpp/actions/runs/31688115193). +CPU job `94408881944` and Vulkan job `94408881974` produced the same boundary: +the isolated socket case passed with all six destruction markers, while the +full 54-case process produced those same six markers and then fast-failed with +exact signed status `-1073740791` (`0xC0000409`) before the doctest summary. +Neither job emitted the diagnostic `std::terminate` marker. This rules out the +three instrumented owners as the final failing boundary and establishes a +source-order cross-test dependency as the next hypothesis. + +The developer approved one diagnostic-only adaptive prefix bisect. It lists the +54 doctest cases in file order, executes every probed `--first=1 --last=N` +prefix in a fresh process, requires the full prefix to reproduce only the exact +native fast-fail, and requires the first case as a known-good short prefix. It +then binary-searches the smallest failing `N`, confirms `N-1` succeeds and `N` +fast-fails in new processes, and runs only case `N` with matching first/last +bounds to distinguish an isolated defect from cumulative contamination. The +probe emits one stable diagnostic containing `N`, the listed test name, all +three confirmation statuses, and the dependency classification. The existing +unfiltered full-suite invocation remains unchanged and still runs afterwards. +Any other probe status stops the release job rather than being classified. + ### Cross-platform PowerShell contract follow-up: issue #599 Fresh review of diagnostic candidate `5a845c28a7a9afe5addf94771e59a71cecd31e81` From 182e4161d845aa17a8dd14a084155dd3b0aaf9b7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 16:05:00 +0000 Subject: [PATCH 23/35] test(windows): bisect OpenAI fast-fail (#537) The isolated socket case completes teardown, but the full native Windows process still fast-fails. Fresh source-order prefixes locate the earliest contaminating test boundary without weakening the original full gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- scripts/build-windows-release.ps1 | 205 ++++++++++++++++++ .../scripts/test_release_windows_metadata.py | 74 ++++++- 2 files changed, 276 insertions(+), 3 deletions(-) diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index d0e5c0464..8812955d7 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -235,10 +235,214 @@ function Invoke-UnsupportedTierContractTests { } } +function Invoke-OpenAiProbeProcess { + param([Parameter(Mandatory)][string]$Program, + [Parameter(Mandatory)][AllowEmptyCollection()][string[]]$Arguments, + [scriptblock]$Runner) + if ($null -eq $Runner) { + $output = @(& $Program @Arguments 2>&1) + return [pscustomobject]@{ + ExitCode = [int]$LASTEXITCODE + Output = @($output) + } + } + $result = & $Runner $Program $Arguments + if ($null -eq $result) { + throw "OpenAI prefix probe runner returned no result" + } + return [pscustomobject]@{ + ExitCode = [int]$result.ExitCode + Output = @($result.Output) + } +} + +function Invoke-OpenAiPrefixRange { + param([Parameter(Mandatory)][string]$Program, + [Parameter(Mandatory)][int]$First, + [Parameter(Mandatory)][int]$Last, + [scriptblock]$Runner) + return Invoke-OpenAiProbeProcess -Program $Program -Arguments @( + "--order-by=file", + "--first=$First", + "--last=$Last", + "--success=true", + "--duration=true", + "--no-colors=true" + ) -Runner $Runner +} + +function Invoke-OpenAiPrefixBisect { + param([Parameter(Mandatory)][string]$TestProgram, + [scriptblock]$Runner) + $expectedFastFailStatus = -1073740791 + $listResult = Invoke-OpenAiProbeProcess -Program $TestProgram -Arguments @( + "--list-test-cases", + "--order-by=file", + "--no-version=true", + "--no-colors=true" + ) -Runner $Runner + if ($listResult.ExitCode -ne 0) { + throw "OpenAI test listing exited with status $($listResult.ExitCode)" + } + + $testCases = [System.Collections.Generic.List[string]]::new() + $separatorCount = 0 + foreach ($outputLine in @($listResult.Output)) { + $line = [string]$outputLine + if ($line -match '^=+$') { + $separatorCount++ + if ($separatorCount -eq 2) { break } + continue + } + if ($separatorCount -eq 1 -and $line.Length -gt 0) { + $testCases.Add($line) | Out-Null + } + } + if ($separatorCount -ne 2 -or $testCases.Count -lt 2) { + throw "OpenAI test listing did not contain a complete source-order case list" + } + $testCount = $testCases.Count + + $fullPrefixResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First 1 -Last $testCount -Runner $Runner + if ($fullPrefixResult.ExitCode -ne $expectedFastFailStatus) { + throw "OpenAI full prefix exited with status $($fullPrefixResult.ExitCode) instead of $expectedFastFailStatus" + } + $shortPrefixResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First 1 -Last 1 -Runner $Runner + if ($shortPrefixResult.ExitCode -ne 0) { + throw "OpenAI first-case prefix exited with status $($shortPrefixResult.ExitCode) instead of 0" + } + + $low = 1 + $high = $testCount + while ($high - $low -gt 1) { + $mid = [int][math]::Floor(($low + $high) / 2) + $prefixResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First 1 -Last $mid -Runner $Runner + if ($prefixResult.ExitCode -eq 0) { + $low = $mid + } elseif ($prefixResult.ExitCode -eq $expectedFastFailStatus) { + $high = $mid + } else { + throw "OpenAI prefix 1..$mid exited with unexpected status $($prefixResult.ExitCode)" + } + } + + $firstBad = $high + $predecessorResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First 1 -Last ($firstBad - 1) -Runner $Runner + if ($predecessorResult.ExitCode -ne 0) { + throw "OpenAI confirmed predecessor prefix exited with status $($predecessorResult.ExitCode) instead of 0" + } + $badPrefixResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First 1 -Last $firstBad -Runner $Runner + if ($badPrefixResult.ExitCode -ne $expectedFastFailStatus) { + throw "OpenAI confirmed bad prefix exited with status $($badPrefixResult.ExitCode) instead of $expectedFastFailStatus" + } + $isolatedResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First $firstBad -Last $firstBad -Runner $Runner + if ($isolatedResult.ExitCode -eq 0) { + $dependency = "cumulative" + } elseif ($isolatedResult.ExitCode -eq $expectedFastFailStatus) { + $dependency = "isolated" + } else { + throw "OpenAI isolated case $firstBad exited with unexpected status $($isolatedResult.ExitCode)" + } + + $testName = $testCases[$firstBad - 1] + Write-Host "OpenAI prefix bisect: first_bad=$firstBad/$testCount test=`"$testName`" predecessor_status=$($predecessorResult.ExitCode) prefix_status=$($badPrefixResult.ExitCode) isolated_status=$($isolatedResult.ExitCode) dependency=$dependency" + return [pscustomobject]@{ + FirstBad = $firstBad + TestCount = $testCount + TestName = $testName + Dependency = $dependency + } +} + +function Invoke-OpenAiPrefixBisectContractTests { + $firstBad = 27 + $testCount = 54 + $calls = [System.Collections.Generic.List[object]]::new() + $runner = { + param([string]$Program, [string[]]$Arguments) + $calls.Add([pscustomobject]@{ + Program = $Program + Arguments = @($Arguments) + }) | Out-Null + if ($Arguments -contains "--list-test-cases") { + $listing = [System.Collections.Generic.List[string]]::new() + $listing.Add("[doctest] listing all test case names") | Out-Null + $listing.Add("===============================================================================") | Out-Null + foreach ($index in 1..$testCount) { + $listing.Add("case $index") | Out-Null + } + $listing.Add("===============================================================================") | Out-Null + $listing.Add("[doctest] unskipped test cases passing the current filters: $testCount") | Out-Null + return [pscustomobject]@{ ExitCode = 0; Output = @($listing) } + } + + $firstArgument = @($Arguments | Where-Object { $_ -like "--first=*" }) + $lastArgument = @($Arguments | Where-Object { $_ -like "--last=*" }) + if ($firstArgument.Count -ne 1 -or $lastArgument.Count -ne 1 -or + $Arguments -notcontains "--order-by=file") { + return [pscustomobject]@{ ExitCode = 2; Output = @("bad range arguments") } + } + $first = [int]$firstArgument[0].Substring("--first=".Length) + $last = [int]$lastArgument[0].Substring("--last=".Length) + $exitCode = if ($first -eq 1 -and $last -ge $firstBad) { + -1073740791 + } else { + 0 + } + return [pscustomobject]@{ ExitCode = $exitCode; Output = @() } + }.GetNewClosure() + + $result = Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` + -Runner $runner + if ($result.FirstBad -ne $firstBad -or + $result.TestCount -ne $testCount -or + $result.TestName -cne "case 27" -or + $result.Dependency -cne "cumulative") { + throw "OpenAI prefix bisect returned the wrong boundary" + } + $listCalls = @($calls | Where-Object { + $_.Arguments -contains "--list-test-cases" + }) + if ($listCalls.Count -ne 1) { + throw "OpenAI prefix bisect did not list tests exactly once" + } + foreach ($call in @($calls | Where-Object { + $_.Arguments -notcontains "--list-test-cases" + })) { + $firstArguments = @($call.Arguments | Where-Object { $_ -like "--first=*" }) + $lastArguments = @($call.Arguments | Where-Object { $_ -like "--last=*" }) + if ($firstArguments.Count -ne 1 -or $lastArguments.Count -ne 1 -or + $call.Arguments -notcontains "--order-by=file") { + throw "OpenAI prefix bisect emitted a non-source-order range probe" + } + } + foreach ($bounds in @( + @("--first=1", "--last=26"), + @("--first=1", "--last=27"), + @("--first=27", "--last=27") + )) { + $matches = @($calls | Where-Object { + $_.Arguments -contains $bounds[0] -and + $_.Arguments -contains $bounds[1] + }) + if ($matches.Count -lt 1) { + throw "OpenAI prefix bisect omitted confirmation $($bounds -join '..')" + } + } + Write-Host "OpenAI prefix bisect contract OK" +} + if ($ContractTest) { Invoke-CheckedContractTests Invoke-CrtContractTests Invoke-UnsupportedTierContractTests + Invoke-OpenAiPrefixBisectContractTests Write-Host "Windows PowerShell/CRT contract tests OK" exit 0 } @@ -303,6 +507,7 @@ Invoke-Checked $openaiApiServerTest @( "--success=true", "--duration=true" ) +Invoke-OpenAiPrefixBisect -TestProgram $openaiApiServerTest foreach ($test in @( "test_openai_api_server.exe", diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index faab78e52..eb7c4f8b6 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -102,7 +102,7 @@ def test_contract_test_precedes_required_release_environment(self) -> None: self.assertLess(contract, contract_exit) self.assertLess(contract_exit, environment) - def test_openai_socket_teardown_probe_precedes_unchanged_full_suite(self) -> None: + def test_openai_socket_probe_and_prefix_bisect_precede_unchanged_full_suite(self) -> None: script = (ROOT / "scripts/build-windows-release.ps1").read_text( encoding="utf-8" ) @@ -117,6 +117,7 @@ def test_openai_socket_teardown_probe_precedes_unchanged_full_suite(self) -> Non ' "--duration=true"\n' ")" ) + bisect = "Invoke-OpenAiPrefixBisect -TestProgram $openaiApiServerTest" full_suite = ( 'foreach ($test in @(\n' ' "test_openai_api_server.exe",' @@ -127,22 +128,82 @@ def test_openai_socket_teardown_probe_precedes_unchanged_full_suite(self) -> Non ) self.assertEqual(script.count(test_path), 1) self.assertEqual(script.count(probe), 1) + self.assertEqual(script.count(bisect), 1) self.assertEqual(script.count(full_suite), 1) self.assertLess(script.index(test_path), script.index(probe)) - self.assertLess(script.index(probe), script.index(full_suite)) + self.assertLess(script.index(probe), script.index(bisect)) + self.assertLess(script.index(bisect), script.index(full_suite)) suite_start = script.index(full_suite) suite_end = script.index("\n}\n", suite_start) + 2 suite = script[suite_start:suite_end] body_start = suite.index(")) {") + len(")) {") - invoke_start = suite.index(full_suite_invoke, body_start) self.assertEqual(suite.count(full_suite_invoke), 1) + invoke_start = suite.index(full_suite_invoke, body_start) self.assertEqual( suite[body_start:invoke_start].strip(), "", "the unchanged full suite must run before any loop-body control flow", ) + def test_openai_prefix_bisect_pins_adaptive_exact_status_contract(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text( + encoding="utf-8" + ) + range_start = script.find("function Invoke-OpenAiPrefixRange {") + self.assertNotEqual(range_start, -1, "fresh-process prefix range is missing") + range_end = script.index("\n}\n", range_start) + len("\n}\n") + range_helper = script[range_start:range_end] + for argument in ( + '"--order-by=file"', + '"--first=$First"', + '"--last=$Last"', + ): + with self.subTest(argument=argument): + self.assertEqual(range_helper.count(argument), 1) + self.assertEqual(range_helper.count("Invoke-OpenAiProbeProcess"), 1) + + helper_start = script.find("function Invoke-OpenAiPrefixBisect {") + self.assertNotEqual(helper_start, -1, "adaptive prefix bisect is missing") + helper_end = script.index("\n}\n", helper_start) + len("\n}\n") + helper = script[helper_start:helper_end] + for statement in ( + "$expectedFastFailStatus = -1073740791", + '"--list-test-cases"', + "$fullPrefixResult.ExitCode -ne $expectedFastFailStatus", + "$shortPrefixResult.ExitCode -ne 0", + "$prefixResult.ExitCode -eq 0", + "$prefixResult.ExitCode -eq $expectedFastFailStatus", + "$predecessorResult.ExitCode -ne 0", + "$badPrefixResult.ExitCode -ne $expectedFastFailStatus", + "$isolatedResult.ExitCode -eq 0", + "$isolatedResult.ExitCode -eq $expectedFastFailStatus", + 'dependency=$dependency', + ): + with self.subTest(statement=statement): + self.assertIn(statement, helper) + self.assertEqual(helper.count("Invoke-OpenAiPrefixRange"), 6) + self.assertIn("while ($high - $low -gt 1)", helper) + self.assertIn("[math]::Floor(($low + $high) / 2)", helper) + + contract_start = script.find( + "function Invoke-OpenAiPrefixBisectContractTests {" + ) + self.assertNotEqual(contract_start, -1, "prefix bisect contract is missing") + contract_end = script.index("\n}\n", contract_start) + len("\n}\n") + contract = script[contract_start:contract_end] + for guarantee in ( + "$firstBad = 27", + "$testCount = 54", + '"--first=1"', + '"--last=26"', + '"--last=27"', + '"--first=27"', + '"OpenAI prefix bisect contract OK"', + ): + with self.subTest(guarantee=guarantee): + self.assertIn(guarantee, contract) + def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8") handler_start = main.index( @@ -335,6 +396,13 @@ def test_contract_test_executes_real_children_under_powershell_core(self) -> Non ) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn("Windows PowerShell/CRT contract tests OK", result.stdout) + self.assertIn( + 'OpenAI prefix bisect: first_bad=27/54 test="case 27" ' + 'predecessor_status=0 prefix_status=-1073740791 isolated_status=0 ' + 'dependency=cumulative', + result.stdout, + ) + self.assertIn("OpenAI prefix bisect contract OK", result.stdout) pid_diagnostic = re.search( r"(?m)^Invoke-Checked PID contract: parent=(?P[0-9]+) " r"failure_child=(?P[0-9]+)\r?$", From 7a0734712e0d33b0d95583b3a43bc8900af68d0d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 16:28:15 +0000 Subject: [PATCH 24/35] test(windows): close prefix probe gaps (#537) The adaptive bisect contract did not distinguish listing order, execute both fail-closed status branches, or pin its diagnostic schema. The strengthened live contract makes each guarantee independently observable. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 14 ++ scripts/build-windows-release.ps1 | 128 ++++++++++++++++-- .../scripts/test_release_windows_metadata.py | 10 +- 3 files changed, 141 insertions(+), 11 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index c1e7364e1..ff05f0684 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -617,6 +617,20 @@ three confirmation statuses, and the dependency classification. The existing unfiltered full-suite invocation remains unchanged and still runs afterwards. Any other probe status stops the release job rather than being classified. +Fresh mutation review of the adaptive probe found three false-green contract +gaps: the listing fixture did not distinguish file order from name order, the +unexpected-status branches at an intermediate midpoint and isolated probe were +not executed, and the emitted diagnostic field names were not pinned. The +repaired live PowerShell contract uses a deliberately name-order-inverted +source-order listing, injects status 7 independently at the midpoint and +isolated boundaries and requires both invocations to throw, and captures the +real host output for an exact diagnostic-schema comparison. Mutating the +listing call to name order, accepting either unexpected status, or renaming +`predecessor_status` now makes the contract red. The restored candidate passes +the 12-test Windows metadata suite, the combined 130-test metadata/pipeline/ +portability suite, the live PowerShell contract, and all four direct Windows +release checkers. + ### Cross-platform PowerShell contract follow-up: issue #599 Fresh review of diagnostic candidate `5a845c28a7a9afe5addf94771e59a71cecd31e81` diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index 8812955d7..e27ca6990 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -363,6 +363,29 @@ function Invoke-OpenAiPrefixBisect { function Invoke-OpenAiPrefixBisectContractTests { $firstBad = 27 $testCount = 54 + $caseNames = @( + foreach ($index in 1..$testCount) { + if ($index -eq $firstBad) { + "aaa source case $index" + } elseif ($index -eq 1) { + "zzz source case $index" + } else { + "source case $index" + } + } + ) + if (@($caseNames | Sort-Object)[0] -ceq $caseNames[0]) { + throw "OpenAI prefix bisect fixture does not distinguish file order from name order" + } + $listing = [System.Collections.Generic.List[string]]::new() + $listing.Add("[doctest] listing all test case names") | Out-Null + $listing.Add("===============================================================================") | Out-Null + foreach ($caseName in $caseNames) { + $listing.Add($caseName) | Out-Null + } + $listing.Add("===============================================================================") | Out-Null + $listing.Add("[doctest] unskipped test cases passing the current filters: $testCount") | Out-Null + $calls = [System.Collections.Generic.List[object]]::new() $runner = { param([string]$Program, [string[]]$Arguments) @@ -371,14 +394,14 @@ function Invoke-OpenAiPrefixBisectContractTests { Arguments = @($Arguments) }) | Out-Null if ($Arguments -contains "--list-test-cases") { - $listing = [System.Collections.Generic.List[string]]::new() - $listing.Add("[doctest] listing all test case names") | Out-Null - $listing.Add("===============================================================================") | Out-Null - foreach ($index in 1..$testCount) { - $listing.Add("case $index") | Out-Null + $orderArguments = @($Arguments | Where-Object { $_ -like "--order-by=*" }) + if ($orderArguments.Count -ne 1 -or + $orderArguments[0] -cne "--order-by=file") { + return [pscustomobject]@{ + ExitCode = 7 + Output = @($caseNames | Sort-Object) + } } - $listing.Add("===============================================================================") | Out-Null - $listing.Add("[doctest] unskipped test cases passing the current filters: $testCount") | Out-Null return [pscustomobject]@{ ExitCode = 0; Output = @($listing) } } @@ -398,11 +421,20 @@ function Invoke-OpenAiPrefixBisectContractTests { return [pscustomobject]@{ ExitCode = $exitCode; Output = @() } }.GetNewClosure() - $result = Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` - -Runner $runner + $captured = @(& { + Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` + -Runner $runner + } 6>&1) + $results = @($captured | Where-Object { + $null -ne $_.PSObject.Properties["FirstBad"] + }) + if ($results.Count -ne 1) { + throw "OpenAI prefix bisect did not return exactly one result" + } + $result = $results[0] if ($result.FirstBad -ne $firstBad -or $result.TestCount -ne $testCount -or - $result.TestName -cne "case 27" -or + $result.TestName -cne "aaa source case 27" -or $result.Dependency -cne "cumulative") { throw "OpenAI prefix bisect returned the wrong boundary" } @@ -412,6 +444,14 @@ function Invoke-OpenAiPrefixBisectContractTests { if ($listCalls.Count -ne 1) { throw "OpenAI prefix bisect did not list tests exactly once" } + if ($listCalls[0].Arguments.Count -ne 4 -or + $listCalls[0].Arguments[0] -cne "--list-test-cases" -or + $listCalls[0].Arguments[1] -cne "--order-by=file" -or + $listCalls[0].Arguments[2] -cne "--no-version=true" -or + $listCalls[0].Arguments[3] -cne "--no-colors=true") { + throw "OpenAI prefix bisect listing did not request exact file order" + } + Write-Host "OpenAI prefix bisect listing order contract OK" foreach ($call in @($calls | Where-Object { $_.Arguments -notcontains "--list-test-cases" })) { @@ -435,6 +475,74 @@ function Invoke-OpenAiPrefixBisectContractTests { throw "OpenAI prefix bisect omitted confirmation $($bounds -join '..')" } } + + $midpointInjected = $false + $unexpectedMidpointRunner = { + param([string]$Program, [string[]]$Arguments) + if ($Arguments -notcontains "--list-test-cases") { + $firstArgument = @($Arguments | Where-Object { $_ -like "--first=*" }) + $lastArgument = @($Arguments | Where-Object { $_ -like "--last=*" }) + if ($firstArgument.Count -eq 1 -and $lastArgument.Count -eq 1 -and + $firstArgument[0] -ceq "--first=1" -and + $lastArgument[0] -ceq "--last=27" -and + -not $midpointInjected) { + $midpointInjected = $true + return [pscustomobject]@{ ExitCode = 7; Output = @("injected midpoint failure") } + } + } + return & $runner $Program $Arguments + }.GetNewClosure() + $midpointRejected = $false + try { + Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` + -Runner $unexpectedMidpointRunner | Out-Null + } catch { + if ($_.Exception.Message -cne + "OpenAI prefix 1..27 exited with unexpected status 7") { + throw + } + $midpointRejected = $true + } + if (-not $midpointRejected) { + throw "OpenAI prefix bisect accepted an unexpected midpoint status" + } + Write-Host "OpenAI prefix bisect unexpected midpoint status contract OK" + + $unexpectedIsolatedRunner = { + param([string]$Program, [string[]]$Arguments) + if ($Arguments -notcontains "--list-test-cases" -and + $Arguments -contains "--first=27" -and + $Arguments -contains "--last=27") { + return [pscustomobject]@{ ExitCode = 7; Output = @("injected isolated failure") } + } + return & $runner $Program $Arguments + }.GetNewClosure() + $isolatedRejected = $false + try { + Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` + -Runner $unexpectedIsolatedRunner | Out-Null + } catch { + if ($_.Exception.Message -cne + "OpenAI isolated case 27 exited with unexpected status 7") { + throw + } + $isolatedRejected = $true + } + if (-not $isolatedRejected) { + throw "OpenAI prefix bisect accepted an unexpected isolated status" + } + Write-Host "OpenAI prefix bisect unexpected isolated status contract OK" + + $expectedDiagnostic = 'OpenAI prefix bisect: first_bad=27/54 test="aaa source case 27" predecessor_status=0 prefix_status=-1073740791 isolated_status=0 dependency=cumulative' + $diagnostics = @($captured | Where-Object { + ([string]$_).StartsWith("OpenAI prefix bisect: first_bad=") + }) + if ($diagnostics.Count -ne 1 -or + [string]$diagnostics[0] -cne $expectedDiagnostic) { + throw "OpenAI prefix bisect emitted an unstable diagnostic schema" + } + Write-Host "OpenAI prefix bisect diagnostic schema contract OK" + Write-Host $expectedDiagnostic Write-Host "OpenAI prefix bisect contract OK" } diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index eb7c4f8b6..6623a50c1 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -396,8 +396,16 @@ def test_contract_test_executes_real_children_under_powershell_core(self) -> Non ) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn("Windows PowerShell/CRT contract tests OK", result.stdout) + for proof in ( + "OpenAI prefix bisect listing order contract OK", + "OpenAI prefix bisect unexpected midpoint status contract OK", + "OpenAI prefix bisect unexpected isolated status contract OK", + "OpenAI prefix bisect diagnostic schema contract OK", + ): + with self.subTest(proof=proof): + self.assertIn(proof, result.stdout) self.assertIn( - 'OpenAI prefix bisect: first_bad=27/54 test="case 27" ' + 'OpenAI prefix bisect: first_bad=27/54 test="aaa source case 27" ' 'predecessor_status=0 prefix_status=-1073740791 isolated_status=0 ' 'dependency=cumulative', result.stdout, From ba39785f20c9fbe9c2dd5254a503687a416b4c9f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 16:53:43 +0000 Subject: [PATCH 25/35] spec(release): track LTX2 Windows portability regression Record issue #645 and constrain the repair to a standard C++ constant substitution without weakening the portability detector. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 23 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 6931562b8..ad0871a7a 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -106,6 +106,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#537](https://github.com/mudler/vllm.cpp/issues/537) | `ENG-RELEASE-WINDOWS` | Windows OpenAI socket smoke fast-fails instead of reporting a reset response | bug | | [#540](https://github.com/mudler/vllm.cpp/issues/540) | `ENG-RELEASE-WINDOWS` | Windows Vulkan strict build rejects shadowed KV-cache test locals | bug | | [#599](https://github.com/mudler/vllm.cpp/issues/599) | `ENG-RELEASE-WINDOWS` | Windows release contract failure fixture is not portable under PowerShell on Linux | bug | +| [#645](https://github.com/mudler/vllm.cpp/issues/645) | `ENG-RELEASE-WINDOWS` | LTX2 sources reintroduce non-portable `M_PI` and break the Windows release gate | bug | | [#515](https://github.com/mudler/vllm.cpp/issues/515) | `ENG-RECORD-CONFLICT-SURFACES` | check-doc-checkpoint treats every root `CMakeLists.txt` edit as a usage change, so adding a source file demands a `docs/USAGE.md` edit with nothing true to say | bug | | [#547](https://github.com/mudler/vllm.cpp/issues/547) | — | GB10 reports `UnifiedMemory()` true, so `ReferenceTierEligible(kCUDA)` runs the CPU host kernel over `cudaMalloc` pointers; it needs `DeviceMemoryIsHostAddressable()` | bug | | [#569](https://github.com/mudler/vllm.cpp/issues/569) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | The Nemotron revision pin was existence-only: a substituted checkpoint still resolved, and W6's token gate cannot see that | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index ff05f0684..a13389bb8 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -617,6 +617,29 @@ three confirmation statuses, and the dependency classification. The existing unfiltered full-suite invocation remains unchanged and still runs afterwards. Any other probe status stops the release job rather than being classified. +### Current-main portability regression: issue #645 + +Current `main` at `cefacd2d00cb9b4776331cd213116773cd97f811` added LTX2 +sources containing the non-standard `M_PI` macro. The existing real-tree +Windows portability regression is RED and names exactly these three files: + +- `src/vllm/model_executor/models/ltx2.cpp` +- `src/vllm/model_executor/models/ltx2_video_vae.cpp` +- `src/vllm/model_executor/models/ltx2_audio_vae.cpp` + +The repair is limited to replacing those uses with a standard C++ constant +whose type and value preserve the current expressions. Do not define feature +macros, add a checker exemption, weaken the real-tree scan, or alter LTX2 +algorithms. The existing failing real-tree test is the RED regression; focused +green requires that test, the complete Windows portability suite, the combined +release script suite, and a clean CPU build covering the affected translation +units. Hosted MSVC CPU and Vulkan builds remain binding. + +Fresh review must mutate at least one repaired expression back to `M_PI` and +must perturb the replacement constant enough to prove an LTX2 numerical test +detects it. Stop with `NEEDS_DECISION` if preserving the current value requires +an algorithmic or tolerance change rather than a constant substitution. + Fresh mutation review of the adaptive probe found three false-green contract gaps: the listing fixture did not distinguish file order from name order, the unexpected-status branches at an intermediate midpoint and isolated probe were From c70a1929d1937ba13ce461c15802a949681b5ca7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:07:53 +0000 Subject: [PATCH 26/35] fix(windows): use portable LTX2 pi constants (#645) Replace the non-standard M_PI macro in LTX2 numerical expressions with the C++20 double-precision constant, and remove the unused video VAE fallback. This preserves the existing expression types and values while allowing native MSVC builds to compile without feature macros. Record the RED/GREEN portability and upstream-golden evidence, and project the Windows portability guarantee in the existing feature row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 17 +++++++++++++++ docs/FEATURES.md | 2 +- src/vllm/model_executor/models/ltx2.cpp | 9 +++++--- .../model_executor/models/ltx2_audio_vae.cpp | 21 ++++++++++++------- .../model_executor/models/ltx2_video_vae.cpp | 4 ---- 5 files changed, 37 insertions(+), 16 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index a13389bb8..9a42b0c39 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -640,6 +640,23 @@ must perturb the replacement constant enough to prove an LTX2 numerical test detects it. Stop with `NEEDS_DECISION` if preserving the current value requires an algorithmic or tolerance change rather than a constant substitution. +#### #645 implementation outcome + +The regression was the non-standard macro itself: `ltx2.cpp` consumed `M_PI`, +while both VAE files carried fallback definitions even though the video VAE did +not consume the macro. The C++20 production expressions now use +`std::numbers::pi_v`, which preserves the previous double type and +rounded value, and the unused video fallback is gone. No algorithm or tolerance +changed. + +Before the repair, the real-tree portability regression failed with exactly the +three specified files. Afterwards that regression and all 76 Windows +portability tests pass; the combined Windows metadata, release-pipeline, and +portability suite passes 130 tests. A CPU Release build compiled all three +affected translation units and linked `test_ltx2` and `test_ltx2_vae`; their +upstream-golden numerical gates pass 30/30 cases with 1627 assertions and 36/36 +cases with 3039 assertions, respectively. + Fresh mutation review of the adaptive probe found three false-green contract gaps: the listing fixture did not distinguish file order from name order, the unexpected-status branches at an intermediate midpoint and isolated probe were diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7586e32ac..3bdf55891 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -27,7 +27,7 @@ are our reading of their documented behavior, not measurements. | Correctness gate | token-exact vs vLLM | reference | own | own | | Architectures | 38 registered, 27 gated | 130+ | 100+ | 100+ | | Downloadable server binaries | ✅ v0.0.2: eight indexed archives with checksums, provenance, manifests, and SBOMs. Windows ZIP downloads do not exist; native CPU/Vulkan lanes await hosted runtime, dry-run, prerelease, and authenticated audit gates | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries | -| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, central `NOMINMAX`, UTF-8, aligned allocation, runtime ISA dispatch. Local closure includes the float-domain DeepSeek probe; hosted compile/runtime/release pending | ✅ | ✅ | ✅ | +| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, central `NOMINMAX`, UTF-8, aligned allocation, runtime ISA dispatch; LTX2 uses C++20 math constants, not `M_PI`. Local DeepSeek float probe green; hosted compile/runtime/release pending | ✅ | ✅ | ✅ | ## Serving and scheduling diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 7e5a4cffc..9713126e6 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include @@ -69,7 +70,7 @@ float Silu(float x) { return x / (1.0f + std::exp(-x)); } // torch.nn.functional.gelu(..., approximate="tanh"), the activation // `activation_fn="gelu-approximate"` selects (gelu_approx.py:10). float GeluTanh(float x) { - const float kBeta = static_cast(std::sqrt(2.0 / M_PI)); + const float kBeta = static_cast(std::sqrt(2.0 / std::numbers::pi_v)); const float kKappa = 0.044715f; const float inner = kBeta * (x + kKappa * x * x * x); return 0.5f * x * (1.0f + std::tanh(inner)); @@ -544,7 +545,8 @@ std::vector FreqGridPytorch(double theta, int64_t n_pos_dims, int64_t dim const float t = i < halfway ? step * static_cast(i) : 1.0f - step * static_cast(n - 1 - i); out[static_cast(i)] = - std::pow(static_cast(theta), t) * static_cast(M_PI / 2.0); + std::pow(static_cast(theta), t) * + static_cast(std::numbers::pi_v / 2.0); } return out; } @@ -559,7 +561,8 @@ std::vector FreqGridNumpy(double theta, int64_t n_pos_dims, int64_t dim) for (int64_t i = 0; i < n; ++i) { // numpy's linspace is arange(n) * step, with the final sample forced to `stop`. const double t = (i == n - 1) ? 1.0 : step * static_cast(i); - out[static_cast(i)] = static_cast(std::pow(theta, t) * (M_PI / 2.0)); + out[static_cast(i)] = + static_cast(std::pow(theta, t) * (std::numbers::pi_v / 2.0)); } return out; } diff --git a/src/vllm/model_executor/models/ltx2_audio_vae.cpp b/src/vllm/model_executor/models/ltx2_audio_vae.cpp index 6c9251319..7998adbd7 100644 --- a/src/vllm/model_executor/models/ltx2_audio_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_vae.cpp @@ -24,16 +24,13 @@ #include #include #include +#include #include #include #include "vllm/model_executor/models/minimax_h3.h" #include "vt/dtype.h" -#ifndef M_PI -#define M_PI 3.14159265358979323846 -#endif - namespace vllm { const std::vector& Ltx2VaeWeights::Get(const std::string& name) const { @@ -433,8 +430,14 @@ std::vector Ltx2HannSincResampleFilter1d(int64_t ratio, int64_t* kernel_s std::max(-static_cast(lowpass_filter_width), std::min(static_cast(lowpass_filter_width), t)); const double window = - std::pow(std::cos(clamped * M_PI / static_cast(lowpass_filter_width) / 2.0), 2.0); - const double sinc = t == 0.0 ? 1.0 : std::sin(M_PI * t) / (M_PI * t); + std::pow(std::cos(clamped * std::numbers::pi_v / + static_cast(lowpass_filter_width) / 2.0), + 2.0); + const double sinc = + t == 0.0 + ? 1.0 + : std::sin(std::numbers::pi_v * t) / + (std::numbers::pi_v * t); filter[static_cast(i)] = static_cast(sinc * window * rolloff / static_cast(ratio)); } @@ -1042,7 +1045,8 @@ std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, std::vector window(static_cast(n_fft)); for (int64_t i = 0; i < n_fft; ++i) { window[static_cast(i)] = - 0.5 - 0.5 * std::cos(2.0 * M_PI * static_cast(i) / static_cast(n_fft)); + 0.5 - 0.5 * std::cos(2.0 * std::numbers::pi_v * static_cast(i) / + static_cast(n_fft)); } // `center=True, pad_mode="reflect"`: pad n_fft/2 on BOTH sides, so frame 0 is @@ -1080,7 +1084,8 @@ std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, for (int64_t f = 0; f < n_freqs; ++f) { double real = 0.0; double imag = 0.0; - const double omega = -2.0 * M_PI * static_cast(f) / static_cast(n_fft); + const double omega = -2.0 * std::numbers::pi_v * static_cast(f) / + static_cast(n_fft); for (int64_t i = 0; i < n_fft; ++i) { const double angle = omega * static_cast(i); real += frame[static_cast(i)] * std::cos(angle); diff --git a/src/vllm/model_executor/models/ltx2_video_vae.cpp b/src/vllm/model_executor/models/ltx2_video_vae.cpp index c7cb88825..592f5b8b4 100644 --- a/src/vllm/model_executor/models/ltx2_video_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_video_vae.cpp @@ -55,10 +55,6 @@ #include "vllm/model_executor/models/minimax_h3.h" #include "vt/dtype.h" -#ifndef M_PI -#define M_PI 3.14159265358979323846 -#endif - namespace vllm { namespace { From bf91fb536e45d591acbaede9ab0a42163776a147 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:15:17 +0000 Subject: [PATCH 27/35] spec(release): track VideoEngine Windows regression Record issue #648 and constrain the repair to non-throwing C++20 filesystem queries with unchanged path classification. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/roadmap_v1.md | 1 + .agents/specs/release-dry-run-gate-repairs.md | 26 +++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index ad0871a7a..feef79905 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -107,6 +107,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#540](https://github.com/mudler/vllm.cpp/issues/540) | `ENG-RELEASE-WINDOWS` | Windows Vulkan strict build rejects shadowed KV-cache test locals | bug | | [#599](https://github.com/mudler/vllm.cpp/issues/599) | `ENG-RELEASE-WINDOWS` | Windows release contract failure fixture is not portable under PowerShell on Linux | bug | | [#645](https://github.com/mudler/vllm.cpp/issues/645) | `ENG-RELEASE-WINDOWS` | LTX2 sources reintroduce non-portable `M_PI` and break the Windows release gate | bug | +| [#648](https://github.com/mudler/vllm.cpp/issues/648) | `ENG-RELEASE-WINDOWS` | `VideoEngine` uses unguarded POSIX `stat` in the native Windows build | bug | | [#515](https://github.com/mudler/vllm.cpp/issues/515) | `ENG-RECORD-CONFLICT-SURFACES` | check-doc-checkpoint treats every root `CMakeLists.txt` edit as a usage change, so adding a source file demands a `docs/USAGE.md` edit with nothing true to say | bug | | [#547](https://github.com/mudler/vllm.cpp/issues/547) | — | GB10 reports `UnifiedMemory()` true, so `ReferenceTierEligible(kCUDA)` runs the CPU host kernel over `cudaMalloc` pointers; it needs `DeviceMemoryIsHostAddressable()` | bug | | [#569](https://github.com/mudler/vllm.cpp/issues/569) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | The Nemotron revision pin was existence-only: a substituted checkpoint still resolved, and W6's token gate cannot see that | bug | diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 9a42b0c39..5f8ebbd1d 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -657,6 +657,32 @@ affected translation units and linked `test_ltx2` and `test_ltx2_vae`; their upstream-golden numerical gates pass 30/30 cases with 1627 assertions and 36/36 cases with 3039 assertions, respectively. +### Current-main VideoEngine portability regression: issue #648 + +The configured direct Windows checker on current `main` reports three sites in +`src/vllm/multimodal/video_engine.cpp`: the unconditional `` include +and the `::stat` calls in `IsDir` and `Exists`. These landed with the generalized +video seam at `cefacd2d00cb9b4776331cd213116773cd97f811` and cannot reach the +native MSVC release build. + +Replace that POSIX dependency with C++20 `std::filesystem` queries using the +non-throwing `std::error_code` overloads. `IsDir` remains true only for a +directory; `Exists` remains true for any existing filesystem entry; missing or +uninspectable paths remain false. Do not add a Windows-only branch, guard the +POSIX include, exempt the file from the checker, or change family resolution. + +RED evidence is the configured direct checker naming all three sites. Focused +behavior must cover an existing directory, a regular file, and a missing path +through the public VideoEngine resolution surface; add only the smallest test +needed if existing coverage cannot prove each classification. Green requires +the direct checker, the complete portability and combined release suites, a +clean build of the affected translation unit, and the VideoEngine tests. + +Fresh review must restore the POSIX implementation to prove the direct checker +is red and mutate directory classification to an existence-only query to prove +behavior coverage rejects a regular file. Stop with `NEEDS_DECISION` if the +portable implementation changes an observable resolution result or error. + Fresh mutation review of the adaptive probe found three false-green contract gaps: the listing fixture did not distinguish file order from name order, the unexpected-status branches at an intermediate midpoint and isolated probe were From 90e1d771db114192d48e6ff5a34f2ebe7dff7dbb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:26:19 +0000 Subject: [PATCH 28/35] fix(windows): port VideoEngine paths (#648) The generalized video seam used POSIX stat for path classification. This prevents its translation unit from reaching native MSVC builds. Use non-throwing C++20 filesystem queries. They preserve the existing refusals for missing and uninspectable paths. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 20 +++++++++++++++++++ src/vllm/multimodal/video_engine.cpp | 12 +++++------ tests/vllm/multimodal/test_video_engine.cpp | 19 ++++++++++++++++++ 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index 5f8ebbd1d..d28e324b6 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -683,6 +683,26 @@ is red and mutate directory classification to an existence-only query to prove behavior coverage rejects a regular file. Stop with `NEEDS_DECISION` if the portable implementation changes an observable resolution result or error. +#### #648 implementation outcome + +The regression came from the generalized VideoEngine seam using POSIX `stat` +for two private classification helpers even though C++20 filesystem support is +already part of the project baseline. The direct portability checker failed at +the unconditional `` include and both `::stat` calls before the +repair. `IsDir` now uses the non-throwing `std::filesystem::is_directory` +overload and `Exists` uses the corresponding `exists` overload; both inspect an +`std::error_code`, so missing and uninspectable paths still return false rather +than throwing. + +The focused public-surface case distinguishes a directory without a shard +index, a regular non-checkpoint file, and a missing path through +`ReadVideoCheckpointTensorNames`. This pins the existing three refusal reasons +and makes an existence-only directory predicate misclassify the regular file. +After the repair the direct checker passed, the VideoEngine target built and +its 12 cases / 260 assertions passed, and the combined Windows metadata, +release-pipeline, and portability suite passed all 130 tests. No family +resolution rule or public error changed. + Fresh mutation review of the adaptive probe found three false-green contract gaps: the listing fixture did not distinguish file order from name order, the unexpected-status branches at an intermediate midpoint and isolated probe were diff --git a/src/vllm/multimodal/video_engine.cpp b/src/vllm/multimodal/video_engine.cpp index 656a312f1..9736ebdfa 100644 --- a/src/vllm/multimodal/video_engine.cpp +++ b/src/vllm/multimodal/video_engine.cpp @@ -11,15 +11,15 @@ #include #include #include +#include #include #include #include #include +#include #include #include -#include - #include #include "vllm/entrypoints/openai/video_api.h" @@ -55,13 +55,13 @@ std::vector& RegistryStorage() { const std::vector& OrderedRegistry() { return RegistryStorage(); } bool IsDir(const std::string& path) { - struct stat st {}; - return ::stat(path.c_str(), &st) == 0 && S_ISDIR(st.st_mode); + std::error_code ec; + return std::filesystem::is_directory(path, ec) && !ec; } bool Exists(const std::string& path) { - struct stat st {}; - return ::stat(path.c_str(), &st) == 0; + std::error_code ec; + return std::filesystem::exists(path, ec) && !ec; } std::string StripTrailingSlash(const std::string& dir) { diff --git a/tests/vllm/multimodal/test_video_engine.cpp b/tests/vllm/multimodal/test_video_engine.cpp index 8854fd3a8..3fcc433db 100644 --- a/tests/vllm/multimodal/test_video_engine.cpp +++ b/tests/vllm/multimodal/test_video_engine.cpp @@ -152,6 +152,25 @@ TEST_CASE("video engine registry: detection resolves the H3 checkpoint by what i CHECK(vllm::multimodal::DetectVideoFamilies(not_a_dit).empty()); } +TEST_CASE("video checkpoint reader: directory, regular file, and missing path stay distinct") { + SeamWorkspace ws; + std::vector names; + std::string why; + + CHECK_FALSE(vllm::multimodal::ReadVideoCheckpointTensorNames(ws.fixture, &names, &why)); + CHECK(why.find("directory holds neither") != std::string::npos); + + why.clear(); + CHECK_FALSE(vllm::multimodal::ReadVideoCheckpointTensorNames( + ws.fixture + "/video_vae_config.json", &names, &why)); + CHECK(why.find("neither GGUF nor safetensors") != std::string::npos); + + why.clear(); + CHECK_FALSE(vllm::multimodal::ReadVideoCheckpointTensorNames( + ws.root + "/missing-checkpoint", &names, &why)); + CHECK(why == "no such file or directory"); +} + // ─── the byte-identity gate: the ABSTRACT seam lands on the pre-fold bytes ─── TEST_CASE("video engine seam: an auto-detected H3 render reproduces the pre-fold goldens") { SeamWorkspace ws; From 89d610529c513ce39c7c49778d93a9aeea5561d0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 18:18:28 +0000 Subject: [PATCH 29/35] spec(release): narrow isolated OpenAI crash witness (#537) Record the hosted prefix result and bind the next diagnostic to the isolated explicit-CPU serving case. Require captured output plus phase witnesses, then removal and a RED regression before any production repair. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index d28e324b6..decc28870 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -617,6 +617,23 @@ three confirmation statuses, and the dependency classification. The existing unfiltered full-suite invocation remains unchanged and still runs afterwards. Any other probe status stops the release job rather than being classified. +Hosted run `31728014706` resolved that boundary to source-order case 47/54, +`api_server: an explicit-cpu device-selected engine serves /v1/completions`. +Both its prefix and its isolated invocation terminate with the exact native +fast-fail status, while the confirmed predecessor succeeds. The defect is +therefore isolated to that test or the production path it exercises, not a +cross-test lifetime dependency. + +The next diagnostic is limited to that isolated process. Preserve and print +its already-captured output, and add flushed phase witnesses around construction +of `LoadedEngine`, construction of the async serving stack, completion dispatch, +response validation, and scope teardown. Do not change timing, add sleeps, +weaken assertions, or accept this instrumentation as the repair. The hosted +Windows CPU/Vulkan result must identify the last completed phase. Then remove +the diagnostic and add the smallest RED regression for the actual owner before +changing production code. If the phase evidence does not distinguish an owner, +stop and extend the spec rather than guessing. + ### Current-main portability regression: issue #645 Current `main` at `cefacd2d00cb9b4776331cd213116773cd97f811` added LTX2 From 77ba6fa018d88ba506f60c6018a5408d374a4af7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 18:37:06 +0000 Subject: [PATCH 30/35] test(release): expose isolated OpenAI crash phase (#537) Forward the isolated prefix probe output and add flushed phase witnesses around the explicit-CPU serving test so hosted Windows identifies the last completed lifetime boundary without changing the full-suite gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- scripts/build-windows-release.ps1 | 23 ++++++- .../scripts/test_release_windows_metadata.py | 58 +++++++++++++++++ .../entrypoints/openai/test_api_server.cpp | 63 ++++++++++++------- 3 files changed, 121 insertions(+), 23 deletions(-) diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index e27ca6990..f1abf10f4 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -342,6 +342,11 @@ function Invoke-OpenAiPrefixBisect { } $isolatedResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` -First $firstBad -Last $firstBad -Runner $Runner + Write-Host "OpenAI isolated output: begin case=$firstBad" + foreach ($isolatedOutputLine in @($isolatedResult.Output)) { + Write-Host ([string]$isolatedOutputLine) + } + Write-Host "OpenAI isolated output: end case=$firstBad" if ($isolatedResult.ExitCode -eq 0) { $dependency = "cumulative" } elseif ($isolatedResult.ExitCode -eq $expectedFastFailStatus) { @@ -418,7 +423,12 @@ function Invoke-OpenAiPrefixBisectContractTests { } else { 0 } - return [pscustomobject]@{ ExitCode = $exitCode; Output = @() } + $output = if ($first -eq $firstBad -and $last -eq $firstBad) { + @("OPENAI_EXPLICIT_CPU_PHASE: fixture-isolated") + } else { + @() + } + return [pscustomobject]@{ ExitCode = $exitCode; Output = @($output) } }.GetNewClosure() $captured = @(& { @@ -475,6 +485,17 @@ function Invoke-OpenAiPrefixBisectContractTests { throw "OpenAI prefix bisect omitted confirmation $($bounds -join '..')" } } + $forwardedOutput = @( + "OpenAI isolated output: begin case=27", + "OPENAI_EXPLICIT_CPU_PHASE: fixture-isolated", + "OpenAI isolated output: end case=27" + ) + foreach ($line in $forwardedOutput) { + if (@($captured | Where-Object { [string]$_ -ceq $line }).Count -ne 1) { + throw "OpenAI prefix bisect did not forward isolated output exactly once: $line" + } + } + Write-Host "OpenAI prefix bisect isolated output forwarding contract OK" $midpointInjected = $false $unexpectedMidpointRunner = { diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 6623a50c1..e7bdbf8cf 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -204,6 +204,63 @@ def test_openai_prefix_bisect_pins_adaptive_exact_status_contract(self) -> None: with self.subTest(guarantee=guarantee): self.assertIn(guarantee, contract) + def test_openai_isolated_probe_forwards_stable_phase_witness_output(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text( + encoding="utf-8" + ) + helper_start = script.find("function Invoke-OpenAiPrefixBisect {") + self.assertNotEqual(helper_start, -1, "adaptive prefix bisect is missing") + helper_end = script.index("\n}\n", helper_start) + len("\n}\n") + helper = script[helper_start:helper_end] + for statement in ( + 'Write-Host "OpenAI isolated output: begin case=$firstBad"', + "foreach ($isolatedOutputLine in @($isolatedResult.Output))", + "Write-Host ([string]$isolatedOutputLine)", + 'Write-Host "OpenAI isolated output: end case=$firstBad"', + ): + with self.subTest(statement=statement): + self.assertEqual(helper.count(statement), 1) + + source = ( + ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" + ).read_text(encoding="utf-8") + for phase in ( + "before-loaded-engine", + "after-loaded-engine", + "before-serving-stack", + "after-serving-stack", + "before-completion-dispatch", + "after-completion-dispatch", + "before-response-validation", + "after-response-validation", + "before-scope-teardown", + "after-scope-teardown", + ): + with self.subTest(phase=phase): + self.assertEqual( + source.count(f'OpenAiExplicitCpuPhaseWitness("{phase}")'), 1 + ) + witness_start = source.index( + "void OpenAiExplicitCpuPhaseWitness(const char* phase) noexcept {" + ) + witness_end = source.index("\n}", witness_start) + witness = source[witness_start:witness_end] + self.assertEqual(witness.count('"OPENAI_EXPLICIT_CPU_PHASE: %s\\n"'), 1) + self.assertEqual(witness.count("std::fflush(stderr)"), 1) + contract_start = script.index( + "function Invoke-OpenAiPrefixBisectContractTests {" + ) + contract_end = script.index("\n}\n", contract_start) + len("\n}\n") + contract = script[contract_start:contract_end] + for forwarded_line in ( + "OpenAI isolated output: begin case=27", + "OPENAI_EXPLICIT_CPU_PHASE: fixture-isolated", + "OpenAI isolated output: end case=27", + "OpenAI prefix bisect isolated output forwarding contract OK", + ): + with self.subTest(forwarded_line=forwarded_line): + self.assertIn(forwarded_line, contract) + def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8") handler_start = main.index( @@ -401,6 +458,7 @@ def test_contract_test_executes_real_children_under_powershell_core(self) -> Non "OpenAI prefix bisect unexpected midpoint status contract OK", "OpenAI prefix bisect unexpected isolated status contract OK", "OpenAI prefix bisect diagnostic schema contract OK", + "OpenAI prefix bisect isolated output forwarding contract OK", ): with self.subTest(proof=proof): self.assertIn(proof, result.stdout) diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 408669293..dbe2177f6 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -115,6 +115,11 @@ using vllm::test::ScopedServerThread; namespace { +void OpenAiExplicitCpuPhaseWitness(const char* phase) noexcept { + std::fprintf(stderr, "OPENAI_EXPLICIT_CPU_PHASE: %s\n", phase); + std::fflush(stderr); +} + class TeardownProbeMarker { public: explicit TeardownProbeMarker(const char* message) : message_(message) {} @@ -2402,28 +2407,42 @@ TEST_CASE("api_server: an explicit-cpu device-selected engine serves /v1/complet // The server's own parse of `--device cpu` (an unknown name throws there at // startup; pinned in test_loaded_engine_dense.cpp). params.device = vllm::DeviceFromString("cpu"); - vllm::entrypoints::LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), - params); - // The observable seam: the runner of the explicitly-cpu engine is on the CPU - // device (on a CUDA build this is the force-CPU pin; auto would select CUDA). - CHECK(loaded.runner().device().type == vt::DeviceType::kCPU); - - OpenAIServingModels models("test-model"); - OpenAIServingCompletion completion(loaded.async_engine(), "test-model", - /*enable_force_include_usage=*/false); - OpenAIServingChat chat(loaded.async_engine(), "test-model", InVocabChatPrompt, - "hermes", /*reasoning_parser_name=*/std::string(), - /*enable_force_include_usage=*/false); - ApiServer server(completion, chat, models, "9.9.9"); - - const std::string body = - R"({"model":"test-model","prompt":"hello","max_tokens":5,"temperature":0.0})"; - ApiServer::DispatchResult r = server.handle_completions(body); - CHECK(r.status == 200); - json j = json::parse(r.body); - CHECK(j.at("object") == "text_completion"); - CHECK(j.at("choices").at(0).at("finish_reason") == "length"); - CHECK(j.at("usage").at("completion_tokens") == 5); + OpenAiExplicitCpuPhaseWitness("before-loaded-engine"); + { + vllm::entrypoints::LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), + params); + OpenAiExplicitCpuPhaseWitness("after-loaded-engine"); + // The observable seam: the runner of the explicitly-cpu engine is on the + // CPU device (on a CUDA build this is the force-CPU pin; auto would select + // CUDA). + CHECK(loaded.runner().device().type == vt::DeviceType::kCPU); + + OpenAiExplicitCpuPhaseWitness("before-serving-stack"); + OpenAIServingModels models("test-model"); + OpenAIServingCompletion completion(loaded.async_engine(), "test-model", + /*enable_force_include_usage=*/false); + OpenAIServingChat chat( + loaded.async_engine(), "test-model", InVocabChatPrompt, "hermes", + /*reasoning_parser_name=*/std::string(), + /*enable_force_include_usage=*/false); + ApiServer server(completion, chat, models, "9.9.9"); + OpenAiExplicitCpuPhaseWitness("after-serving-stack"); + + const std::string body = + R"({"model":"test-model","prompt":"hello","max_tokens":5,"temperature":0.0})"; + OpenAiExplicitCpuPhaseWitness("before-completion-dispatch"); + ApiServer::DispatchResult r = server.handle_completions(body); + OpenAiExplicitCpuPhaseWitness("after-completion-dispatch"); + OpenAiExplicitCpuPhaseWitness("before-response-validation"); + CHECK(r.status == 200); + json j = json::parse(r.body); + CHECK(j.at("object") == "text_completion"); + CHECK(j.at("choices").at(0).at("finish_reason") == "length"); + CHECK(j.at("usage").at("completion_tokens") == 5); + OpenAiExplicitCpuPhaseWitness("after-response-validation"); + OpenAiExplicitCpuPhaseWitness("before-scope-teardown"); + } + OpenAiExplicitCpuPhaseWitness("after-scope-teardown"); } // ─── /v1/embeddings (ARCH-ONE-SURFACE ROW 6) ───────────────────────────────── From 5ee2c50dfcbcd218094b01654c445cbb4fcdae97 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 19:22:49 +0000 Subject: [PATCH 31/35] spec(release): split Windows construction witness (#537) Record the hosted phase boundary and bind the final diagnostic split across synthetic weight creation, tokenizer creation, and LoadedEngine construction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/specs/release-dry-run-gate-repairs.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index decc28870..e10db4d55 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -634,6 +634,27 @@ the diagnostic and add the smallest RED regression for the actual owner before changing production code. If the phase evidence does not distinguish an owner, stop and extend the spec rather than guessing. +Hosted CPU run `31732971268` printed only +`OPENAI_EXPLICIT_CPU_PHASE: before-loaded-engine` in both the isolated probe and +the unchanged full-suite invocation, then terminated with `0xC0000409`. It did +not print `after-loaded-engine`. This excludes the async serving stack, +completion dispatch, response validation, and teardown, but it does not yet +distinguish the three expressions in the construction statement: synthetic +weight creation, tokenizer-fixture creation, and the `LoadedEngine` constructor +itself. C++ does not impose a useful ordering between those argument +evaluations, so their absence cannot identify which one terminated. + +The final diagnostic split keeps the same test and values but materializes the +synthetic weights and tokenizer into named locals, with flushed witnesses before +and after each factory and before and after `LoadedEngine` construction. It +must move those locals into the constructor so ownership matches the original +by-value call. No production source, timing, assertion, environment, or engine +parameter may change. Hosted CPU and Vulkan must agree on the last completed +factory/construction phase. If both factories complete and construction still +fast-fails, stop and spec constructor-internal attribution rather than guessing; +otherwise remove every prefix/phase diagnostic and write the smallest RED +regression for the failing factory before repairing it. + ### Current-main portability regression: issue #645 Current `main` at `cefacd2d00cb9b4776331cd213116773cd97f811` added LTX2 From d39c43d62d4cd0cbb4397d917031810f34cc228f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 19:34:20 +0000 Subject: [PATCH 32/35] test(release): split Windows construction witness (#537) Materialize synthetic weights and the tokenizer fixture before LoadedEngine construction so hosted Windows logs identify the exact fast-failing phase while preserving values and ownership. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .../scripts/test_release_windows_metadata.py | 32 +++++++++++++++++++ .../entrypoints/openai/test_api_server.cpp | 10 ++++-- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index e7bdbf8cf..79887693b 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -224,7 +224,17 @@ def test_openai_isolated_probe_forwards_stable_phase_witness_output(self) -> Non source = ( ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" ).read_text(encoding="utf-8") + case_start = source.index( + 'TEST_CASE("api_server: an explicit-cpu device-selected engine serves ' + '/v1/completions") {' + ) + case_end = source.index("\n}\n", case_start) + len("\n}\n") + explicit_cpu_case = source[case_start:case_end] for phase in ( + "before-make-weights", + "after-make-weights", + "before-build-fixture", + "after-build-fixture", "before-loaded-engine", "after-loaded-engine", "before-serving-stack", @@ -240,6 +250,28 @@ def test_openai_isolated_probe_forwards_stable_phase_witness_output(self) -> Non self.assertEqual( source.count(f'OpenAiExplicitCpuPhaseWitness("{phase}")'), 1 ) + ordered_construction = ( + 'OpenAiExplicitCpuPhaseWitness("before-make-weights")', + "Qwen3_5MoeWeights weights = MakeWeights(c)", + 'OpenAiExplicitCpuPhaseWitness("after-make-weights")', + 'OpenAiExplicitCpuPhaseWitness("before-build-fixture")', + "Tokenizer tokenizer = BuildFixture()", + 'OpenAiExplicitCpuPhaseWitness("after-build-fixture")', + 'OpenAiExplicitCpuPhaseWitness("before-loaded-engine")', + "std::move(weights)", + "std::move(tokenizer)", + 'OpenAiExplicitCpuPhaseWitness("after-loaded-engine")', + ) + positions = [] + for statement in ordered_construction: + with self.subTest(statement=statement): + self.assertEqual(explicit_cpu_case.count(statement), 1) + positions.append(explicit_cpu_case.index(statement)) + self.assertEqual(positions, sorted(positions)) + self.assertNotIn( + "LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), params)", + explicit_cpu_case, + ) witness_start = source.index( "void OpenAiExplicitCpuPhaseWitness(const char* phase) noexcept {" ) diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index dbe2177f6..b9baf80b7 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -2407,10 +2407,16 @@ TEST_CASE("api_server: an explicit-cpu device-selected engine serves /v1/complet // The server's own parse of `--device cpu` (an unknown name throws there at // startup; pinned in test_loaded_engine_dense.cpp). params.device = vllm::DeviceFromString("cpu"); + OpenAiExplicitCpuPhaseWitness("before-make-weights"); + Qwen3_5MoeWeights weights = MakeWeights(c); + OpenAiExplicitCpuPhaseWitness("after-make-weights"); + OpenAiExplicitCpuPhaseWitness("before-build-fixture"); + Tokenizer tokenizer = BuildFixture(); + OpenAiExplicitCpuPhaseWitness("after-build-fixture"); OpenAiExplicitCpuPhaseWitness("before-loaded-engine"); { - vllm::entrypoints::LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), - params); + vllm::entrypoints::LoadedEngine loaded( + c, std::move(weights), std::move(tokenizer), params); OpenAiExplicitCpuPhaseWitness("after-loaded-engine"); // The observable seam: the runner of the explicitly-cpu engine is on the // CPU device (on a CUDA build this is the force-CPU pin; auto would select From f6b1313cbf9bbf30a1bbc7f394d6e240e3b4f5e3 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 20:42:49 +0000 Subject: [PATCH 33/35] docs(release): specify Windows constructor attribution (#537) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GPT-5 [CODEX] --- .agents/specs/release-dry-run-gate-repairs.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/.agents/specs/release-dry-run-gate-repairs.md b/.agents/specs/release-dry-run-gate-repairs.md index e10db4d55..b56f2f015 100644 --- a/.agents/specs/release-dry-run-gate-repairs.md +++ b/.agents/specs/release-dry-run-gate-repairs.md @@ -655,6 +655,75 @@ fast-fails, stop and spec constructor-internal attribution rather than guessing; otherwise remove every prefix/phase diagnostic and write the smallest RED regression for the failing factory before repairing it. +Hosted run +[`31737430012`](https://github.com/mudler/vllm.cpp/actions/runs/31737430012) +reached that stop condition. CPU job +[`94572345175`](https://github.com/mudler/vllm.cpp/actions/runs/31737430012/job/94572345175) +and Vulkan job +[`94572345101`](https://github.com/mudler/vllm.cpp/actions/runs/31737430012/job/94572345101) +agree in both the isolated probe and the unchanged full-suite invocation: each +printed `before-make-weights`, `after-make-weights`, `before-build-fixture`, +`after-build-fixture`, and `before-loaded-engine`, then fast-failed with signed +status `-1073740791` (`0xC0000409`) without printing `after-loaded-engine`. +Synthetic weight construction and tokenizer-fixture construction are therefore +excluded. The failure is inside `LoadedEngine` construction, before its first +constructor-body log, on both explicit CPU and Vulkan release builds. + +The next increment is diagnostic-only constructor-internal attribution. Enable +it explicitly only for the fresh-process isolated case-47 invocation; the +ordinary full-suite command and every production invocation remain unmodified +and do not enable the witnesses. Emit and flush a stable before/after phase for +the delegating `MakeQwen3_5MoeLoadedModel` call, then for every private +`LoadedEngine` initialization stage in declaration order through `engine_`: +`hash_ready_`, `config_`, `resolved_spec_config_`, `dflash_draft_`, `model_`, +the default `kv_connector_`, `tokenizer_`, `kv_cfg_`, `max_model_len_`, +`max_num_batched_tokens_`, `prefix_caching_enabled_`, +`jump_forward_enabled_`, `runner_`, `async_scheduling_enabled_`, +`max_concurrent_batches_`, `structured_output_manager_`, `scheduler_`, +`executor_`, `engine_core_`, `input_processor_`, `output_processor_`, +`block_hasher_`, and `engine_`. The contract must pin this sequence to the real +member-declaration order and require exactly one matching before/after pair for +every completed stage. Instrumentation must preserve every initializer value, +move, evaluation dependency, and construction order; it may not introduce a +replacement constructor path. + +Make the same isolated diagnostic dense below `runner_` so a failure there is +attributed in this hosted run rather than another coarse loop. Bracket the +`GPUModelRunner` member-initializer boundary through `input_batch_`, then its +constructor-body assignments, `async_input_combine_` resolution, pooling-model +branch, `initialize_kv_cache`, and `ModelRegistry::Prepare`. Inside +`initialize_kv_cache`, bracket the scalar/state-slot setup, KV-group scan, +Mamba shape/dtype validation, full-attention spec geometry, residency/buffer +setup, and each layer-indexed allocation family separately: GDN SSM storage, +GDN convolution storage, and full-attention storage. Also bracket creation of +the full-attention views, optional draft-attention storage, and GDN state views, +so success of an allocation cannot be confused with failure while viewing it. +Each nested marker names its function, stage, and layer or group index where +applicable. The first before marker lacking its matching after marker must +identify one exact initializer, function, or allocation/view family. + +These witnesses are temporary evidence, not a repair. They may not add an +assertion, sleep, timeout, catch, parameter change, device substitution, or +production semantic change. Because `0xC0000409` is an MSVC fail-fast and the +earlier `std::terminate` marker did not fire, the isolated Windows process must +also install a flushed `_set_invalid_parameter_handler` witness that reports +the CRT expression, function, file, and line. Install a similarly flushed +`_set_purecall_handler` witness when the existing MSVC runtime exposes it +without a new dependency. These Windows-only handlers are diagnostic: they may +observe but must not suppress or recover from the original failure, and a +surviving process restores the prior handlers before exit. If neither the +first incomplete phase nor a CRT witness names one owner, collect a native +Windows crash dump and symbolized stack from the same isolated workload instead +of inferring one. Before any actual repair, remove all adaptive-prefix, phase, +and CRT diagnostics, write the smallest automated regression that is RED for +the attributed defect, and observe that RED against the unfixed code. Only then +may a fresh implementation repair the root cause. + +The device-leakage failure reported beside these jobs is independent. Issue +[#553](https://github.com/mudler/vllm.cpp/issues/553) fixes it on current `main` +at `11cc1d5896b480a1b652db9249319242053aca93`; this campaign absorbs that change +when it merges current `main`. This #537 diagnostic must not edit that scope. + ### Current-main portability regression: issue #645 Current `main` at `cefacd2d00cb9b4776331cd213116773cd97f811` added LTX2 From ee09fb36bf14027cfb6b8d05f153160149100046 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:25:38 +0000 Subject: [PATCH 34/35] test(release): attribute Windows constructor fast-fail (#537) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GPT-5 [CODEX] --- docs/USAGE.md | 6 + scripts/build-windows-release.ps1 | 53 ++++- scripts/env-doc-allowlist.txt | 1 + src/vllm/diagnostics/constructor_witness.h | 89 ++++++++ src/vllm/entrypoints/model_loader.cpp | 205 +++++++++++------- src/vllm/v1/worker/gpu/runner.cpp | 92 +++++++- .../scripts/test_release_windows_metadata.py | 165 ++++++++++++++ .../entrypoints/openai/test_api_server.cpp | 75 +++++++ 8 files changed, 597 insertions(+), 89 deletions(-) create mode 100644 src/vllm/diagnostics/constructor_witness.h diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..a9f306dc1 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -694,6 +694,12 @@ imports before running the staged executable's `--help`, forced-tier, or HTTP shutdown smokes. The Win32 console-control regression uses bounded waits so a teardown failure reports an error instead of hanging the gate. +While issue #537 is being attributed, the Windows release helper also runs an +internal constructor witness only in the fresh-process isolated OpenAI case +found by its prefix probe. The helper restores the process environment before +continuing; the ordinary prefix probes, complete OpenAI suite, staged server, +and production invocations do not enable or inherit that diagnostic. + The CUDA graph-replay profiler and its FIFO diagnostic controls remain POSIX-only and are not exposed by native Windows server builds. Native Windows process launch, environment updates, process IDs, and console shutdown stay on diff --git a/scripts/build-windows-release.ps1 b/scripts/build-windows-release.ps1 index f1abf10f4..eb000a6ff 100644 --- a/scripts/build-windows-release.ps1 +++ b/scripts/build-windows-release.ps1 @@ -340,8 +340,16 @@ function Invoke-OpenAiPrefixBisect { if ($badPrefixResult.ExitCode -ne $expectedFastFailStatus) { throw "OpenAI confirmed bad prefix exited with status $($badPrefixResult.ExitCode) instead of $expectedFastFailStatus" } - $isolatedResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` - -First $firstBad -Last $firstBad -Runner $Runner + $diagnosticEnvironmentName = "VLLM_WINDOWS_CTOR_DIAGNOSTIC" + $priorDiagnosticEnvironment = [Environment]::GetEnvironmentVariable( + $diagnosticEnvironmentName, "Process") + try { + [Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, "1", "Process") + $isolatedResult = Invoke-OpenAiPrefixRange -Program $TestProgram ` + -First $firstBad -Last $firstBad -Runner $Runner + } finally { + [Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, $priorDiagnosticEnvironment, "Process") + } Write-Host "OpenAI isolated output: begin case=$firstBad" foreach ($isolatedOutputLine in @($isolatedResult.Output)) { Write-Host ([string]$isolatedOutputLine) @@ -397,6 +405,8 @@ function Invoke-OpenAiPrefixBisectContractTests { $calls.Add([pscustomobject]@{ Program = $Program Arguments = @($Arguments) + DiagnosticEnvironment = [Environment]::GetEnvironmentVariable( + "VLLM_WINDOWS_CTOR_DIAGNOSTIC", "Process") }) | Out-Null if ($Arguments -contains "--list-test-cases") { $orderArguments = @($Arguments | Where-Object { $_ -like "--order-by=*" }) @@ -431,10 +441,41 @@ function Invoke-OpenAiPrefixBisectContractTests { return [pscustomobject]@{ ExitCode = $exitCode; Output = @($output) } }.GetNewClosure() - $captured = @(& { - Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` - -Runner $runner - } 6>&1) + $contractDiagnosticEnvironment = [Environment]::GetEnvironmentVariable( + "VLLM_WINDOWS_CTOR_DIAGNOSTIC", "Process") + try { + [Environment]::SetEnvironmentVariable( + "VLLM_WINDOWS_CTOR_DIAGNOSTIC", "fixture-prior", "Process") + $captured = @(& { + Invoke-OpenAiPrefixBisect -TestProgram "fake-openai-test.exe" ` + -Runner $runner + } 6>&1) + if ([Environment]::GetEnvironmentVariable( + "VLLM_WINDOWS_CTOR_DIAGNOSTIC", "Process") -cne + "fixture-prior") { + throw "OpenAI constructor diagnostic environment was not restored" + } + } finally { + [Environment]::SetEnvironmentVariable( + "VLLM_WINDOWS_CTOR_DIAGNOSTIC", $contractDiagnosticEnvironment, + "Process") + } + Write-Host "OpenAI constructor diagnostic restoration contract OK" + $diagnosticCalls = @($calls | Where-Object { + $_.DiagnosticEnvironment -ceq "1" + }) + if ($diagnosticCalls.Count -ne 1 -or + $diagnosticCalls[0].Arguments -notcontains "--first=27" -or + $diagnosticCalls[0].Arguments -notcontains "--last=27") { + throw "OpenAI constructor diagnostic was not limited to the isolated case" + } + $ordinaryDiagnosticCalls = @($calls | Where-Object { + $_.DiagnosticEnvironment -ne "1" + }) + if ($ordinaryDiagnosticCalls.Count -ne ($calls.Count - 1)) { + throw "OpenAI constructor diagnostic leaked into an ordinary probe" + } + Write-Host "OpenAI constructor diagnostic activation contract OK" $results = @($captured | Where-Object { $null -ne $_.PSObject.Properties["FirstBad"] }) diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 36cab9ceb..cf10cd7a5 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -1,6 +1,7 @@ VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH VLLM_GEMMA4_MM_DEBUG VLLM_MM_TOWER_PROFILE +VLLM_WINDOWS_CTOR_DIAGNOSTIC VT_ARCH_TACTIC_STATS VT_ASYNC_EXECUTOR_NO_DBUF VT_ASYNC_EXECUTOR_POISON diff --git a/src/vllm/diagnostics/constructor_witness.h b/src/vllm/diagnostics/constructor_witness.h new file mode 100644 index 000000000..cdfcdadf9 --- /dev/null +++ b/src/vllm/diagnostics/constructor_witness.h @@ -0,0 +1,89 @@ +#ifndef VLLM_DIAGNOSTICS_CONSTRUCTOR_WITNESS_H_ +#define VLLM_DIAGNOSTICS_CONSTRUCTOR_WITNESS_H_ + +#include +#include +#include +#include +#include +#include + +namespace vllm::diagnostics { + +constexpr const char* kConstructorWitnessEnvironment = "VLLM_WINDOWS_CTOR_DIAGNOSTIC"; + +inline bool ConstructorWitnessEnabled() noexcept { + const char* value = std::getenv(kConstructorWitnessEnvironment); + return value != nullptr && value[0] == '1' && value[1] == '\0'; +} + +inline void ConstructorWitness(const char* function, const char* stage, + const char* phase, + long long index = -1) noexcept { + if (!ConstructorWitnessEnabled()) return; + if (index < 0) { + std::fprintf(stderr, + "VLLM_CTOR_DIAGNOSTIC: function=%s stage=%s phase=%s\n", + function, stage, phase); + } else { + std::fprintf( + stderr, + "VLLM_CTOR_DIAGNOSTIC: function=%s stage=%s phase=%s index=%lld\n", + function, stage, phase, index); + } + std::fflush(stderr); +} + +inline void ConstructorWitnessBefore(const char* function, const char* stage, + long long index = -1) noexcept { + ConstructorWitness(function, stage, "before", index); +} + +inline void ConstructorWitnessAfter(const char* function, const char* stage, + long long index = -1) noexcept { + ConstructorWitness(function, stage, "after", index); +} + +// A mem-initializer is one full expression. This temporary is constructed before +// that expression and destroyed only after the target member was successfully +// initialized. During exception unwinding it deliberately omits "after", so the +// first unmatched marker remains the failing initializer. +class ConstructorWitnessPhase { + public: + ConstructorWitnessPhase(const char* function, const char* stage) noexcept + : function_(function), + stage_(stage), + enabled_(ConstructorWitnessEnabled()), + uncaught_exceptions_(std::uncaught_exceptions()) { + if (enabled_) ConstructorWitnessBefore(function_, stage_); + } + + ConstructorWitnessPhase(const ConstructorWitnessPhase&) = delete; + ConstructorWitnessPhase& operator=(const ConstructorWitnessPhase&) = delete; + + ~ConstructorWitnessPhase() noexcept { + if (enabled_ && std::uncaught_exceptions() == uncaught_exceptions_) { + ConstructorWitnessAfter(function_, stage_); + } + } + + private: + const char* function_; + const char* stage_; + bool enabled_; + int uncaught_exceptions_; +}; + +template +auto ConstructorWitnessCall(const char* function, const char* stage, + Function&& call) + -> std::invoke_result_t { + ConstructorWitnessBefore(function, stage); + auto result = std::invoke(std::forward(call)); + ConstructorWitnessAfter(function, stage); + return result; +} + +} // namespace vllm::diagnostics + +#endif // VLLM_DIAGNOSTICS_CONSTRUCTOR_WITNESS_H_ diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index d99c1d770..8509ac82b 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -4,6 +4,7 @@ // e24d1b24) as exercised by examples/server/main.cpp and the test harness. #include "vllm/entrypoints/model_loader.h" #include "vllm/model_executor/models/qwen3_dflash_gguf.h" +#include "vllm/diagnostics/constructor_witness.h" #include #include @@ -43,6 +44,8 @@ namespace vllm::entrypoints { namespace fs = std::filesystem; +using diagnostics::ConstructorWitnessCall; +using diagnostics::ConstructorWitnessPhase; // `architecture` is the model's registered architecture string. It is what lets // a PARTIAL backend decline a model whose kernels it has not registered, instead @@ -948,7 +951,11 @@ int LoadedEngine::ResolveMaxModelLen(const EngineParams& params, LoadedEngine::LoadedEngine(HfConfig config, Qwen3_5MoeWeights weights, tok::Tokenizer tokenizer, const EngineParams& params) : LoadedEngine(std::move(config), - MakeQwen3_5MoeLoadedModel(std::move(weights)), + ConstructorWitnessCall( + "LoadedEngine", "MakeQwen3_5MoeLoadedModel", + [&weights]() { + return MakeQwen3_5MoeLoadedModel(std::move(weights)); + }), std::move(tokenizer), params) {} LoadedEngine::LoadedEngine(HfConfig config, Qwen3_5DenseWeights weights, @@ -963,59 +970,82 @@ LoadedEngine::LoadedEngine(HfConfig config, const EngineParams& params, vt::Queue* preselected_queue, std::unique_ptr dflash_draft) - : hash_ready_(EnsureNoneHash()), - config_(std::move(config)), + : hash_ready_((ConstructorWitnessPhase{"LoadedEngine", "hash_ready_"}, + EnsureNoneHash())), + config_((ConstructorWitnessPhase{"LoadedEngine", "config_"}, + std::move(config))), // SPEC-MTP I5d: finalize the speculative config against the checkpoint // (n_predict + resolved k). nullopt on the production default path. - resolved_spec_config_(ResolveSpecConfig(params, config_)), + resolved_spec_config_((ConstructorWitnessPhase{ + "LoadedEngine", "resolved_spec_config_"}, + ResolveSpecConfig(params, config_))), // SPEC-DFLASH D5: the separately-loaded DFlash draft (null for mtp/non-spec). - dflash_draft_(std::move(dflash_draft)), - model_(std::move(model)), - tokenizer_(std::move(tokenizer)), + dflash_draft_((ConstructorWitnessPhase{"LoadedEngine", "dflash_draft_"}, + std::move(dflash_draft))), + model_((ConstructorWitnessPhase{"LoadedEngine", "model_"}, + std::move(model))), + kv_connector_((ConstructorWitnessPhase{"LoadedEngine", "kv_connector_"}, + nullptr)), + tokenizer_((ConstructorWitnessPhase{"LoadedEngine", "tokenizer_"}, + std::move(tokenizer))), // ROAD-V1-MEM M1: resolve the block count from the sizing knobs // (num_blocks override > kv_cache_memory_bytes > util fallback) against the // model's own per-block byte geometry. FIRST, because max_model_len_ is // resolved against this pool. - kv_cfg_(MakeKVCacheResolved( - *model_, config_, params.block_size > 0 ? params.block_size : 32, - params, resolved_spec_config_)), + kv_cfg_((ConstructorWitnessPhase{"LoadedEngine", "kv_cfg_"}, + MakeKVCacheResolved( + *model_, config_, + params.block_size > 0 ? params.block_size : 32, params, + resolved_spec_config_))), // The serving length, checked (pinned) or auto-fitted (unpinned) against // kv_cfg_. See ResolveMaxModelLen. - max_model_len_(ResolveMaxModelLen( - params, config_, kv_cfg_, - params.block_size > 0 ? params.block_size : 32)), - max_num_batched_tokens_(ResolveMaxNumBatchedTokens( - params, max_model_len_, ModelRegistry::IsDenseModel(*model_))), - prefix_caching_enabled_(ResolveEnablePrefixCaching( - params, model_->registration().info)), + max_model_len_((ConstructorWitnessPhase{"LoadedEngine", "max_model_len_"}, + ResolveMaxModelLen( + params, config_, kv_cfg_, + params.block_size > 0 ? params.block_size : 32))), + max_num_batched_tokens_((ConstructorWitnessPhase{ + "LoadedEngine", "max_num_batched_tokens_"}, + ResolveMaxNumBatchedTokens( + params, max_model_len_, + ModelRegistry::IsDenseModel(*model_)))), + prefix_caching_enabled_((ConstructorWitnessPhase{ + "LoadedEngine", "prefix_caching_enabled_"}, + ResolveEnablePrefixCaching( + params, model_->registration().info))), // ENG-SGLANG-BEHAVIOR-FLAG SW3: resolve jump-forward once (config field + // VT_ENABLE_JUMP_FORWARD env override). Default nullopt+no-env => false => // the byte-identical decode path (jump-forward is inert until enabled). - jump_forward_enabled_( - vllm::v1::JumpForwardEnabled(params.enable_jump_forward)), + jump_forward_enabled_((ConstructorWitnessPhase{ + "LoadedEngine", "jump_forward_enabled_"}, + vllm::v1::JumpForwardEnabled( + params.enable_jump_forward))), // runner_ FIRST (W3): the async-scheduling flip reads // runner_.runner_supports_async(). SPEC-MTP I5d: when speculation is on, // pass the resolved config + the MTP draft (built from the retained mtp.* // weights, sharing the target embed/lm_head). The draft KV `fa_draft` group // is allocated by the runner from kv_cfg_ (empty vector here), so the loop // reaches it via runner-owned storage. nullopt/null on the default path. - runner_(config_, *model_, kv_cfg_, - preselected_queue != nullptr - ? *preselected_queue - : SelectQueueForModel(model_->registration().architecture, - params.device), - /*max_num_reqs=*/params.max_num_seqs > 0 ? params.max_num_seqs : 8, - max_model_len_, - /*max_num_batched_tokens=*/max_num_batched_tokens_, - resolved_spec_config_, - // Only the MTP method builds an in-target MTP draft; DFlash (D5) - // loads a SEPARATE draft, wired via set_dflash_draft in the body. - resolved_spec_config_.has_value() && - resolved_spec_config_->method == "mtp" && - model_->supports_mtp_draft() - ? model_->BuildMtpDraft(config_) - : nullptr, - /*draft_kv=*/{}), + runner_((ConstructorWitnessPhase{"LoadedEngine", "runner_"}, + vllm::v1::GPUModelRunner( + config_, *model_, kv_cfg_, + preselected_queue != nullptr + ? *preselected_queue + : SelectQueueForModel( + model_->registration().architecture, params.device), + /*max_num_reqs=*/params.max_num_seqs > 0 + ? params.max_num_seqs + : 8, + max_model_len_, + /*max_num_batched_tokens=*/max_num_batched_tokens_, + resolved_spec_config_, + // Only the MTP method builds an in-target MTP draft; DFlash + // (D5) loads a SEPARATE draft, wired in the body. + resolved_spec_config_.has_value() && + resolved_spec_config_->method == "mtp" && + model_->supports_mtp_draft() + ? model_->BuildMtpDraft(config_) + : nullptr, + /*draft_kv=*/{}))), // Resolve the enable-flip from the now-constructed runner + VT_ASYNC_SCHED, // then size the batch-queue depth (2 under async scheduling → depth-2 // step_with_batch_queue; 1 otherwise). Since the 2026-07-17 flip the default @@ -1025,55 +1055,84 @@ LoadedEngine::LoadedEngine(HfConfig config, // post_step path, which is the SYNCHRONOUS scheduler's contract; the // async-scheduling draft-in-output variant is deferred (spec §2.5), so a // configured speculator forces sync scheduling here. - async_scheduling_enabled_(!resolved_spec_config_.has_value() && - ResolveAsyncEnabled( - MakeSchedulerConfig( - max_model_len_, - params.max_num_seqs > 0 ? params.max_num_seqs : 8, - max_num_batched_tokens_, params.policy), - runner_.runner_supports_async(), - model_->registration().info.is_pooling_model)), - max_concurrent_batches_(MakeSchedulerConfig( - max_model_len_, - params.max_num_seqs > 0 ? params.max_num_seqs - : 8, - max_num_batched_tokens_, params.policy) - .MaxConcurrentBatches(async_scheduling_enabled_)), + async_scheduling_enabled_((ConstructorWitnessPhase{ + "LoadedEngine", + "async_scheduling_enabled_"}, + !resolved_spec_config_.has_value() && + ResolveAsyncEnabled( + MakeSchedulerConfig( + max_model_len_, + params.max_num_seqs > 0 + ? params.max_num_seqs + : 8, + max_num_batched_tokens_, + params.policy), + runner_.runner_supports_async(), + model_->registration() + .info.is_pooling_model))), + max_concurrent_batches_((ConstructorWitnessPhase{ + "LoadedEngine", "max_concurrent_batches_"}, + MakeSchedulerConfig( + max_model_len_, + params.max_num_seqs > 0 ? params.max_num_seqs + : 8, + max_num_batched_tokens_, params.policy) + .MaxConcurrentBatches( + async_scheduling_enabled_))), // The engine-wide structured-output manager, native backend over the // tokenizer (upstream EngineCore constructs one unconditionally, // core.py:134). Wired into the scheduler + engine cores below so // response_format / C-ABI structured constraints gate decoding. - structured_output_manager_( - params.max_num_seqs > 0 ? params.max_num_seqs : 8, - vllm::v1::MakeNativeBackendFactory( - tokenizer_, static_cast(config_.vocab_size))), + structured_output_manager_((ConstructorWitnessPhase{ + "LoadedEngine", + "structured_output_manager_"}, + vllm::v1::StructuredOutputManager( + params.max_num_seqs > 0 + ? params.max_num_seqs + : 8, + vllm::v1::MakeNativeBackendFactory( + tokenizer_, static_cast( + config_.vocab_size))))), // AsyncScheduler when the flip resolved ON, else the synchronous Scheduler. - scheduler_(MakeScheduler( - async_scheduling_enabled_, - MakeSchedulerConfig( - max_model_len_, params.max_num_seqs > 0 ? params.max_num_seqs : 8, - max_num_batched_tokens_, params.policy), - kv_cfg_, params.block_size > 0 ? params.block_size : 32, - /*enable_caching=*/prefix_caching_enabled_, - &structured_output_manager_, resolved_spec_config_)), - executor_(runner_), + scheduler_((ConstructorWitnessPhase{"LoadedEngine", "scheduler_"}, + MakeScheduler( + async_scheduling_enabled_, + MakeSchedulerConfig( + max_model_len_, + params.max_num_seqs > 0 ? params.max_num_seqs : 8, + max_num_batched_tokens_, params.policy), + kv_cfg_, params.block_size > 0 ? params.block_size : 32, + /*enable_caching=*/prefix_caching_enabled_, + &structured_output_manager_, resolved_spec_config_))), + executor_((ConstructorWitnessPhase{"LoadedEngine", "executor_"}, + vllm::v1::Executor(runner_))), // SPEC-MTP I5d: with a speculator configured, EngineCore pulls the runner's // out-of-band drafts each step (take_draft_token_ids -> update_draft_token_ids) // so the next verify step schedules them. Default false (no-op post_step). - engine_core_(*scheduler_, executor_, &structured_output_manager_, - /*check_for_draft_tokens=*/resolved_spec_config_.has_value()), + engine_core_((ConstructorWitnessPhase{"LoadedEngine", "engine_core_"}, + vllm::v1::EngineCore( + *scheduler_, executor_, &structured_output_manager_, + /*check_for_draft_tokens=*/ + resolved_spec_config_.has_value()))), // The admission-time prompt-length check validates against the RESOLVED // serving length, which is what upstream's model_config.max_model_len is // (input_processor.py:399-401). Passing config_ alone would check against // the raw checkpoint context and let through prompts the pool cannot hold. - input_processor_(tokenizer_, config_, max_model_len_), - output_processor_(&tokenizer_), - block_hasher_(prefix_caching_enabled_ - ? vllm::v1::get_request_block_hasher( - params.block_size > 0 ? params.block_size : 32, - vllm::v1::sha256_cbor) - : nullptr), - engine_(input_processor_, engine_core_, output_processor_, block_hasher_) { + input_processor_((ConstructorWitnessPhase{"LoadedEngine", "input_processor_"}, + vllm::v1::InputProcessor(tokenizer_, config_, + max_model_len_))), + output_processor_((ConstructorWitnessPhase{ + "LoadedEngine", "output_processor_"}, + vllm::v1::OutputProcessor(&tokenizer_))), + block_hasher_((ConstructorWitnessPhase{"LoadedEngine", "block_hasher_"}, + prefix_caching_enabled_ + ? vllm::v1::get_request_block_hasher( + params.block_size > 0 ? params.block_size : 32, + vllm::v1::sha256_cbor) + : nullptr)), + engine_((ConstructorWitnessPhase{"LoadedEngine", "engine_"}, + vllm::v1::LLMEngine(input_processor_, engine_core_, + output_processor_, block_hasher_))) { (void)hash_ready_; // issue #371: REFUSE an unservable recurrent-state budget instead of // allocating it. Speculation widens the Mamba/GDN state to k+1 snapshot slots diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 22ce5b197..2a9d7d1cf 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -5,6 +5,7 @@ // See include/vllm/v1/worker/gpu/runner.h for scope, the V1-algorithm / MRV2- // contract composition, the four-way ordering contract, and the deferred paths. #include "vllm/v1/worker/gpu/runner.h" +#include "vllm/diagnostics/constructor_witness.h" #include #include @@ -40,6 +41,10 @@ namespace vllm::v1 { +using diagnostics::ConstructorWitnessAfter; +using diagnostics::ConstructorWitnessBefore; +using diagnostics::ConstructorWitnessPhase; + // Logits-gather A/B toggle (perf). Default ON: the forward gathers the // per-request last-token hidden rows BEFORE lm_head (prefill/mixed), so lm_head // runs on num_reqs rows and only [num_reqs,vocab] is Downloaded. VT_LOGITS_GATHER=0 @@ -317,18 +322,29 @@ GPUModelRunner::GPUModelRunner( std::optional spec_config, std::unique_ptr draft_model, std::vector draft_kv) - : config_(config), - model_(&model), - spec_config_(std::move(spec_config)), - draft_model_(std::move(draft_model)), - draft_attn_kv_(std::move(draft_kv)), - queue_(queue), - input_batch_(max_num_reqs, max_model_len, max_num_batched_tokens, - static_cast(config.vocab_size), - group_block_sizes(kv_cache_config), - group_block_sizes(kv_cache_config)) { + : config_((ConstructorWitnessPhase{"GPUModelRunner", "config_"}, config)), + owned_model_((ConstructorWitnessPhase{"GPUModelRunner", "owned_model_"}, + nullptr)), + model_((ConstructorWitnessPhase{"GPUModelRunner", "model_"}, &model)), + spec_config_((ConstructorWitnessPhase{"GPUModelRunner", "spec_config_"}, + std::move(spec_config))), + draft_model_((ConstructorWitnessPhase{"GPUModelRunner", "draft_model_"}, + std::move(draft_model))), + draft_attn_kv_((ConstructorWitnessPhase{"GPUModelRunner", "draft_attn_kv_"}, + std::move(draft_kv))), + queue_((ConstructorWitnessPhase{"GPUModelRunner", "queue_"}, queue)), + input_batch_((ConstructorWitnessPhase{"GPUModelRunner", "input_batch_"}, + InputBatch(max_num_reqs, max_model_len, + max_num_batched_tokens, + static_cast(config.vocab_size), + group_block_sizes(kv_cache_config), + group_block_sizes(kv_cache_config)))) { + ConstructorWitnessBefore("GPUModelRunner", "assign-max-num-reqs"); max_num_reqs_ = max_num_reqs; + ConstructorWitnessAfter("GPUModelRunner", "assign-max-num-reqs"); + ConstructorWitnessBefore("GPUModelRunner", "assign-max-num-batched-tokens"); max_num_batched_tokens_ = max_num_batched_tokens; + ConstructorWitnessAfter("GPUModelRunner", "assign-max-num-batched-tokens"); // SPEC-MTP I5e: the async input-combine splices the device-resident // last_sampled token over each decode row's input id with // num_new_sampled_tokens==1; it is NOT spec-aware and would overwrite the @@ -337,17 +353,25 @@ GPUModelRunner::GPUModelRunner( // spliced into token_ids_cpu by update_req_spec_token_ids + prepare_inputs, // so force the sync host input path here. Byte-identical for non-spec // (spec_config_ is nullopt there, so this is AsyncRunnerEnvDefault()). + ConstructorWitnessBefore("GPUModelRunner", "resolve-async-input-combine"); async_input_combine_ = AsyncRunnerEnvDefault() && !spec_config_.has_value() && QueueSupportsAsyncInputCombine(queue_); + ConstructorWitnessAfter("GPUModelRunner", "resolve-async-input-combine"); // ARCH-ONE-SURFACE ROW 6 (mirror gpu/model_runner.py:368-369): a POOLING // model's runner pools instead of sampling — build the PoolingRunner over // the model-owned Pooler. Null for every text arch (byte-identical). + ConstructorWitnessBefore("GPUModelRunner", "pooling-model-branch"); if (model_->registration().info.is_pooling_model && model_->pooler() != nullptr) { pooling_runner_ = std::make_unique(*model_->pooler()); } + ConstructorWitnessAfter("GPUModelRunner", "pooling-model-branch"); + ConstructorWitnessBefore("GPUModelRunner", "initialize-kv-cache"); initialize_kv_cache(kv_cache_config); + ConstructorWitnessAfter("GPUModelRunner", "initialize-kv-cache"); + ConstructorWitnessBefore("GPUModelRunner", "model-registry-prepare"); ModelRegistry::Prepare(*model_, config_, queue_); + ConstructorWitnessAfter("GPUModelRunner", "model-registry-prepare"); } GPUModelRunner::GPUModelRunner( @@ -437,6 +461,8 @@ GPUModelRunner::CacheBuffer::~CacheBuffer() { } void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "scalar-state-slot-setup"); num_blocks_ = kv_cache_config.num_blocks; // GDN mamba-state slots = max concurrent sequences (one recurrent state per // sequence), decoupled from the attention num_blocks. Guard against a 0 (e.g. @@ -457,6 +483,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { // spec_cols; without speculation spec_cols==1 and this is the pre-spec pool. for (int64_t b = base_slots - 1; b >= 0; --b) gdn_free_slots_.push_back(static_cast(b * spec_cols)); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "scalar-state-slot-setup"); // Resolve the full-attn + GDN(mamba) KV group ids (T0 gate models: exactly one // of each). The block-table group order == kv_cache_groups order. @@ -472,6 +500,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { // byte-identical to the old behavior — the first and only group is chosen. for (int g = 0; g < static_cast(kv_cache_config.kv_cache_groups.size()); ++g) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "kv-group-scan", g); const auto& group = kv_cache_config.kv_cache_groups[static_cast(g)]; const KVCacheSpecKind kind = group.kv_cache_spec->kind(); // MLA campaign W7: an `MLAAttentionSpec` group IS the model's attention @@ -489,6 +519,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { } else if (kind == KVCacheSpecKind::kMamba) { gdn_group_id_ = g; } + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "kv-group-scan", g); } // Allocate one PagedKvCache per full-attn layer and one GdnStateCache per GDN @@ -508,6 +540,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { const MambaSpec* mamba_spec = nullptr; if (gdn_group_id_ >= 0) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "mamba-validation", gdn_group_id_); mamba_spec = dynamic_cast( kv_cache_config.kv_cache_groups[static_cast(gdn_group_id_)] .kv_cache_spec.get()); @@ -534,6 +568,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { VT_CHECK(supported_state_dtype(gdn_conv_cache_dtype_) && supported_state_dtype(gdn_ssm_cache_dtype_), "runner: Qwen3.5 MambaSpec state dtypes must be floating"); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "mamba-validation", gdn_group_id_); } // SPEC-DRIVEN attention-cache sizing and layout (MLA campaign W1). @@ -564,6 +600,9 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { int64_t fa_page_bytes = 0; vt::DType kv_dtype = ResolveKvCacheDType(); if (full_attn_group_id_ >= 0) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "full-attention-geometry", + full_attn_group_id_); const KVCacheSpec* fa_spec = kv_cache_config.kv_cache_groups[static_cast(full_attn_group_id_)] .kv_cache_spec.get(); @@ -600,12 +639,17 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { static_cast(fa_page_bytes), static_cast(num_blocks_)); } + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "full-attention-geometry", + full_attn_group_id_); } // Recorded for the gates: the exact per-block byte cost the allocator used, // sourced from the spec. `fa_page_size_bytes() > 0` is the runtime proof that // the spec-driven path RAN (a compiled-but-unexercised path leaves it 0). fa_page_size_bytes_ = fa_page_bytes; + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "residency-buffer-setup"); const vt::Device dev = queue_.device; const char* device_cache_env = std::getenv("VT_DEVICE_KV_CACHE"); // W0b-1 / work row M3a: this read `is_cuda()`, which is the SAME defect the @@ -656,6 +700,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { vt::DType dtype; }; std::vector fa_dims; + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "residency-buffer-setup"); for (int64_t l = 0; l < config_.num_hidden_layers; ++l) { const bool is_gdn = has_mamba_group && !config_.layer_types.empty() && @@ -669,14 +715,22 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { const size_t conv_es = vt::SizeOf(gdn_conv_cache_dtype_); const int64_t conv_row_elems = conv_dim * conv_state_len; const int64_t ssm_row_elems = Hv * Dv * Dk; + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "gdn-ssm-allocation", l); ssm_buf_.push_back(std::make_unique( dev, queue_, static_cast(gdn_state_slots_ * ssm_row_elems) * ssm_es, kv_cache_backend_resident_)); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "gdn-ssm-allocation", l); + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "gdn-conv-allocation", l); conv_buf_.push_back(std::make_unique( dev, queue_, static_cast(gdn_state_slots_ * conv_row_elems) * conv_es, kv_cache_backend_resident_)); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "gdn-conv-allocation", l); } else { // Bytes come from the SPEC, not from HF-config arithmetic: exactly // `num_blocks * spec->page_size_bytes()`, mirroring upstream's @@ -713,11 +767,15 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { VT_CHECK(l_page > 0, "runner: per-layer attention spec reported a non-positive page"); } + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "full-attention-allocation", l); full_attn_buf_.push_back(std::make_unique( dev, queue_, static_cast(num_blocks_) * static_cast(l_page), kv_cache_backend_resident_)); fa_dims.push_back(FaDims{l_Hkv, l_Dh, l_dtype}); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "full-attention-allocation", l); } } @@ -727,6 +785,9 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { "runner: per-layer KV view geometry out of sync with buffers"); attn_kv_.clear(); for (size_t i = 0; i < full_attn_buf_.size(); ++i) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "full-attention-view", + static_cast(i)); PagedKvCache kv; kv.data = full_attn_buf_[i]->data(); kv.dtype = fa_dims[i].dtype; @@ -735,6 +796,9 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { kv.num_kv_heads = fa_dims[i].num_kv_heads; kv.head_size = fa_dims[i].head_size; attn_kv_.push_back(kv); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "full-attention-view", + static_cast(i)); } // SPEC-MTP I5d: allocate the MTP draft's own paged KV layer (the `fa_draft` @@ -753,6 +817,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { if (group.kv_cache_spec->kind() != KVCacheSpecKind::kFullAttention) { continue; // the GDN group and any non-attn group are not the draft. } + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "draft-attention-storage", g); draft_attn_buf_.push_back(std::make_unique( dev, queue_, static_cast(num_blocks_) * static_cast(fa_page_bytes), @@ -765,12 +831,16 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { dkv.num_kv_heads = Hkv; dkv.head_size = Dh; draft_attn_kv_.push_back(dkv); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "draft-attention-storage", g); break; // exactly one fa_draft group at k=1. } } gdn_state_.clear(); for (size_t g = 0; g < ssm_buf_.size(); ++g) { + ConstructorWitnessBefore("GPUModelRunner::initialize_kv_cache", + "gdn-state-view", static_cast(g)); GdnStateCache gs; gs.ssm_state = vt::Tensor::Contiguous(ssm_buf_[g]->data(), gdn_ssm_cache_dtype_, @@ -781,6 +851,8 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { {gdn_state_slots_, conv_dim, conv_state_len}); gdn_state_.push_back(gs); + ConstructorWitnessAfter("GPUModelRunner::initialize_kv_cache", + "gdn-state-view", static_cast(g)); } } diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 79887693b..8b27cf3b9 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -293,6 +293,171 @@ def test_openai_isolated_probe_forwards_stable_phase_witness_output(self) -> Non with self.subTest(forwarded_line=forwarded_line): self.assertIn(forwarded_line, contract) + def test_openai_constructor_diagnostic_is_isolated_ordered_and_nonrecovering(self) -> None: + script = (ROOT / "scripts/build-windows-release.ps1").read_text( + encoding="utf-8" + ) + helper_start = script.find("function Invoke-OpenAiPrefixBisect {") + self.assertNotEqual(helper_start, -1, "adaptive prefix bisect is missing") + helper_end = script.index("\n}\n", helper_start) + len("\n}\n") + helper = script[helper_start:helper_end] + env_name = "VLLM_WINDOWS_CTOR_DIAGNOSTIC" + for statement in ( + f'$diagnosticEnvironmentName = "{env_name}"', + "$priorDiagnosticEnvironment = [Environment]::GetEnvironmentVariable(", + '[Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, "1", "Process")', + "try {", + "finally {", + "[Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, $priorDiagnosticEnvironment, \"Process\")", + ): + with self.subTest(statement=statement): + self.assertEqual(helper.count(statement), 1) + isolated = helper.index("$isolatedResult = Invoke-OpenAiPrefixRange") + enable = helper.index( + '[Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, "1", "Process")' + ) + restore = helper.index( + '[Environment]::SetEnvironmentVariable($diagnosticEnvironmentName, $priorDiagnosticEnvironment, "Process")' + ) + self.assertLess(enable, isolated) + self.assertLess(isolated, restore) + + contract_start = script.index( + "function Invoke-OpenAiPrefixBisectContractTests {" + ) + contract_end = script.index("\n}\n", contract_start) + len("\n}\n") + contract = script[contract_start:contract_end] + for guarantee in ( + "DiagnosticEnvironment = [Environment]::GetEnvironmentVariable(", + 'DiagnosticEnvironment -ceq "1"', + 'DiagnosticEnvironment -ne "1"', + 'Write-Host "OpenAI constructor diagnostic activation contract OK"', + 'Write-Host "OpenAI constructor diagnostic restoration contract OK"', + ): + with self.subTest(guarantee=guarantee): + self.assertIn(guarantee, contract) + + diagnostic = ( + ROOT / "src/vllm/diagnostics/constructor_witness.h" + ).read_text(encoding="utf-8") + for guarantee in ( + f'constexpr const char* kConstructorWitnessEnvironment = "{env_name}"', + '"VLLM_CTOR_DIAGNOSTIC: function=%s stage=%s phase=%s\\n"', + '"VLLM_CTOR_DIAGNOSTIC: function=%s stage=%s phase=%s index=%lld\\n"', + "std::fflush(stderr)", + "std::uncaught_exceptions() == uncaught_exceptions_", + ): + with self.subTest(guarantee=guarantee): + self.assertIn(guarantee, diagnostic) + + loader = (ROOT / "src/vllm/entrypoints/model_loader.cpp").read_text( + encoding="utf-8" + ) + constructor_start = loader.index( + "LoadedEngine::LoadedEngine(HfConfig config,\n" + " std::unique_ptr model," + ) + constructor_end = loader.index("\n}\n", constructor_start) + len("\n}\n") + constructor = loader[constructor_start:constructor_end] + compact_constructor = re.sub(r"\s+", "", constructor) + loaded_stages = ( + "hash_ready_", "config_", "resolved_spec_config_", "dflash_draft_", + "model_", "kv_connector_", "tokenizer_", "kv_cfg_", "max_model_len_", + "max_num_batched_tokens_", "prefix_caching_enabled_", + "jump_forward_enabled_", "runner_", "async_scheduling_enabled_", + "max_concurrent_batches_", "structured_output_manager_", "scheduler_", + "executor_", "engine_core_", "input_processor_", "output_processor_", + "block_hasher_", "engine_", + ) + positions = [] + for stage in loaded_stages: + marker = f'ConstructorWitnessPhase{{"LoadedEngine","{stage}"}}' + with self.subTest(loaded_engine_stage=stage): + self.assertEqual(compact_constructor.count(marker), 1) + positions.append(compact_constructor.index(marker)) + self.assertEqual(positions, sorted(positions)) + self.assertIn( + 'ConstructorWitnessCall("LoadedEngine","MakeQwen3_5MoeLoadedModel"', + re.sub(r"\s+", "", loader), + ) + + runner = (ROOT / "src/vllm/v1/worker/gpu/runner.cpp").read_text( + encoding="utf-8" + ) + runner_start = runner.index( + "GPUModelRunner::GPUModelRunner(\n const HfConfig& config, LoadedModel& model," + ) + runner_end = runner.index("\n}\n", runner_start) + len("\n}\n") + runner_constructor = runner[runner_start:runner_end] + compact_runner_constructor = re.sub(r"\s+", "", runner_constructor) + runner_stages = ( + "config_", "owned_model_", "model_", "spec_config_", "draft_model_", + "draft_attn_kv_", "queue_", "input_batch_", + ) + runner_positions = [] + for stage in runner_stages: + marker = f'ConstructorWitnessPhase{{"GPUModelRunner","{stage}"}}' + with self.subTest(runner_stage=stage): + self.assertEqual(compact_runner_constructor.count(marker), 1) + runner_positions.append(compact_runner_constructor.index(marker)) + self.assertEqual(runner_positions, sorted(runner_positions)) + for body_stage in ( + "assign-max-num-reqs", "assign-max-num-batched-tokens", + "resolve-async-input-combine", "pooling-model-branch", + "initialize-kv-cache", "model-registry-prepare", + ): + with self.subTest(runner_body_stage=body_stage): + self.assertEqual( + runner_constructor.count( + f'ConstructorWitnessBefore("GPUModelRunner", "{body_stage}")' + ), + 1, + ) + self.assertEqual( + runner_constructor.count( + f'ConstructorWitnessAfter("GPUModelRunner", "{body_stage}")' + ), + 1, + ) + + init_start = runner.index( + "void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) {" + ) + init_end = runner.index("\n}\n", init_start) + len("\n}\n") + initialize = runner[init_start:init_end] + compact_initialize = re.sub(r"\s+", "", initialize) + for stage in ( + "scalar-state-slot-setup", "kv-group-scan", "mamba-validation", + "full-attention-geometry", "residency-buffer-setup", + "gdn-ssm-allocation", "gdn-conv-allocation", + "full-attention-allocation", "full-attention-view", + "draft-attention-storage", "gdn-state-view", + ): + with self.subTest(kv_stage=stage): + self.assertIn( + f'"GPUModelRunner::initialize_kv_cache","{stage}"', + compact_initialize, + ) + + api_test = ( + ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" + ).read_text(encoding="utf-8") + for guarantee in ( + "_set_invalid_parameter_handler", + "_set_purecall_handler", + "_invoke_watson", + "WINDOWS_CRT_INVALID_PARAMETER:", + "WINDOWS_CRT_PURECALL", + "prior_invalid_parameter_handler_", + "prior_purecall_handler_", + ): + with self.subTest(crt_guarantee=guarantee): + self.assertIn(guarantee, api_test) + self.assertLess( + api_test.index("WINDOWS_CRT_INVALID_PARAMETER:"), + api_test.index("_invoke_watson"), + ) + def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8") handler_start = main.index( diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index b9baf80b7..edccb2213 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -12,6 +12,7 @@ // (tiny hybrid-MoE Qwen3.6 + the BPE fixture, vocab ids 0..21). #include "vllm/entrypoints/openai/api_server.h" #include "vllm/entrypoints/openai/video_api.h" +#include "vllm/diagnostics/constructor_witness.h" #include "vllm/multimodal/parakeet_transcription.h" #include "scoped_server_thread.h" @@ -36,6 +37,11 @@ #include #endif +#if defined(_MSC_VER) +#include +#include +#endif + #include #include @@ -115,6 +121,74 @@ using vllm::test::ScopedServerThread; namespace { +#if defined(_MSC_VER) +_invalid_parameter_handler prior_invalid_parameter_handler = nullptr; +_purecall_handler prior_purecall_handler = nullptr; + +[[noreturn]] void WindowsCrtInvalidParameterWitness( + const wchar_t* expression, const wchar_t* function, const wchar_t* file, + unsigned int line, uintptr_t reserved) { + std::fprintf(stderr, + "WINDOWS_CRT_INVALID_PARAMETER: expression=%ls function=%ls " + "file=%ls line=%u\n", + expression != nullptr ? expression : L"(null)", + function != nullptr ? function : L"(null)", + file != nullptr ? file : L"(null)", line); + std::fflush(stderr); + if (prior_invalid_parameter_handler != nullptr) { + prior_invalid_parameter_handler(expression, function, file, line, reserved); + } + _invoke_watson(expression, function, file, line, reserved); +} + +[[noreturn]] void WindowsCrtPurecallWitness() { + std::fputs("WINDOWS_CRT_PURECALL\n", stderr); + std::fflush(stderr); + if (prior_purecall_handler != nullptr) { + prior_purecall_handler(); + } + _invoke_watson(L"pure virtual function call", L"_purecall", L"(runtime)", + 0, 0); +} +#endif + +class ScopedWindowsCrtConstructorWitness final { + public: + ScopedWindowsCrtConstructorWitness() noexcept { +#if defined(_MSC_VER) + if (!vllm::diagnostics::ConstructorWitnessEnabled()) return; + prior_invalid_parameter_handler_ = + _set_invalid_parameter_handler(WindowsCrtInvalidParameterWitness); + prior_invalid_parameter_handler = prior_invalid_parameter_handler_; + prior_purecall_handler_ = _set_purecall_handler(WindowsCrtPurecallWitness); + prior_purecall_handler = prior_purecall_handler_; + installed_ = true; +#endif + } + + ScopedWindowsCrtConstructorWitness( + const ScopedWindowsCrtConstructorWitness&) = delete; + ScopedWindowsCrtConstructorWitness& operator=( + const ScopedWindowsCrtConstructorWitness&) = delete; + + ~ScopedWindowsCrtConstructorWitness() noexcept { +#if defined(_MSC_VER) + if (!installed_) return; + _set_purecall_handler(prior_purecall_handler_); + _set_invalid_parameter_handler(prior_invalid_parameter_handler_); + prior_purecall_handler = nullptr; + prior_invalid_parameter_handler = nullptr; +#endif + } + + private: +#if defined(_MSC_VER) + _invalid_parameter_handler prior_invalid_parameter_handler_ = nullptr; + _purecall_handler prior_purecall_handler_ = nullptr; + bool installed_ = false; +#endif +}; + void OpenAiExplicitCpuPhaseWitness(const char* phase) noexcept { std::fprintf(stderr, "OPENAI_EXPLICIT_CPU_PHASE: %s\n", phase); std::fflush(stderr); @@ -2398,6 +2472,7 @@ TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { // accelerator; explicit cuda never falls back) is test_loaded_engine_dense.cpp; // the C-ABI plumb is test_capi.cpp. TEST_CASE("api_server: an explicit-cpu device-selected engine serves /v1/completions") { + ScopedWindowsCrtConstructorWitness crt_witness; const HfConfig c = MakeConfig(); vllm::entrypoints::EngineParams params; params.block_size = kBlockSize; From d915c30b70bbb1dccdff7ef4c7cc1ee56e75a3bb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:46:49 +0000 Subject: [PATCH 35/35] test(release): strengthen Windows constructor witness contract (#537) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GPT-5 [CODEX] --- docs/FEATURES.md | 2 +- docs/USAGE.md | 5 +- src/vllm/diagnostics/constructor_witness.h | 13 -- src/vllm/entrypoints/model_loader.cpp | 7 +- .../model_executor/models/qwen3_5_moe.cpp | 10 +- .../scripts/test_release_windows_metadata.py | 125 ++++++++++++++---- 6 files changed, 111 insertions(+), 51 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index ae6355ec7..41d6b9629 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -27,7 +27,7 @@ are our reading of their documented behavior, not measurements. | Correctness gate | token-exact vs vLLM | reference | own | own | | Architectures | 38 registered, 27 gated | 130+ | 100+ | 100+ | | Downloadable server binaries | ✅ v0.0.2: eight indexed archives with checksums, provenance, manifests, and SBOMs. Windows ZIP downloads do not exist; native CPU/Vulkan lanes await hosted runtime, dry-run, prerelease, and authenticated audit gates | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries | -| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, central `NOMINMAX`, UTF-8, aligned allocation, runtime ISA dispatch; LTX2 uses C++20 math constants, not `M_PI`. Local DeepSeek float probe green; hosted compile/runtime/release pending | ✅ | ✅ | ✅ | +| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, `NOMINMAX`, UTF-8, aligned allocation, runtime ISA dispatch. Issue #537 has an isolated constructor/KV/CRT witness. Local float probe green; hosted compile/runtime/release pending | ✅ | ✅ | ✅ | ## Serving and scheduling diff --git a/docs/USAGE.md b/docs/USAGE.md index a9f306dc1..4a6c9db2c 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -698,7 +698,10 @@ While issue #537 is being attributed, the Windows release helper also runs an internal constructor witness only in the fresh-process isolated OpenAI case found by its prefix probe. The helper restores the process environment before continuing; the ordinary prefix probes, complete OpenAI suite, staged server, -and production invocations do not enable or inherit that diagnostic. +and production invocations do not enable or inherit that diagnostic. Its +ordered markers cover the model factory, engine and runner member construction, +and indexed KV-cache allocation; Windows invalid-parameter and purecall faults +flush their own marker and terminate through the CRT fallback. The CUDA graph-replay profiler and its FIFO diagnostic controls remain POSIX-only and are not exposed by native Windows server builds. Native Windows diff --git a/src/vllm/diagnostics/constructor_witness.h b/src/vllm/diagnostics/constructor_witness.h index cdfcdadf9..72bab8159 100644 --- a/src/vllm/diagnostics/constructor_witness.h +++ b/src/vllm/diagnostics/constructor_witness.h @@ -4,9 +4,6 @@ #include #include #include -#include -#include -#include namespace vllm::diagnostics { @@ -74,16 +71,6 @@ class ConstructorWitnessPhase { int uncaught_exceptions_; }; -template -auto ConstructorWitnessCall(const char* function, const char* stage, - Function&& call) - -> std::invoke_result_t { - ConstructorWitnessBefore(function, stage); - auto result = std::invoke(std::forward(call)); - ConstructorWitnessAfter(function, stage); - return result; -} - } // namespace vllm::diagnostics #endif // VLLM_DIAGNOSTICS_CONSTRUCTOR_WITNESS_H_ diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 8509ac82b..cc92cb0e2 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -44,7 +44,6 @@ namespace vllm::entrypoints { namespace fs = std::filesystem; -using diagnostics::ConstructorWitnessCall; using diagnostics::ConstructorWitnessPhase; // `architecture` is the model's registered architecture string. It is what lets @@ -951,11 +950,7 @@ int LoadedEngine::ResolveMaxModelLen(const EngineParams& params, LoadedEngine::LoadedEngine(HfConfig config, Qwen3_5MoeWeights weights, tok::Tokenizer tokenizer, const EngineParams& params) : LoadedEngine(std::move(config), - ConstructorWitnessCall( - "LoadedEngine", "MakeQwen3_5MoeLoadedModel", - [&weights]() { - return MakeQwen3_5MoeLoadedModel(std::move(weights)); - }), + MakeQwen3_5MoeLoadedModel(std::move(weights)), std::move(tokenizer), params) {} LoadedEngine::LoadedEngine(HfConfig config, Qwen3_5DenseWeights weights, diff --git a/src/vllm/model_executor/models/qwen3_5_moe.cpp b/src/vllm/model_executor/models/qwen3_5_moe.cpp index 1e33afb76..4c03db8e7 100644 --- a/src/vllm/model_executor/models/qwen3_5_moe.cpp +++ b/src/vllm/model_executor/models/qwen3_5_moe.cpp @@ -23,6 +23,7 @@ #include "vllm/model_executor/models/qwen3_5_mtp.h" // SPEC-MTP I5d-pre draft #include "vllm/model_executor/models/qwen3_5_weights.h" #include "vllm/platforms/interface.h" // GetPlatform(device.type) memory-model seam +#include "vllm/diagnostics/constructor_witness.h" namespace vllm { namespace { @@ -200,9 +201,12 @@ const ModelFactory kQwen3_5MoeFactory{ std::unique_ptr MakeQwen3_5MoeLoadedModel( Qwen3_5MoeWeights weights) { - return std::make_unique( - RegistrationFor("Qwen3_5MoeForConditionalGeneration"), - std::move(weights)); + using diagnostics::ConstructorWitnessPhase; + return (ConstructorWitnessPhase{"LoadedEngine", + "MakeQwen3_5MoeLoadedModel"}, + std::make_unique( + RegistrationFor("Qwen3_5MoeForConditionalGeneration"), + std::move(weights))); } std::unique_ptr BorrowQwen3_5MoeLoadedModel( diff --git a/tests/scripts/test_release_windows_metadata.py b/tests/scripts/test_release_windows_metadata.py index 8b27cf3b9..b2a6c10d2 100644 --- a/tests/scripts/test_release_windows_metadata.py +++ b/tests/scripts/test_release_windows_metadata.py @@ -369,6 +369,23 @@ def test_openai_constructor_diagnostic_is_isolated_ordered_and_nonrecovering(sel "executor_", "engine_core_", "input_processor_", "output_processor_", "block_hasher_", "engine_", ) + loader_header = ( + ROOT / "include/vllm/entrypoints/model_loader.h" + ).read_text(encoding="utf-8") + loader_members = loader_header[ + loader_header.index(" bool hash_ready_;"): + loader_header.index(" vllm::v1::LLMEngine engine_;") + + len(" vllm::v1::LLMEngine engine_;") + ] + declared_loaded_stages = tuple( + re.findall( + r"^\s*(?!//)(?:const\s+)?(?:[\w:]+(?:<[^;\n]+>)?)" + r"[\s*&]+([A-Za-z]\w*_)\s*(?:=[^;]*)?;\s*(?://.*)?$", + loader_members, + re.MULTILINE, + ) + ) + self.assertEqual(loaded_stages, declared_loaded_stages) positions = [] for stage in loaded_stages: marker = f'ConstructorWitnessPhase{{"LoadedEngine","{stage}"}}' @@ -376,9 +393,15 @@ def test_openai_constructor_diagnostic_is_isolated_ordered_and_nonrecovering(sel self.assertEqual(compact_constructor.count(marker), 1) positions.append(compact_constructor.index(marker)) self.assertEqual(positions, sorted(positions)) + model_factory = ( + ROOT / "src/vllm/model_executor/models/qwen3_5_moe.cpp" + ).read_text(encoding="utf-8") + compact_model_factory = re.sub(r"\s+", "", model_factory) + self.assertNotIn("ConstructorWitnessCall", diagnostic + loader) self.assertIn( - 'ConstructorWitnessCall("LoadedEngine","MakeQwen3_5MoeLoadedModel"', - re.sub(r"\s+", "", loader), + 'return(ConstructorWitnessPhase{"LoadedEngine",' + '"MakeQwen3_5MoeLoadedModel"},std::make_unique<', + compact_model_factory, ) runner = (ROOT / "src/vllm/v1/worker/gpu/runner.cpp").read_text( @@ -394,6 +417,23 @@ def test_openai_constructor_diagnostic_is_isolated_ordered_and_nonrecovering(sel "config_", "owned_model_", "model_", "spec_config_", "draft_model_", "draft_attn_kv_", "queue_", "input_batch_", ) + runner_header = ( + ROOT / "include/vllm/v1/worker/gpu/runner.h" + ).read_text(encoding="utf-8") + runner_members = runner_header[ + runner_header.index(" const HfConfig& config_;"): + runner_header.index(" InputBatch input_batch_;") + + len(" InputBatch input_batch_;") + ] + declared_runner_stages = tuple( + re.findall( + r"^\s*(?!//)(?:const\s+)?(?:[\w:]+(?:<[^;\n]+>)?)" + r"[\s*&]+([A-Za-z]\w*_)\s*(?:=[^;]*)?;\s*(?://.*)?$", + runner_members, + re.MULTILINE, + ) + ) + self.assertEqual(runner_stages, declared_runner_stages) runner_positions = [] for stage in runner_stages: marker = f'ConstructorWitnessPhase{{"GPUModelRunner","{stage}"}}' @@ -426,37 +466,68 @@ def test_openai_constructor_diagnostic_is_isolated_ordered_and_nonrecovering(sel init_end = runner.index("\n}\n", init_start) + len("\n}\n") initialize = runner[init_start:init_end] compact_initialize = re.sub(r"\s+", "", initialize) - for stage in ( - "scalar-state-slot-setup", "kv-group-scan", "mamba-validation", - "full-attention-geometry", "residency-buffer-setup", - "gdn-ssm-allocation", "gdn-conv-allocation", - "full-attention-allocation", "full-attention-view", - "draft-attention-storage", "gdn-state-view", - ): - with self.subTest(kv_stage=stage): - self.assertIn( - f'"GPUModelRunner::initialize_kv_cache","{stage}"', - compact_initialize, + kv_stages = { + "scalar-state-slot-setup": None, + "kv-group-scan": "g", + "mamba-validation": "gdn_group_id_", + "full-attention-geometry": "full_attn_group_id_", + "residency-buffer-setup": None, + "gdn-ssm-allocation": "l", + "gdn-conv-allocation": "l", + "full-attention-allocation": "l", + "full-attention-view": "static_cast(i)", + "draft-attention-storage": "g", + "gdn-state-view": "static_cast(g)", + } + for stage, index in kv_stages.items(): + suffix = "" if index is None else f",{index}" + for phase in ("Before", "After"): + marker = ( + f'ConstructorWitness{phase}(' + f'"GPUModelRunner::initialize_kv_cache","{stage}"' + f'{suffix});' ) + with self.subTest(kv_stage=stage, phase=phase): + self.assertEqual(compact_initialize.count(marker), 1) api_test = ( ROOT / "tests/vllm/entrypoints/openai/test_api_server.cpp" ).read_text(encoding="utf-8") - for guarantee in ( - "_set_invalid_parameter_handler", - "_set_purecall_handler", - "_invoke_watson", - "WINDOWS_CRT_INVALID_PARAMETER:", - "WINDOWS_CRT_PURECALL", - "prior_invalid_parameter_handler_", - "prior_purecall_handler_", - ): - with self.subTest(crt_guarantee=guarantee): - self.assertIn(guarantee, api_test) - self.assertLess( - api_test.index("WINDOWS_CRT_INVALID_PARAMETER:"), - api_test.index("_invoke_watson"), + invalid_start = api_test.index( + "[[noreturn]] void WindowsCrtInvalidParameterWitness(" + ) + pure_start = api_test.index( + "[[noreturn]] void WindowsCrtPurecallWitness() {" + ) + invalid_handler = api_test[invalid_start:pure_start] + pure_end = api_test.index("\n}\n#endif", pure_start) + len("\n}\n") + pure_handler = api_test[pure_start:pure_end] + handler_contracts = ( + ( + "invalid-parameter", + invalid_handler, + "[[noreturn]] void WindowsCrtInvalidParameterWitness(", + "WINDOWS_CRT_INVALID_PARAMETER:", + "prior_invalid_parameter_handler(expression, function, file, line, reserved);", + ), + ( + "purecall", + pure_handler, + "[[noreturn]] void WindowsCrtPurecallWitness() {", + "WINDOWS_CRT_PURECALL", + "prior_purecall_handler();", + ), ) + for name, body, signature, diagnostic_line, prior_call in handler_contracts: + with self.subTest(crt_handler=name): + self.assertTrue(body.startswith(signature)) + self.assertEqual(body.count(diagnostic_line), 1) + self.assertEqual(body.count("std::fflush(stderr)"), 1) + self.assertEqual(body.count(prior_call), 1) + self.assertEqual(body.count("_invoke_watson("), 1) + self.assertLess(body.index(diagnostic_line), body.index("std::fflush(stderr)")) + self.assertLess(body.index("std::fflush(stderr)"), body.index(prior_call)) + self.assertLess(body.index(prior_call), body.index("_invoke_watson(")) def test_socket_teardown_probe_marks_each_owner_and_terminate_reason(self) -> None: main = (ROOT / "tests/doctest_main.cpp").read_text(encoding="utf-8")