Skip to content

fix(release): run no-argv tests on Windows - #524

Open
localai-bot wants to merge 49 commits into
mainfrom
row/ENG-RELEASE-WINDOWS-512
Open

fix(release): run no-argv tests on Windows#524
localai-bot wants to merge 49 commits into
mainfrom
row/ENG-RELEASE-WINDOWS-512

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

The merged-SHA release dry run and hosted Windows CI exposed a chain of independent release blockers. This PR keeps their fixes and evidence together so the release can be proven end to end.

What changed:

  • permit zero-argument checked PowerShell child processes;
  • preserve non-empty argv through a PowerShell-native recorder;
  • prove failure targets run in a real child and propagate the exact exit status;
  • use portable environment APIs in the Vulkan cross-device test;
  • pair CPU-floor /proc/stat busy and total samples;
  • drain accepted httplib sockets and preserve server-thread failure propagation;
  • remove MSVC shadow warnings in the Vulkan test;
  • diagnose the remaining native Windows OpenAI fast-fail with an adaptive source-order prefix bisect;
  • forward the isolated failing process output and add reviewed, mutation-proven engine/runner/KV construction witnesses with Windows CRT fallbacks;
  • replace LTX2 M_PI dependencies with portable C++20 constants;
  • replace VideoEngine POSIX stat calls with portable filesystem queries.

Verification at pushed head 5268337a795be737cccb078392c3ff9e01e3be52:

  • Windows CPU and Vulkan run 31728014706 independently identified source-order case 47/54, api_server: an explicit-cpu device-selected engine serves /v1/completions, failing in isolation with exact status -1073740791 (0xC0000409);
  • phase-diagnostic RED-first contract, implementation, and fresh mutation review passed;
  • native isolated OpenAI case: 1/1, 5/5 assertions, reviewed ordered constructor witness contract;
  • native full OpenAI suite: 54/54, 635/635 assertions;
  • Windows metadata: 13/13; release pipeline: 42/42; portability: 76/76;
  • direct trailer checker and full operator preflight passed at the exact pushed head;
  • latest main (a3aa02e19) is integrated, with keyed roadmap restored target-wholesale plus only the six scoped release rows.

The new hosted Windows CPU/Vulkan jobs are the binding proof for the exact crash phase. The diagnostic is temporary: after the phase evidence identifies the owner, this PR must remove the probes, add a RED regression for the actual defect, land the production repair, and return fully green before merge. Then the complete ten-platform non-publishing release workflow must pass at the exact merge SHA before v0.0.3-pre.1 is tagged.

Fixes #512
Fixes #514
Fixes #525
Fixes #530
Fixes #537
Fixes #540
Fixes #599
Fixes #645
Fixes #648

mudler added 6 commits August 12, 2026 18:29
The exact merged-SHA release run compiles Windows, then PowerShell rejects every checked test that takes no arguments.

Bind the repair to the real helper contract and require both Windows release lanes before the pre-alpha tag.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
The exact merged-SHA Vulkan build reaches a second failure: its cross-device test calls POSIX environment APIs under MSVC.

Bind the repair to a test-local platform seam and preserve both fused-chain tiers plus environment restoration.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
The Windows release driver passes @() to tests that take no arguments. PowerShell rejects the mandatory array before it invokes each test.

Allow the empty collection and exercise zero, exact nonempty, and failed child invocations in the hosted Windows contract.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Use checked native environment APIs in the cross-device fused-chain test while preserving tier selection and prior-state restoration on Windows and POSIX.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
Reject a dead Invoke-Checked contract path and require both fused-tier restoration branches to target the exact environment key.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
Preserve the reviewed #512/#514 fixes while incorporating origin/main through a89b3c4.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
mudler added 6 commits August 12, 2026 20:31
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Replace the lossy batch recorder with a native remaining-arguments JSON fixture while preserving the real checked empty, nonempty, and exit-23 invocations.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Derive busy and total jiffies from one aggregate CPU-line read for each quiet-window and benchmark-leg snapshot.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Record the hosted fast-fail, the upstream accepted-socket drain fix, scoped server-thread teardown, RED-first mutations, and immutable release acceptance gates before implementation.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
mudler added 5 commits August 12, 2026 21:53
Record the hosted C4456 failures, test-only rename scope, six independent mutations, and native MSVC acceptance before implementation.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
Windows can reset a completed response when unread peer bytes remain.
The reset then unwinds through joinable test threads and aborts the suite.

Drain accepted sockets within upstream bounds. Give each real-socket
test scoped server ownership so doctest can report fatal assertions.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
MSVC /W4 /WX rejects the nested unbound-flash CPU oracle because its short local names shadow the enclosing oracle. Give those six locals role-specific names and lock the contract with a focused portability test.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
The prior contract matched six exact declaration spellings. Equivalent comma declarators could restore an enclosing-oracle name without making the contract fail.

Derive names from the scoped declaration list while respecting initializer nesting. This keeps each forbidden name observable across assignment, brace, and parenthesis forms.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
@localai-bot localai-bot added the bug Something isn't working label Aug 12, 2026
localai-bot added a commit that referenced this pull request Aug 13, 2026
…nobody recorded it (#543, #545)

BENCH-ASSERT-CLOCK-STATE: BENCH-ASSERT-CLOCK-STATE -- the SM clock moves 12.8% between boots and nobody recorded it (#543, #545)

On this GB10 the SM clock differs BETWEEN BOOTS with no throttling active:
median 2470 MHz on one boot, pinned flat at 2190 on the next (max 3003,
applications 2418, throttle 0x0, persistence Enabled). 12.8% clock -> +7.2% step
time. The control proves it: marlin::Marlin, 129 calls/step, byte-identical
invocation, NO source change between a170c81 and 4064558, moved
45.2845 -> 49.6544 ms/step = +9.65%.

That drift is LARGER than either per-call deficit it was used to rank (in_proj
+2.97%, out_proj/o_proj +6.28%). Those deficits are not refuted -- they were
never taken against a clock control, so they are NOT ESTABLISHED. The same effect
explains a same-binary same-arm swing of 382.60 -> 357.59 us/call across a
reboot, and two probes disagreeing ~6% uniformly eight minutes apart in one boot.

So every leg now RECORDS SM clock min/median/max/n and spread, the max and
applications clocks, the union of throttle reasons, persistence, driver, GPU
name, retained-vs-idle sample counts, and boot_id. And the summary ASSERTS:
boot_id equality between arms, within-run spread, cross-arm median offset, benign
throttle bits, persistence. A ratio that cannot be attributed to a clock state is
voided, and gate_pass requires every ratio eligible.

A first review broke it twice, end to end, and both are fixed here:
  * a window the sampler barely observed scored the CLEANEST possible record --
    over n == 1 the spread is definitionally 0.00%. Three busy samples per arm
    out of 903 passed with no reasons. The incentive was inverted: a window
    actually observed was MORE likely to be voided. Floors now require 30 busy
    samples and a 50% busy fraction, both derived from the sampler rate and the
    leg grid rather than from what happens to pass, and both surfaced in the
    report.
  * --allow-cross-boot waived the MACHINE. It passed a comparison of an
    NVIDIA GB10 on driver 580.159.03 against an H100 80GB on 550.54.15, telling
    the reader only "different boots". STATIC_FIELDS are now asserted ACROSS arms
    unconditionally, ahead of any override; the override waives boot_id alone.

24 mutations, 24 RED, restored by sha256. Two of them taught something worth
keeping: one survived because a leg-level reason already dominated it -- the
third instance of that shape today -- and one was SEMANTICALLY INERT, since
[] or [...] evaluates to the non-empty list, so a checksum proved the file
changed but not the behaviour. Both are recorded as methodology notes.

Thresholds are argued, not fitted. The 1.0% cross-arm offset rests on physics,
not on n=1: for any kernel whose time scales with clock the transfer is bounded
above by 1.0, so 1.0% offset implies <=1.0% effect, under the 2.97% smallest
deficit ranked; the measured 0.7548 sits below that ceiling as a partly
memory-bound kernel should. The 5.0% spread rule deliberately does NOT meet that
criterion, and the spec says so: passing spread establishes an arm was one state,
not that a sub-4% deficit is established.

Scope is stated where it is read, not only where it is specified: only
dgx-online-serving.sh records. finalize_*_trace.py, summarize_torch_kernels.py
and gdn_packed_component.py do not -- and the trace path is exactly where both
retracted findings came from.

tests/tools 233 -> 310, 0 removed. Preflight is green except
test_cpu_x86_llamacpp_floor, which I reproduced FAILING IDENTICALLY on a clean
origin/main checkout at load 110 and which this branch does not touch: narrowly
waived, environmental.

Windows lanes windows-msvc-cpu/vulkan are red here and on every PR: they are
pull_request-only so no main push ever ran them, and across the last 120 PR runs
they are 0 success / 50 failure. Two pre-existing defects (POSIX setenv in
test_backend_cross_device.cpp, and a Mandatory PowerShell [string[]] rejecting
@()), both already fixed in open PR #524. This row is python/docs/shell only and
touches no C++; the ruleset requires no status checks.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]
mudler added 2 commits August 13, 2026 07:34
Resolve the release-runner conflict by retaining the branch's real-process argument contract tests while taking current main's already-landed Windows and parity repairs. Reconcile the keyed issue table from main and reapply only #525, #537, and #540 once.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Record the immutable hosted evidence that the accepted-socket drain and scoped server thread did not eliminate the native fast-fail. Narrow the next probe to the three destruction boundaries without weakening the full gate.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
mudler added 3 commits August 13, 2026 08:08
Track the direct-checker failure exposed by fresh review and bound the repair to a cross-platform real-process failure fixture with exact exit-status evidence.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:gpt-5 [Codex]
mudler added 7 commits August 13, 2026 09:17
Run the portable PowerShell fixtures through the current host executable so the contract exercises a real process boundary. Pin the boundary with a child PID assertion while retaining exact exit and argv checks.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:gpt-5 [Codex]
Review proved the live Python test trusted the script's own PID guard.

Emit both PIDs so the harness verifies the process boundary independently.

Pin the runtime guard to keep the script fail-closed check.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:gpt-5 [Codex]
Take the DSA and snapshot-pin repairs that landed after the prior local gate.

Rebuild the keyed issue table from main. Reapply only the four release entries.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Hosted Windows now completes every teardown marker before the exact fast-fail. The prefix probe isolates the earliest contaminating boundary.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5 [Codex]
The isolated socket case completes teardown, but the full native Windows process still fast-fails. Fresh source-order prefixes locate the earliest contaminating test boundary without weakening the original full gate.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5 [Codex]
The adaptive bisect contract did not distinguish listing order, execute both fail-closed status branches, or pin its diagnostic schema. The strengthened live contract makes each guarantee independently observable.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:gpt-5 [Codex]
Integrate current main before rerunning the hosted Windows diagnostic. Rebuild the keyed issue table from main and reapply only the four release-campaign issue rows.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
mudler added 2 commits August 13, 2026 16:53
Record issue #645 and constrain the repair to a standard C++ constant substitution without weakening the portability detector.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
Replace the non-standard M_PI macro in LTX2 numerical expressions with the C++20 double-precision constant, and remove the unused video VAE fallback. This preserves the existing expression types and values while allowing native MSVC builds to compile without feature macros.

Record the RED/GREEN portability and upstream-golden evidence, and project the Windows portability guarantee in the existing feature row.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
mudler added 10 commits August 13, 2026 17:15
Record issue #648 and constrain the repair to non-throwing C++20 filesystem queries with unchanged path classification.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
The generalized video seam used POSIX stat for path classification. This prevents its translation unit from reaching native MSVC builds.

Use non-throwing C++20 filesystem queries. They preserve the existing refusals for missing and uninspectable paths.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Integrate current main before pushing the Windows release diagnostic. Rebuild the keyed issue table from the target branch and reapply only the six release-campaign issue rows.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
Record the hosted prefix result and bind the next diagnostic to the isolated explicit-CPU serving case. Require captured output plus phase witnesses, then removal and a RED regression before any production repair.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Forward the isolated prefix probe output and add flushed phase witnesses around the explicit-CPU serving test so hosted Windows identifies the last completed lifetime boundary without changing the full-suite gate.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Integrate LTX-2.5 prompt AdaLN and tool-parser record updates before the next hosted Windows diagnostic. Rebuild the keyed issue table from origin/main and reapply only the six release-campaign issue rows.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Record the hosted phase boundary and bind the final diagnostic split across synthetic weight creation, tokenizer creation, and LoadedEngine construction.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Materialize synthetic weights and the tokenizer fixture before LoadedEngine construction so hosted Windows logs identify the exact fast-failing phase while preserving values and ownership.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GPT-5 [CODEX]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true

AI-Assisted: true

Assisted-by: Codex:GPT-5 [Codex]
mudler added 6 commits August 13, 2026 21:53
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GPT-5 [CODEX]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GPT-5 [CODEX]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment