Skip to content

Fix flaky master CI: relax knife-edge LBFGS objective bound in lotka_volterra test#134

Merged
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-lts-lotka-objective-threshold
Jun 24, 2026
Merged

Fix flaky master CI: relax knife-edge LBFGS objective bound in lotka_volterra test#134
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-lts-lotka-objective-threshold

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Ignore until reviewed by @ChrisRackauckas.

Problem

The master tests / All (julia lts) lane has been failing on the Basic testset (test/lotka_volterra.jl:149):

Basic: Test Failed at test/lotka_volterra.jl:149
  Expression: res2.objective < 0.00015
   Evaluated: 0.00028010684535790533 < 0.00015   # windows lts
   Evaluated: 0.0002385427084901909  < 0.00015   # ubuntu lts

This is the single failing assertion (9 passed, 1 failed, 0 errored, 2 broken).

Root cause: knife-edge threshold on a MaxIters-terminated optimizer

res2 comes from a two-stage UDE training (Adam(1e-3, maxiters=10_000) then LBFGS(maxiters=5000)). The LBFGS stage terminates at MaxIters, not at a gradient tolerance — so the reported objective is whatever value the trajectory happens to reach at the iteration cap. That value depends on BLAS/platform floating-point reassociation, so the assertion flips pass/fail across OSes from run to run:

Run ubuntu lts windows lts macOS lts
2026-06-20 FAIL 2.385e-4 FAIL 2.801e-4 PASS
2026-06-14 PASS PASS FAIL 2.705e-4

A converged fit lands anywhere in ~[3.6e-5, 2.8e-4], and the old 1.5e-4 bound sits inside that jitter band. This is benign cross-platform numerical non-determinism, not a regression — the package's structural/gradient/JET tests all pass and the network does train.

Verification (local, Julia 1.10 LTS, linux)

  • Full Basic safetestset (test deps resolved via [targets].test): 10 Pass, 2 Broken, 0 Fail (passes locally even with the old 1.5e-4 bound).
  • Probe of the exact optimization:
    • DEFAULT_LBFGS res2.objective = 3.583e-5 (retcode MaxIters)
    • Adam-only res.objective = 2.316 (untrained loss is O(1))

So a successful fit is ~4-5 orders of magnitude below the untrained loss, but the exact MaxIters value spans ~10x across platforms.

Fix

Raise the bound 1.5e-4 -> 5e-4 with an explanatory comment. 5e-4 sits safely above the observed jitter band (max observed failing value 2.8e-4, ~1.8x margin) while remaining ~4000x below the untrained O(1) loss, so it still verifies the UDE actually trained without being sensitive to BLAS/platform reassociation. No test is skipped, broken, or silenced; the assertion still fails if training does not occur.

🤖 Generated with Claude Code

The `@test res2.objective < 1.5e-4` in test/lotka_volterra.jl was a knife-edge
threshold on the result of a two-stage UDE training (Adam then LBFGS). The
LBFGS stage terminates at MaxIters (5000) rather than at a gradient tolerance,
so the final objective is whatever value the trajectory happens to reach at the
iteration cap. That value varies with BLAS/platform floating-point
reassociation, so the same assertion has been flipping pass/fail across OSes
run to run on the master `All (julia lts)` lane:

  - 2026-06-20 run: ubuntu lts FAIL (2.385e-4), windows lts FAIL (2.801e-4),
    macOS lts PASS
  - 2026-06-14 run: macOS lts FAIL (2.705e-4), ubuntu+windows lts PASS

A converged fit lands anywhere in ~[3.6e-5, 2.8e-4] (local linux Julia 1.10
reproduces 3.58e-5), and 1.5e-4 sits inside that band. This is benign
cross-platform numerical non-determinism, not a regression: the package's
structural/gradient/JET tests all pass and the network does train (Adam-only
loss is O(1) at ~2.32; the LBFGS stage drives it down ~4-5 orders of magnitude).

Raise the bound to 5e-4, which sits safely above the observed jitter band while
remaining ~4000x below the untrained loss, so it still verifies the UDE
actually trained without being sensitive to BLAS/platform reassociation.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 24, 2026 12:02
@ChrisRackauckas ChrisRackauckas merged commit 1bb3535 into SciML:main Jun 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants