Fix flaky master CI: relax knife-edge LBFGS objective bound in lotka_volterra test#134
Merged
ChrisRackauckas merged 1 commit intoJun 24, 2026
Conversation
The `@test res2.objective < 1.5e-4` in test/lotka_volterra.jl was a knife-edge
threshold on the result of a two-stage UDE training (Adam then LBFGS). The
LBFGS stage terminates at MaxIters (5000) rather than at a gradient tolerance,
so the final objective is whatever value the trajectory happens to reach at the
iteration cap. That value varies with BLAS/platform floating-point
reassociation, so the same assertion has been flipping pass/fail across OSes
run to run on the master `All (julia lts)` lane:
- 2026-06-20 run: ubuntu lts FAIL (2.385e-4), windows lts FAIL (2.801e-4),
macOS lts PASS
- 2026-06-14 run: macOS lts FAIL (2.705e-4), ubuntu+windows lts PASS
A converged fit lands anywhere in ~[3.6e-5, 2.8e-4] (local linux Julia 1.10
reproduces 3.58e-5), and 1.5e-4 sits inside that band. This is benign
cross-platform numerical non-determinism, not a regression: the package's
structural/gradient/JET tests all pass and the network does train (Adam-only
loss is O(1) at ~2.32; the LBFGS stage drives it down ~4-5 orders of magnitude).
Raise the bound to 5e-4, which sits safely above the observed jitter band while
remaining ~4000x below the untrained loss, so it still verifies the UDE
actually trained without being sensitive to BLAS/platform reassociation.
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ignore until reviewed by @ChrisRackauckas.
Problem
The master
tests / All (julia lts)lane has been failing on theBasictestset (test/lotka_volterra.jl:149):This is the single failing assertion (
9 passed, 1 failed, 0 errored, 2 broken).Root cause: knife-edge threshold on a MaxIters-terminated optimizer
res2comes from a two-stage UDE training (Adam(1e-3, maxiters=10_000) thenLBFGS(maxiters=5000)). The LBFGS stage terminates at MaxIters, not at a gradient tolerance — so the reported objective is whatever value the trajectory happens to reach at the iteration cap. That value depends on BLAS/platform floating-point reassociation, so the assertion flips pass/fail across OSes from run to run:A converged fit lands anywhere in ~
[3.6e-5, 2.8e-4], and the old1.5e-4bound sits inside that jitter band. This is benign cross-platform numerical non-determinism, not a regression — the package's structural/gradient/JET tests all pass and the network does train.Verification (local, Julia 1.10 LTS, linux)
Basicsafetestset (test deps resolved via[targets].test):10 Pass, 2 Broken, 0 Fail(passes locally even with the old 1.5e-4 bound).DEFAULT_LBFGS res2.objective = 3.583e-5(retcodeMaxIters)Adam-only res.objective = 2.316(untrained loss is O(1))So a successful fit is ~4-5 orders of magnitude below the untrained loss, but the exact MaxIters value spans ~10x across platforms.
Fix
Raise the bound
1.5e-4 -> 5e-4with an explanatory comment.5e-4sits safely above the observed jitter band (max observed failing value 2.8e-4, ~1.8x margin) while remaining ~4000x below the untrained O(1) loss, so it still verifies the UDE actually trained without being sensitive to BLAS/platform reassociation. No test is skipped, broken, or silenced; the assertion still fails if training does not occur.🤖 Generated with Claude Code