Use SciMLTesting v1.2 (folder-based run_tests)#308
Merged
ChrisRackauckas merged 1 commit intoJun 14, 2026
Conversation
ChrisRackauckas-Claude
force-pushed
the
sciml-testing-rollout
branch
from
c5e86a2 to
fa4d10d
Compare
ChrisRackauckas-Claude
changed the title
Contributor
Author
|
Force-updated this branch to the corrected SciMLTesting v1.1.0 conversion, superseding the v1.0.0 first pass. What changed vs the v1.0.0 commit on this branch:
Behavior is identical to upstream |
Convert the test suite to the SciMLTesting v1.2 folder-discovery model: runtests.jl is now `using SciMLTesting; run_tests()`. - Core = top-level test/*.jl. The tests_on_odes/ files are flattened to the top level and made self-contained (each loads DiffEqParamEstim + its own deps and includes the shared fixture), so each runs isolated in its own @safetestset module. - The shared problem fixture (test_problems.jl), the never-referenced lorenz*.jl files, and the commented-out genetic_algorithm_test.jl move to test/shared/ (a non-group folder, never auto-discovered) so the discovered Core set matches the prior dispatch exactly. - QA = test/qa/ with its existing sub-env Project.toml. - Deps: add SciMLTesting + SafeTestsets to the root test target and the QA sub-env; drop Pkg (only the old harness used it). test_groups.toml unchanged. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ChrisRackauckas-Claude
force-pushed
the
sciml-testing-rollout
branch
from
fa4d10d to
ed5dcfe
Compare
ChrisRackauckas-Claude
changed the title
ChrisRackauckas-Claude
pushed a commit
to ChrisRackauckas-Claude/DiffEqParamEstim.jl
that referenced
this pull request
Jun 16, 2026
After the SciMLTesting v1.2 migration (SciML#308) each test file runs in its own isolated @safetestset module, so latent missing imports that were previously masked by the old include-everything runtests.jl now surface as UndefVarError. `test/likelihood.jl` and `test/test_on_monte.jl` both use the qualified `Optimization.OptimizationProblem`/`Optimization.AutoForwardDiff` but only imported a re-exporting backend (OptimizationBBO / OptimizationOptimJL). `using OptimizationOptimJL` re-exports Optimization's *exported symbols* but does not bind the `Optimization` *module* name into scope, so the qualified references errored with `UndefVarError: Optimization not defined`. This is the same class of failure SciML#310 already fixed in dde_tests.jl. Add `Optimization` (already a test dep via [extras]/[targets]) to the `using` line of both files. No assertions changed. Verified locally on Julia 1.10 (the compat floor / CI lts) by running each file in an isolated @safetestset against the develop'd package: Core/likelihood.jl isolated | 5 Pass, 0 errors Core/test_on_monte.jl isolated | 1 Pass, 0 errors Both Runic-format clean (format_string is a no-op). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ChrisRackauckas-Claude
pushed a commit
to ChrisRackauckas-Claude/DiffEqParamEstim.jl
that referenced
this pull request
Jun 19, 2026
After the SciMLTesting v1.2 migration (SciML#308) each test file runs in its own isolated @safetestset module, so latent missing imports that were previously masked by the old include-everything runtests.jl now surface as UndefVarError. `test/out_of_place_odes.jl` used `L2Loss`, `build_loss_objective`, and `two_stage_objective` (all exported by DiffEqParamEstim) but only imported `OrdinaryDiffEq, Test, SciMLSensitivity, Optimization, OptimizationOptimJL`. Under the old shared-`Main` runtests, a sibling file's `using DiffEqParamEstim` leaked those names into scope; under v1.2 isolation the file aborts the Core run with `UndefVarError: L2Loss not defined`. This was the last file failing all three Core variants (lts/1/pre) and the Downgrade job. Same class of failure already fixed in this PR for dde_tests.jl, likelihood.jl, and test_on_monte.jl. Add `DiffEqParamEstim` to the `using` line. No assertions changed; the `Optimization.AutoZygote()` loss path works here unchanged. Verified locally on Julia 1.10 (CI lts / compat floor) by running the file in an isolated @safetestset against the develop'd package: OOP isolated | 2 Pass, 0 errors (was: UndefVarError: L2Loss not defined) Runic-format clean (format_string is a no-op). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ChrisRackauckas
added a commit
that referenced
this pull request
Jun 20, 2026
…ff (#310) * Fix dde_tests: import OptimizationOptimJL for BFGS and use AutoForwardDiff The DDE test referenced `BFGS` and `Optimization` without importing the packages that provide them. Under the old `include`-everything `runtests.jl`, `optim_test.jl`'s `using Optim`/`OptimizationOptimJL` leaked `BFGS` into the shared `Main` scope, so the missing import went unnoticed. With the SciMLTesting v1.2 migration each test file now runs in its own isolated module, exposing `UndefVarError: BFGS not defined`. Add `using Optimization, OptimizationOptimJL` (both already test deps) and drop the dead, unused `Opt(:GN_ESCH, 1)` line. Fixing the import revealed a second, latent failure: `AutoZygote()` gradient preparation for the DDE loss errors with `MethodError: no method matching _prepare_pullback_aux` in DifferentiationInterface. The sibling `nlopt_test.jl` already documents this ("#zygote behaves weirdly here") and uses `AutoForwardDiff()`; switching the DDE test to `AutoForwardDiff()` makes it run and assert the real result (res.u[1] ≈ 0.5). Verified locally on Julia 1.10 with the CI-pinned stack (Optimization 4.8.0, OptimizationBase 2.14.0, OptimizationOptimJL 0.4.5, DifferentiationInterface 0.7.18) running the file in an isolated @safetestset: 1 Pass, 0 errors. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix likelihood.jl and test_on_monte.jl: import Optimization explicitly After the SciMLTesting v1.2 migration (#308) each test file runs in its own isolated @safetestset module, so latent missing imports that were previously masked by the old include-everything runtests.jl now surface as UndefVarError. `test/likelihood.jl` and `test/test_on_monte.jl` both use the qualified `Optimization.OptimizationProblem`/`Optimization.AutoForwardDiff` but only imported a re-exporting backend (OptimizationBBO / OptimizationOptimJL). `using OptimizationOptimJL` re-exports Optimization's *exported symbols* but does not bind the `Optimization` *module* name into scope, so the qualified references errored with `UndefVarError: Optimization not defined`. This is the same class of failure #310 already fixed in dde_tests.jl. Add `Optimization` (already a test dep via [extras]/[targets]) to the `using` line of both files. No assertions changed. Verified locally on Julia 1.10 (the compat floor / CI lts) by running each file in an isolated @safetestset against the develop'd package: Core/likelihood.jl isolated | 5 Pass, 0 errors Core/test_on_monte.jl isolated | 1 Pass, 0 errors Both Runic-format clean (format_string is a no-op). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: migrate examples to current solver/EnsembleSolution API The Documentation job was red on master after the OrdinaryDiffEq v7 / DifferentialEquations v8 ecosystem bump. Three independent docs-source breakages, all fixed against the current API: - verbose: OrdinaryDiffEq v7 rejects a `Bool` for `verbose` (`ArgumentError: Passing a Bool for verbose is no longer supported ... Use DEVerbosity() or a preset like Standard(), None(), etc. from SciMLLogging`). Replace `verbose = false` with `verbose = None()` (a SciMLLogging AbstractVerbosityPreset, the no-output preset) and add `using SciMLLogging: None`. Affects getting_started.md, generalized_likelihood.md, stochastic_evaluations.md. - StochasticDiffEq solvers: DifferentialEquations v8 was slimmed to {OrdinaryDiffEq, Reexport, SciMLBase} and no longer re-exports the SDE solvers, so `SRIW1`/`SOSRI` in stochastic_evaluations.md errored with UndefVarError. Add an explicit `using StochasticDiffEq: SRIW1, SOSRI` (matching how the package's own tests import them). - EnsembleSolution indexing: `sim[i]` now returns a scalar (flat element indexing), not the i-th trajectory, so the ensemble.md loss `losses[i](sim[i])` hit `MethodError: no method matching (::L2Loss)(::Float64)`. Use `sim.u[i]` to get the i-th ODESolution. Add SciMLLogging and StochasticDiffEq to docs/Project.toml [deps]+[compat]. No test/example was skipped, broken, warnonly-ed, or loosened. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update multiple_shooting_objective_test.jl * Fix out_of_place_odes.jl: import DiffEqParamEstim After the SciMLTesting v1.2 migration (#308) each test file runs in its own isolated @safetestset module, so latent missing imports that were previously masked by the old include-everything runtests.jl now surface as UndefVarError. `test/out_of_place_odes.jl` used `L2Loss`, `build_loss_objective`, and `two_stage_objective` (all exported by DiffEqParamEstim) but only imported `OrdinaryDiffEq, Test, SciMLSensitivity, Optimization, OptimizationOptimJL`. Under the old shared-`Main` runtests, a sibling file's `using DiffEqParamEstim` leaked those names into scope; under v1.2 isolation the file aborts the Core run with `UndefVarError: L2Loss not defined`. This was the last file failing all three Core variants (lts/1/pre) and the Downgrade job. Same class of failure already fixed in this PR for dde_tests.jl, likelihood.jl, and test_on_monte.jl. Add `DiffEqParamEstim` to the `using` line. No assertions changed; the `Optimization.AutoZygote()` loss path works here unchanged. Verified locally on Julia 1.10 (CI lts / compat floor) by running the file in an isolated @safetestset against the develop'd package: OOP isolated | 2 Pass, 0 errors (was: UndefVarError: L2Loss not defined) Runic-format clean (format_string is a no-op). Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ChrisRackauckas-Claude <accounts@chrisrackauckas.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convert the test suite to the SciMLTesting v1.2 folder-discovery model.
test/runtests.jlbecomes:Layout
test/*.jl. The formertest/tests_on_odes/*.jlfiles are flattened to the top level and made self-contained — each now loadsDiffEqParamEstim+ its own deps andincludes the shared problem fixture — so each runs isolated in its own@safetestsetmodule (folder-discovery requires self-contained files; they previously shared one@testsetscope).test_problems.jlfixture, the never-referencedlorenz_test.jl/lorenz_true_test.jl, and the commented-outgenetic_algorithm_test.jl. This keeps the discovered Core set byte-for-byte equal to the prior dispatch.test/qa/with its existing sub-envProject.toml.Deps
SciMLTesting+SafeTestsetsto the root test target and the QA sub-envProject.toml.Pkgfrom the root test deps (only the old harness used it; no test file references it).test/test_groups.tomlunchanged.Verification
Project.toml+test_groups.tomlparses; every test.jlparses; folder-discovery (real SciMLTesting v1.2.0) resolves Core/All to exactly the 16 files the old dispatcher ran and QA toqa.jlin its sub-env.GROUP=QAPkg.test()on Julia 1.11 passed (7 pass / 2 broken — the 2 pre-existing@test_broken), exercising the sub-env activation path.Ignore until reviewed by @ChrisRackauckas.