Skip to content

Uniformize monorepo structure to OrdinaryDiffEq.jl canonical style#1220

Merged
ChrisRackauckas merged 9 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure
Jun 8, 2026
Merged

Uniformize monorepo structure to OrdinaryDiffEq.jl canonical style#1220
ChrisRackauckas merged 9 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Uniformizes Optimization.jl to the canonical OrdinaryDiffEq.jl monorepo structure.

Canonicalization done

OptimizationBase test/Project.toml removed (the only forbidden main test/Project.toml)

  • Folded its [deps] into lib/OptimizationBase/Project.toml [extras] + [targets].test.
  • Moved its [sources] (OptimizationLBFGSB, OptimizationManopt) into the package [sources], corrected to sibling paths ../OptimizationLBFGSB / ../OptimizationManopt (they were ../../ because they previously lived under test/).
  • Added matching [compat] for the new test extras (Aqua, ComponentArrays, IterTools, Lux, Manifolds, ModelingToolkit, Optim, Optimisers, OptimizationLBFGSB, OptimizationManopt, SafeTestsets, Symbolics, Tracker, BenchmarkTools, ChainRulesCore, Random, Pkg, SparseArrays, Test). No [compat] floor was lowered.
  • Names already present in [deps] (ADTypes, DifferentiationInterface, LinearAlgebra, SciMLBase, SparseArrays, SparseConnectivityTracer) are kept only in [targets].test, not duplicated into [extras].

Group threading (OPTIMIZATION_TEST_GROUP)

  • Every lib/<X>/test/runtests.jl is now a SafeTestsets group dispatcher: it reads get(ENV, "OPTIMIZATION_TEST_GROUP", "ALL") and @safetestset-includes the package's existing functional suite (moved verbatim to core_tests.jl) under the Core/ALL group, mirroring the OrdinaryDiffEq sublib runtests.jl pattern.
  • SafeTestsets added to each sublib's [extras] / [compat] / [targets].test.

Root test/runtests.jl

  • Rewritten as the canonical _detect_sublibrary_group dispatcher: handles bare sublib names and <sublib>_<group> suffixes, transitively Pkg.develops a sublib's [sources] on Julia < 1.11, sets OPTIMIZATION_TEST_GROUP, and Pkg.tests the sublib; otherwise runs the root's own Core/GPU @safetestset groups. Fixes the previously-undefined activate_downstream_env in the GPU branch.

Workflows

  • SublibraryCI.yml: passes group-env-name: OPTIMIZATION_TEST_GROUP and check-bounds: auto to sublibrary-project-tests.yml@v1.
  • DowngradeSublibraries.yml: passes group-env-name/group-env-value: Core, extends skip with the locally-developed in-repo sibling packages (OptimizationBase + the seven other [sources] siblings) plus stdlibs and SymbolicAnalysis. No allow-reresolve (removed in v1).
  • Root CI.yml (tests.yml@v1, GROUP-dispatched to Core) and Downgrade.yml (downgrade.yml@v1, no allow-reresolve) were already canonical and contain no hardcoded sublibrary GROUP rows (sublibs are covered by the SublibraryCI project model).

test_groups.toml

  • Added a Core-only test/test_groups.toml to every sublib so the matrix discovery declares the real group (Core on [lts, 1.11, 1, pre], matching the julia 1.10 floor) and does not spawn a phantom default QA-on-[1] job — no sublib has a QA suite (the Aqua extras in Ipopt/MadNLP are unused by their runtests.jl).

Root [sources] justification (invariant B)

The root Optimization package re-exports only OptimizationBase (its sole in-repo [deps] dependency); the solver sublibraries are not root dependencies. Root [sources] therefore lists OptimizationBase plus the four solver sublibraries used by the root test target (OptimizationLBFGSB, OptimizationMOI, OptimizationOptimJL, OptimizationOptimisers). Listing all 27 would inject false dependencies and break the true path graph, so it is intentionally left at 5.

Verified locally (isolated depot, Julia 1.11)

  • TOML-parsed every changed Project.toml + test_groups.toml: all parse; every [targets].test entry resolves via [extras][deps]; no name appears in both [deps] and [extras]; zero test/Project.toml remain (only the canonical test/downstream/Project.toml sub-environment); zero @testitem/ReTestItems anywhere.
  • YAML-parsed all changed workflows: valid.
  • Instantiated two representative sublibs: OptimizationBBO and OptimizationPolyalgorithms (the latter resolving all three in-repo siblings via [sources]).
  • Ran OptimizationBBO Core group through the new @safetestset dispatcher: Core | 25 25 pass, Testing OptimizationBBO tests passed.
  • Unit-tested the root _detect_sublibrary_group routing (OptimizationBBO→sublib/Core, OptimizationBBO_QA→sublib/QA, Core→root-Core, GPU→root-GPU): all correct.
  • Ran Runic on every authored .jl (28 dispatchers + root runtests): already clean, 0 reformats.

Left to CI

  • The full per-sublibrary SublibraryCI matrix (26 of 27 sublibraries were not run locally; many need heavy/external solvers — Ipopt, MadNLP, NLopt, Python via SciPy/PyCMA, etc.).
  • The root Core group end-to-end via Pkg.test (heavy root test env: Enzyme, Lux, SciMLSensitivity, ...) and the Downgrade/DowngradeSublibraries resolves.

Ignore until reviewed by @ChrisRackauckas.

ChrisRackauckas and others added 9 commits June 7, 2026 16:05
Align Optimization.jl with the canonical OrdinaryDiffEq.jl monorepo layout:

- Delete lib/OptimizationBase/test/Project.toml; fold its [deps] into the
  package's [extras] + [targets].test and move its [sources]
  (OptimizationLBFGSB, OptimizationManopt) into the main [sources] (now
  ../OptimizationLBFGSB / ../OptimizationManopt). Add the corresponding
  [compat] entries. No test/Project.toml remains anywhere (the kept
  test/downstream/Project.toml is a per-group sub-environment, matching the
  canonical pattern for special test groups).

- Thread OPTIMIZATION_TEST_GROUP consistently: every lib/<X>/test/runtests.jl
  is now a SafeTestsets group dispatcher that reads
  get(ENV, "OPTIMIZATION_TEST_GROUP", "ALL") and @safetestset-includes the
  package's functional suite (moved to core_tests.jl) under the Core/ALL
  group. SafeTestsets added to each sublib's [extras]/[compat]/[targets].test.

- Rewrite root test/runtests.jl as the canonical _detect_sublibrary_group
  dispatcher (handles bare sublib names and <sublib>_<group> suffixes,
  transitively develops a sublib's [sources] on Julia < 1.11, sets
  OPTIMIZATION_TEST_GROUP, and Pkg.tests the sublib; otherwise runs the
  root's own Core/GPU @safetestset groups). Fixes the previously-undefined
  activate_downstream_env in the GPU branch.

- SublibraryCI.yml: pass group-env-name=OPTIMIZATION_TEST_GROUP and
  check-bounds=auto to sublibrary-project-tests.yml@v1.

- DowngradeSublibraries.yml: pass group-env-name/group-env-value=Core and
  extend the skip list with the locally-developed in-repo sibling packages.
  No allow-reresolve input (removed in v1).

- Add a Core-only test/test_groups.toml to every sublib so the matrix
  discovery declares the real group (Core on [lts,1.11,1,pre]) and does not
  spawn a phantom default QA-on-[1] job (no sublib has a QA suite).

Root [sources] intentionally lists only OptimizationBase (the sole in-repo
[deps] dependency) plus the four solver sublibraries used by the root test
target; the root package re-exports only OptimizationBase, so adding the
remaining sublibraries would inject false dependencies and break the true
[sources] path graph.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match OrdinaryDiffEq.jl canonical casing: the sublibrary test/runtests.jl
files defaulted to and compared against the all-caps "ALL" group name.
Capitalize to Title-case "All" so the group-name strings line up with the
centralized matrix convention ("Core"/"QA"/test_groups.toml keys passed
verbatim, capitalized). Casing-only change; dispatch logic, env var names,
and fallback chains are unchanged.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… to 1.11

- TagBot.yml: add top-level permissions block, workflow_dispatch lookback
  input, root TagBot-Optimization job, and a TagBot-Subpackages matrix
  (fail-fast: false) over the 26 registered lib/<Name> sublibraries, each
  calling JuliaRegistries/TagBot@v1 with subdir: lib/<Name>. Matches
  OrdinaryDiffEq.jl's canonical TagBot structure. OptimizationQuadDIRECT is
  omitted since it is not registered in the General registry.
- Documentation.yml: convert from the old inline build/deploy form to the
  central SciML/.github documentation.yml@v1 caller. Sublibraries resolve
  via the existing docs/Project.toml [sources] entries.
- Downgrade.yml / DowngradeSublibraries.yml: set julia-version to 1.11.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add lib/<Name>/README.md to the 26 sublibraries that lacked one, using the
canonical OrdinaryDiffEq.jl sublibrary README template: the standard SciML
badge block (Zulip, Global Docs, ColPrac, SciML Code Style) plus the
"component package of Optimization.jl in the SciML ecosystem" wording with a
one-line description of each package's role. SimpleOptimization already had a
README and is left unchanged.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match OrdinaryDiffEq.jl's canonical monorepo layout:
- Add .codecov.yml with `comment: false` to disable Codecov PR comments.
- Rename the root LICENSE to LICENSE.md (content preserved via git mv).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the root package's Quality Assurance tests (Aqua + JET) into an
isolated test/qa/ Project so the QA tooling dependencies no longer enter
the main test target's resolve, matching the canonical OrdinaryDiffEq
lib/<Name>/test/qa/ structure.

- Add test/qa/Project.toml: deps Aqua, JET, Optimization, Test;
  [sources] Optimization = {path = "../.."}; [compat] for each + julia 1.10.
- Move test/qa.jl -> test/qa/qa.jl (Aqua + JET tests preserved verbatim;
  add `using Test` so the file is self-contained in the isolated env).
- runtests.jl: add activate_qa_env() (Pkg.activate(test/qa) + instantiate,
  plus develop-by-path on Julia < 1.11 where [sources] is ignored). The QA
  safetestset now runs last in the Core group after activating the qa env,
  so the main-environment tests above are unaffected. CI dispatches only
  GROUP=Core, so QA continues to run exactly as before.
- Remove Aqua and JET from the main Project.toml [extras], [targets].test,
  and [compat]; they now live solely in the qa environment.

Verified on Julia 1.11 in an isolated depot: the qa env instantiates and
resolves (Aqua 0.8.16, JET 0.9.20, Optimization 5.6.1 via ../.., Test
1.11.0); the Aqua testset runs green (10/10 pass). No sublibrary runs QA
today (the three that list Aqua in [extras] never call it), so per the
preserve-existing-behavior rule none were given a qa env.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ase set

Set every lib/*/test/test_groups.toml [Core] group to the canonical
base/standard version matrix ["lts", "1", "pre"], dropping the redundant
"1.11" entry so the standard test matrix matches the uniform monorepo
contract (base = lts/1/pre).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the root package follow the canonical group-folder contract:

- runtests.jl: QA (Aqua + JET, isolated test/qa environment) was running
  inside the Core/All block, which both pulled the isolated qa env into the
  base run and ran QA on the Core version matrix. Move it into its own
  `GROUP == "QA"` branch so the base/Core run stays in the main test env and
  QA runs only when its group is selected.
- Add root test/test_groups.toml declaring Core (lts/1/pre), QA (lts/1),
  and GPU (1, self-hosted) with the canonical version matrix.
- CI.yml: run Core on [lts, 1, pre] and QA on [lts, 1] via a group/version
  matrix instead of a single hardcoded Core/[1.11, lts] job.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert TagBot.yml from the inline JuliaRegistries/TagBot form to the
canonical SciML/.github tagbot.yml@v1 thin caller, with a
tagbot-subpackages matrix over the 26 registered lib/* sublibraries
(OptimizationQuadDIRECT excluded: not registered in General).

Reduce the downgrade callers' skip lists to only the genuine non-stdlib,
non-sublib extra (SymbolicAnalysis); stdlibs and in-repo sublibraries are
now auto-populated by the centralized downgrade workflows. Move
julia-version from 1.11 to lts (the minimum-supported floor).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 8, 2026 09:30
@ChrisRackauckas ChrisRackauckas merged commit bdaa669 into SciML:master Jun 8, 2026
12 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants