Update CUTEst benchmarks to the Optimization.jl interface

[andoriyaprashant]

Summary

Adds a single CUTEst benchmark using the Optimization.jl interface through
OptimizationNLPModels.

This replaces the older fragmented CUTEst benchmark files with one benchmark that covers:

• unconstrained CUTEst problems
• bounded constrained CUTEst problems
• unbounded constrained CUTEst problems
• quadratic problems with linear constraints

Each problem/solver pair is run in a separate Julia process so stalls, crashes and solver
failures are recorded without hanging the SciMLBenchmarks build.

Details

• Uses OptimizationNLPModels.OptimizationProblem to convert CUTEst/NLPModels problems to the Optimization.jl interface.
• Loops over Optimization.jl-compatible solvers.
• Records return codes, completion rates, success rates and median solve times.
• Uses a small known bad skip list and per solve timeout protection.

Validation

Locally ran:

using SciMLBenchmarks
SciMLBenchmarks.weave_file("benchmarks/OptimizationCUTEst", "CUTEst_optimization.jmd")

The benchmark weaves successfully and generates markdown/plots.

Addresses #935

Related small-grant claim: SciML/sciml.ai#236

[andoriyaprashant]

Hey @ChrisRackauckas

This PR is ready for review. I've verified that the benchmark weaves successfully and generates the expected output locally.
Please let me know if there are any changes or improvements you'd like me to make
Thanks

[ChrisRackauckas]

No, please keep the split, having them all in one will make it difficult to read and will likely time out.

[andoriyaprashant]

No, please keep the split, having them all in one will make it difficult to read and will likely time out.

@ChrisRackauckas Yeah, that makes sense. I updated the PR to keep the CUTEst benchmarks split across the original benchmark pages instead of combining everything into one large weave target

[ChrisRackauckas]

Why is this a separate file?

[andoriyaprashant]

This is separate so each CUTEst problem/solver run happens in its own Julia process.

Some CUTEst cases can hang, segfault, OOM or ignore maxtime and if the solve runs directly inside the .jmd weave process then one bad case can stall or kill the whole benchmark build. Keeping this as a small child process entrypoint lets the parent benchmark kill just that solve after the timeout and record it as TIMEOUT/CRASHED/FAILED, while the rest of the benchmark continues.

The parent benchmark calls it with the benchmark folder’s active project, roughly like:

julia --project=benchmarks/OptimizationCUTEst benchmarks/OptimizationCUTEst/solve_cutest_case.jl ROSENBR LBFGS

[ChrisRackauckas]

Rebase, it had an issue with detecting changes.

[andoriyaprashant]

Hi @ChrisRackauckas please let me know if any further changes are required

[ChrisRackauckas]

Runners having issues. Not entirely sure what that is.

[andoriyaprashant]

Runners having issues. Not entirely sure what that is.

@ChrisRackauckas Yeah From the logs, it looks like the failure happens during checkout before any benchmark code is executed, so it doesn't seem to be caused by the changes in this PR.

[ChrisRackauckas]

Found it, an old job was still running that oomed the machine and it needed a restart 😅

[andoriyaprashant]

Found it, an old job was still running that oomed the machine and it needed a restart 😅

Haha, glad you found it 😄. Please let me know if you'd like any further changes or improvements from my side. Happy to make them.

[andoriyaprashant]

Hey @ChrisRackauckas could you please review this pull request when you get a chance

I noticed that the timeline mentions June 26, so I wanted to check if there are any changes needed. If there are any, I'll make them and submit a request to extend the timeline if necessary, then update the PR based on your feedback.

[andoriyaprashant]

Hi @ChrisRackauckas,

I've updated the PR by syncing it with the latest master and fixing the Runic formatting issues. It looks like the remaining GitHub Actions are waiting for maintainer approval to run

Could you please approve the workflows and take another look when you get a chance
Thanks!