Fix and update AdaptiveSDE benchmarks (#126)#1605
Conversation
Modernize the AdaptiveSDE benchmark set so it builds under the current SciML interfaces and the Weave build server. - Replace manual Distributed parallelism (addprocs/@everywhere/ ParallelDataTransfer.sendto) with multithreaded EnsembleProblem + EnsembleThreads(), which Weave can execute. - Drop the removed DiffEqMonteCarlo dependency; the Monte Carlo runs now use the standard EnsembleProblem interface. - Rename DiffEqBase.calculate_monte_errors -> calculate_ensemble_errors (error_means/error_medians/elapsedTime fields are unchanged) and add DiffEqBase as a direct dependency. - Fix solver keyword drift: maxIters -> maxiters; keep SciMLLogging.None() for verbosity (Bool verbose is no longer supported). - Use ctx.sim_id (EnsembleContext) for per-trajectory seeding and index trajectories via sol.u[i] rather than the flattened sol[i]. - qmaxDetermination: the 'other problems' loop now builds the ensemble from probs[k] instead of always re-running problem 1. - Update Project.toml/Manifest.toml: add DiffEqBase, drop unused Distributed/ParallelDataTransfer, refresh the manifest.
|
@ChrisRackauckas the CI run for Could you let me know what runtime limit or parameter scale you'd want here, trajectories, length of the Let me know if you have any thoughts, happy to iterate further on your suggestions. |
|
Did the runner actually pick up this benchmark? Iterations are never a problem as they are expected to run for longer duration. If it timed out because no runner picked it(which looks like the case here) you will have to wait |
|
Thanks @AdityaPandeyCN, so it's runner availability, not the workload. I'll keep the parameters at full scale then. Happy to wait for a runner to pick it up. |
|
|
Thanks. I looked into it and the fails are flat across qmax in every configuration I tried, so the section can't determine a qmax as written. Loosening the tolerance or How do you want to handle it: drop the oval2 qmax section, swap in a problem or solver where qmax actually matters, or change the metric? |
|
It should be in the new controller setup arguments. |
The qmax solve keyword is inert for SRIW1 in the current stack: StochasticDiffEqCore's solve accepts qmax but does not forward it to the integrator, so every qmax value gave identical results. Set qmax on the step-size controller instead. Pass controller = PIController(SRIW1(); qmax = q) in both the Oval2 and the other-problems sweeps. PIController(SRIW1(); qmax = qmax_default(SRIW1())) reproduces SRIW1's default stepping exactly, so this varies only qmax. Add OrdinaryDiffEqCore to deps for PIController.
|
Done, routed qmax through the controller in both sweeps. On the Oval2 test though, it still fails ~100% across all qmax. Should I leave it as-is, or swap it for a solver/tolerance that completes? |
|
Why does it fail? If you remove the try/catch stuff, what is it saying? |
|
No try/catch, just verbose = SciMLLogging.None(). |
|
It shouldn't be failing like that? |
|
Right, it's the solver not the problem. |
Oval2 with large fluctuations is a stiff SDE with a tight stability region, so SRIW1 (not stability-optimized) collapses its step size to eps and fails ~100% at any tolerance, leaving the qmax sweep with no signal. SOSRI solves it, and the sweep now differentiates (0 fails at qmax around 1.03, degrading as qmax grows). Tolerances abstol=2^-13, reltol=2^-7 match the StiffSDE Oval2 benchmarks.
|
Swapped the Oval2 test to SOSRI, qmax curve is clean now. |
|
There are a bunch of adaptive failures. See an old version to see what you should be getting when it's corrected. Also, the figures of the other one need to transpose the labels to fix the plot legend. |
…erance - AdaptiveEfficiencyTests.jmd: make the RSwM1/2/3 legend a row vector so Plots maps each label to the correct curve (fixes the mislabeled plot legend). - qmaxDetermination.jmd: use the reference abstol=2^-15, reltol=2^-10 for the Oval2 qmax sweep (matches the original benchmark).
|
Transposed the labels on |
|
The time stepper is probably very different from before, so it might need to be tuned. |
The current adaptive stepper is more aggressive than the 2019 reference, so the default safety factor (gamma=0.9) overshoots and fails a large fraction of paths at high qmax. Setting gamma=0.7 in the PIController gives 0 adaptive failures across the whole qmax sweep (verified at 100 trajectories/qmax), matching the reference output, with the qmax signal carried by the timing.
|
That did it. Dropping the controller safety factor to |
3 participants
Fixes #126. This is the AdaptiveSDE small-grant work (declared in SciML/sciml.ai#242).
The benchmarks weren't building. The main blocker was the manual
Distributedparallelism (addprocs/@everywhere/ParallelDataTransfer.sendto), which Weave can't run, plus some bitrot against the current interfaces. Both files now useEnsembleProblem+EnsembleThreads()and weave cleanly.A few things changed along the way. Dropped
DiffEqMonteCarlo(the Monte Carlo runs go throughEnsembleProblemnow) and the unusedDistributed/ParallelDataTransfer. Renamedcalculate_monte_errorstocalculate_ensemble_errors(fields unchanged) and addedDiffEqBaseas a dep. Fixed the keyword drift too,maxItersbecamemaxiters, and I keptSciMLLogging.None()since aBoolverboseisn't accepted anymore. Also moved toctx.sim_idfor per-trajectory seeding andsol.u[i]for indexing. And there was a small existing bug inqmaxDeterminationwhere the "other problems" loop was re-running problem 1 instead ofprobs[k], that's fixed now.Project.toml/Manifest.tomlare updated to match.I validated both files by weaving them locally at reduced and medium scale (
Ns=[5,5,5], 200 trajectories, full qmax sweep). No errors, and the plots regenerate fine.One thing to flag, the qmax-on-Oval2 runs all hit
DtLessThanMinacross the whole sweep (SRIW1 atabstol=2^-15on Oval2). This matches the original solve settings so the port is faithful, but that section won't show qmax differentiation as-is, happy to relax the tolerances there if you'd prefer.