[Codex analysis][Acados v0.5.5][5/5] Runtime numerical parameters

[mickaelbegon]

Status

WIP — Codex-generated analysis and implementation. No pull request is open.

This is step 5 of the Acados v0.5.5 integration roadmap tracked in #1088. It is stacked on the diagnostics work in #1092.

• Analysis branch: mickaelbegon:codex/acados-v055-runtime-parameters
• Runtime-parameter commit: f15ab93a
• Exact primal-dual warm-start commit: 4b3bd5da

Motivation

Bioptim already represents node-wise measured data through numerical_data_timeseries, for example external-force profiles. These values appear as symbols in dynamics, costs, and constraints, but the Acados interface previously declared AcadosModel.p as empty. Consequently, such problems could not use the data through Acados, and repeated MHE/RHO solves had no managed path for updating them without regenerating the solver.

This path is relevant to warm-start and homotopy workflows when the new OCP uses updated measured data or a resampled grid: the trajectory warm start and the node-wise runtime data can now be updated independently.

Current WIP scope

The branch:

• maps Bioptim numerical_data_timeseries symbols to stage-wise AcadosModel.p;
• flattens numerical values in the same key/component order used to declare the symbols;
• initializes AcadosOcp.parameter_values for code generation;
• updates every shooting node and the terminal node before each solve;
• performs full parameter updates on the first solve or when a whole stage changes;
• uses Acados v0.5.5 set_params_sparse when only part of a stage changed;
• caches a detached copy of the last synchronized values;
• rejects changes to the number of runtime parameters after code generation with an explicit rebuild message.

Bioptim optimization parameters are intentionally unchanged: they remain augmented decision states in the current Acados transcription.

WIP follow-up: exact-grid primal-dual warm start

The first post-roadmap priority is now integrated on the same branch for structurally identical Acados OCPs:

• Solution carries a dedicated opaque solver_state, separate from generic diagnostics and IPOPT-like multipliers;
• the Acados backend captures detached flattened copies of x, u, pi, lam, sl, and su with get_flat_iterate();
• Solution.copy() deep-copies this state;
• AcadosInterface.set_lagrange_multiplier() now implements Bioptim's common warm-start hook and queues the Acados state when present;
• the queued iterate is restored after runtime data, bounds, references, and generic primal initial guesses are updated, and immediately before the native solve;
• the format version, horizon length, field presence, and every flattened field dimension are checked before restoration;
• solutions without an Acados state retain the previous state/control/parameter initial-guess behavior.

Algebraic z is intentionally excluded because the Bioptim Acados backend still rejects algebraic variables.

This first implementation is exact-grid only. It does not yet:

• interpolate a trajectory to another grid;
• shift or interpolate dual variables for an advancing RHO window;
• project an interpolated primal trajectory onto dynamics/constraints;
• decide whether a primal-dual warm start is better than Acados' retained internal memory for a given RHO workflow.

Those points remain part of the grid-homotopy and benchmark follow-up.

Validation

Using the Acados v0.5.5 Python interface and native libraries:

pytest tests/shard1/test_acados_interface.py -q --disable-warnings
36 passed

The runtime-parameter test solves the same OCP twice, modifies two components at two different nodes, verifies the two sparse-update calls, and reads the resulting p vectors back from Acados.

The new warm-start test:

1. solves an OCP and captures its full supported flattened iterate;
2. overwrites the mutable native Acados iterate;
3. verifies that the Solution snapshot remained detached;
4. solves again with warm_start=first_solution;
5. verifies immediately before the native solve that all six supported fields exactly match the first snapshot;
6. verifies that Solution.copy(skip_data=True) owns a deep copy of the solver state.

An additional native check used Bioptim's external-force example. It converged with status 0 and 18 runtime parameters per node, confirming that dynamics depending on numerical_data_timeseries now compile and solve.

Why p_global is deferred

Acados v0.5.5 also provides p_global and set_p_global_and_precompute_dependencies, which can be valuable for large shared lookup tables or coefficients. Bioptim currently has no public distinction between global runtime data and optimization parameters, so automatically promoting values to p_global would create ambiguous semantics and could change differentiation behavior.

A later focused design could add an explicit Bioptim global-runtime-data concept before exposing this Acados optimization.

Post-roadmap Acados v0.5.5 API audit

The five-step upgrade does not exhaust the useful v0.5.5 API. The following capabilities are relevant to Bioptim, especially to warm-start, homotopy, MHE/RHO, Cocofest, and repeated solves.

Acados capability	Potential Bioptim use	Current WIP status or gap
Complete flattened iterates (x, u, z, pi, lam, sl, su) through get_flat_iterate() / set_iterate()	Preserve a high-quality primal-dual warm start between RHO iterations; interpolate the primal part when the grid changes, then initialize or project the dual part	Exact-grid capture/restore of x/u/pi/lam/sl/su is implemented in commit 4b3bd5da. Grid transformation, RHO shifting, projection, and z remain open
set_new_time_steps() and update_qp_solver_cond_N()	Fixed- or changed-cardinality grid homotopy without new code export/compilation; non-uniform node placement around fast dynamics	Bioptim initializes Acados with only tf and N_horizon; no public Acados path forwards non-uniform time_steps or updates a compiled solver's grid
SQP-RTI / AS-RTI preparation and feedback phases through runtime rti_phase and as_rti_level	Separate preparation from feedback in real-time MHE/RHO and reduce feedback latency	Solver modes can now be selected, but AcadosInterface.solve() always performs one monolithic solve() call and does not expose the two-phase scheduling contract
p_global, dependency precomputation, value/solution sensitivities	Efficient shared splines, lookup tables, model coefficients, outer-loop identification, differentiable tuning	Bioptim has no solver-independent global runtime-data concept. Its optimized parameters are augmented into x, which is semantically correct for decisions but increases the Acados state/QP dimension and is not a replacement for fixed p_global data
AcadosOcpBatchSolver	Parallel parameter sweeps, multi-start, scenario evaluation, and batched sensitivity calculations while sharing generated code	Bioptim MultiStart creates independent OCPs and uses process-level parallelism; there is no batch Acados backend or batched Solution representation
AcadosMultiphaseOcp	Native phase transitions, contacts, changing dynamics, and multi-phase RHO/Cocofest problems	AcadosInterface rejects ocp.n_phases > 1
Algebraic variables and implicit/DAE models	Equilibrium constraints and DAE formulations without manually eliminating algebraic variables	Acados supports z, but the Bioptim interface rejects algebraic states before export
Stage-wise constraint/bound updates	Time-varying envelopes, measurements, contacts, and RHO window-dependent bounds	Acados exposes stage-wise lbx/ubx/lbu/ubu/lh/uh, while Bioptim restricts state/control bound interpolation, requires control bounds to be identical at shooting nodes, and accepts nonlinear constraints only at all shooting nodes or the terminal node
EXTERNAL and CONVEX_OVER_NONLINEAR costs	More direct scalar costs, exact/custom Hessians, and convex-over-nonlinear structure	Bioptim's Acados backend currently implements only LINEAR_LS and NONLINEAR_LS; EXTERNAL is explicitly rejected and CONVEX_OVER_NONLINEAR is not mapped
custom_update()	Low-overhead compiled updates for complex online data or problem-specific precomputation	No safe Bioptim API or lifecycle is defined for compiling and invoking an Acados custom update

This audit is specific to the Acados v0.5.5 source embedded on the analysis branch. It is not an audit of features added after v0.5.5.

Bioptim design choices that currently limit Acados

Some restrictions are deliberate compatibility choices rather than missing setters:

1. Solver-independent Solution previously lost Acados-specific dual structure. The WIP now adds a dedicated opaque solver_state, avoiding exposure of Acados ordering as the generic Bioptim multiplier format. Maintainers should confirm whether this generic extension point is acceptable before a PR.
2. Optimization parameters are represented as constant augmented states. This lets Acados optimize them, but adds zero dynamics and enlarges every shooting-node state. Keep this for true decision variables; introduce a distinct immutable/updateable data category for stage-wise p and global p_global.
3. One shared symbolic dynamics graph per phase is mandatory. Runtime numerical variation is now correctly represented by stage-wise p; ONE_PER_NODE should only be needed when the graph or dimensions actually change. Native Acados multiphase support is the better path for structural changes.
4. Time is reduced to a scalar phase duration. This hides Acados non-uniform grids and prevents in-place grid homotopy. A grid descriptor should separate node count, interval durations, and final time while preserving the current uniform-grid default.
5. The solve lifecycle is monolithic. This is convenient for offline OCPs, but limits preparation/feedback RTI, batch execution, and application-controlled online updates. These should be explicit advanced lifecycle methods rather than hidden behavior in the regular solve().
6. Bounds and constraints are normalized too early to a small set of node patterns. The Acados backend can update more fields stage by stage; preserving node-indexed bounds until backend translation would remove unnecessary restrictions.
7. The backend is SX-only. This is compatible with Acados code generation but means MX-based Bioptim problems must be rebuilt with use_sx=True; this is an expected backend constraint, not a v0.5.5 regression.

Suggested follow-up order

Priority 1 — directly useful for warm-start and RHO

1. WIP implemented for identical grids: add an Acados-specific opaque iterate snapshot and restore path for x/u/pi/lam/sl/su (z remains deferred until algebraic variables are supported).
2. Add a grid-update API around set_new_time_steps() and reinterpolate the primal trajectory using the source collocation polynomial before restoring/projecting dual quantities.
3. Expose explicit SQP-RTI preparation and feedback calls.
4. Forward node-wise state/control/nonlinear bounds instead of enforcing constant intermediate values.

These four items can improve repeated solves without redesigning the full Bioptim model layer. The exact-grid snapshot is now available, but the benchmark of primal-only, primal-dual, and Acados-retained-memory strategies remains necessary. Grid-changing work should allow falling back to Acados' own initialization if interpolated multipliers are not reliable.

Priority 2 — larger model/API work

5. Define solver-independent global runtime data, then map it to Acados p_global and selectively expose sensitivities.
6. Prototype AcadosMultiphaseOcp for a minimal two-phase Bioptim problem before attempting general phase transitions.
7. Add algebraic-state/DAE export with a focused implicit-dynamics example.
8. Map EXTERNAL / CONVEX_OVER_NONLINEAR costs only where Bioptim can state clear Hessian semantics.

Priority 3 — specialized throughput features

9. Evaluate AcadosOcpBatchSolver against the current process-based MultiStart on parameter sweeps and scenario solves.
10. Expose custom_update() only after defining ownership, code-generation, validation, and failure semantics.

Maintainer feedback requested later

Before turning the analysis into PRs, it would be useful to decide whether:

1. modifying ocp.nlp[0].numerical_data_timeseries between solves is sufficient, or a public update method is preferable;
2. sparse updates should remain automatic or be user-selectable;
3. a solver-independent global runtime-data API is desirable before integrating Acados p_global;
4. the current WIP choice of a dedicated opaque Solution.solver_state is acceptable, or should be replaced by a solver-independent primal-dual warm-start structure;
5. non-uniform grid updates belong to the OCP API, the Acados solver API, or an RHO-specific orchestration layer;
6. multiphase and algebraic-state support are in scope for the Acados backend before the warm-start/RTI items are benchmarked.

This issue, branch, and post-roadmap audit were produced as a Codex analysis for maintainer discussion.