Solve-time developer hooks: par-loader + forcedPars + dydt forcing - #1127
Open
mattfidler wants to merge 23 commits into
Open
Solve-time developer hooks: par-loader + forcedPars + dydt forcing#1127mattfidler wants to merge 23 commits into
mattfidler wants to merge 23 commits into
Conversation
Add a registry of package-supplied loader callbacks that rxode2 invokes once per solve, after gpars is populated and before the parallel integration, so a package can overwrite reserved par_ptr slots with externally-owned values (e.g. neural-network weights held in a torch module, trained outside nlmixr2). - rxRegisterParLoader / rxRemoveParLoader registered as C callables; typedef t_rxParLoader and prototypes exported in inst/include/rxode2.h. - Loaders run single-threaded at the assignGpars call site; gpars is passed with npars and ncols (>= nsub*nsim and nPopPar) so a population-constant block is written to every column. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… table Move rxRegisterParLoader / rxRemoveParLoader off R_RegisterCCallable and onto rxode2's positional external-pointer table (indices 81/82), the CRAN-preferred, ABI-safe cross-package mechanism (see CLAUDE.md). t_rxParLoader moves outside the __RXODE2PTR_H__ guard so downstream packages see the typedef; rxode2ptr.h gains the two typedefs, iniRxodePtrs0 assignments, and NULL-inits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rxRegisterParLoader supports several registered loaders (a fixed-capacity array cycled in rxCallParLoaders). Add test-only sentinel loaders A/B (write 111/222 to parameters 0/1) with _rxode2_rxRegisterTestParLoaders/_rxRemoveTestParLoaders, and a testthat test: two loaders both apply in series, one applies alone, and removal restores pass-through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/init.c
…d object Parameters written by a par-loader hook (e.g. trained neural-network weights supplied out-of-band rather than in the params vector) are now captured on the solved object and restored when re-solving from it -- so a solved/fitted object reproduces those values even in a session without the injecting package. - rxCallParLoaders diffs the population parameter block before/after the loaders to record exactly which parameters were injected; _rxode2_rxGetInjectedPars returns them (0-based indices + values). - .rxSolveMaterializeParams saves them on the object as .injectedPars (mapped to parameter names); rxInjectedPars() exposes them. - rxSolve.rxSolve applies them (override) when re-solving from a solved object. - test-par-loader.R: inject -> save on object -> remove loaders -> re-solve restores the injected values; a non-injected solve reports NULL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a registry of dydt forcing hooks the generated model invokes at the end of its RHS (dydt) evaluation, letting a plugin ADD forcing to state derivatives -- the par-loader analogue for the RHS. This is the mechanism NN-weight forward-sensitivity (variational) states use to receive their b_j term on top of the J*s_j part rxode2's sensitivity codegen already produces. - rxRegisterDydtForce / rxRemoveDydtForce exposed to downstream packages via the function-pointer table (indices 84/85, nVec 86); rxCallDydtForce R_RegisterCCallable'd for generated model code. - codegen: model resolves rxCallDydtForce at DLL load and calls it at the end of dydt (no-op branch when nothing is registered, so normal models are unaffected). Declaration + resolution added to the rxode2_model_shared.c template so the generated codegen2.h picks them up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaQkwAM1PeW5msx6EFSyKD
…nsitivity Adds the etaFD() model-block directive baked into modelVars$etaFD (per-param 0/1 flag, RxMv_etaFD=33), with parser plumbing (tran.g grammar + parse*.h), genModelVars emission, rxData char->modelVars builder, mu-ref/symengine directive allowlisting, and rxUiGet.etaFDLines. Used by nlmixr2est to force FD sensitivity of an eta (e.g. invisible NN-weight etas). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A rxUdfUi modification function can append etas to the model iniDf during parsing
(e.g. individual neural-network weight etas). .env$eta was set once from the
ini({}) omega matrix before those UDFs ran and never refreshed, so UDF-appended
etas were present in the (authoritative) iniDf but missing from the eta list --
and thus mis-classified as covariates by mu-referencing / covariate detection
(the fit then failed a data-name check). Refresh .env$eta from the final iniDf
(ordered by neta1) right after `.env$iniDf <- .env$df`. No-op for models whose
etas are all declared in ini({}).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reverts 2358f71. The etaFD() directive existed only to force finite-difference sensitivity of invisible NN-weight etas (nni); the DeepPumas-style latent-input random effect uses analytic input derivatives instead, so the directive is unused. Shrinks modelVars back to 33 elements (drops RxMv_etaFD=33), so nlmixr2est must be rebuilt. The generated tran.g.d_parser.h reverts to its pre-etaFD state (grammar was untouched since that commit). The unrelated eta-refresh fix in err.R is retained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Adds a hidden named-numeric `forcedPars` slot to the rxUi (stored in the model meta env, so it travels through piping and into fits) plus public `rxForcedPars()` / `rxForcedPars<-()` accessors. At solve setup the ui's forced values are resolved to solve-param indices and injected into every gpars column at rxCallParLoaders (rxData.cpp) -- the one point every solve path hits after gpars assignment -- overriding params/data/inits. This is the first-class successor to the global par-loader hook: a model/fit carries its externally-owned values (e.g. trained NN weights) and re-solves, predicts and simulates with them, self-contained and portable. C: _rxForcedIdx/_rxForcedVal buffer + _rxode2_rxSetForcedPars/ClearForcedPars, injected before registered loaders in rxCallParLoaders (also captured by the existing injected-pars diff, so rxInjectedPars() reports them). R: rxForcedPars getter/setter, .rxApplyForcedPars honored in rxSolve.rxUi. Test: test-forced-pars.R (override, unset->unchanged, non-param names ignored, piping carry). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Adds a 'Forced parameters' section to the rxode2 covariates vignette hunk with a runnable rxForcedPars()/rxForcedPars<-() + rxInjectedPars() example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Storing forcedPars in the model meta env printed it with the model. Store it directly on the ui environment instead (funPrint only deparses meta, so it stays hidden) and register 'forcedPars' in the model's sticky vector so it survives model piping (including significant model-block changes, via .getKeepEnv). rxUiCompress/Decompress round-trip preserves it. Test: forcedPars absent from the printed model + carried across a model() pipe (and still applied on the piped model's solve). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
The iniDf-based eta refresh (37a9454) folded occ-level (IOV) etas into the id-level .env$eta list; a mu-referenced 'theta + eta + iov' then parsed as two population etas and failed with 'currently do not theta + eta1 + eta2'. Exclude names already classified in .env$level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New pkgdown article documenting the extension hooks a downstream package uses to change what an rxode2 solve sees: rxForcedPars() (R-level forced parameters carried with the model), the C par-loader hook (rxRegisterParLoader, inject a par_ptr block each solve), the C dydt forcing hook (rxRegisterDydtForce, add a derivative term), and the function-pointer table (.rxode2ptrs()/iniRxodePtrs) that shares rxode2's C entry points cross-package. Companion to the custom-functions article; referenced by nlmixr2est's "Extending nlmixr2est" vignette. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Par-loaders were a GLOBAL registry -- every registered loader ran on every solve,
so a package's parameter injector (e.g. an nn-weight loader) could overwrite an
UNRELATED model's par_ptr just by being registered. Now a loader may be registered
NAMED ("<package>:<function>") via rxRegisterParLoaderNamed (pointer-table index 86),
and runs ONLY while a model flags that name; unnamed loaders keep the legacy
always-run behavior.
- rxData.cpp: registry stores {name, cb}; rxCallParLoaders skips a named loader
unless its name == the active flag (_rxActiveParLoader). New C entry points
_rxode2_rxSetActiveParLoader / _rxode2_rxClearActiveParLoader.
- R: rxParLoader()/rxParLoader<- store the injector name on the ui (sticky, like
rxForcedPars); rxSolve.rxUi bridges it (set active flag before the solve, clear
after) alongside .rxApplyForcedPars.
- init.c: register the two .Call entries + the named-register pointer (nVec 86->87);
rxode2.h + rxode2ptr.h expose rxRegisterParLoaderNamed to downstream packages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
… article Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds solve-time extensibility points so downstream packages can inject/override parameters and force derivatives during rxSolve() without patching rxode2, plus fixes an eta-refresh edge case after UDF parsing.
Changes:
- Introduces C-level hooks: par-loader registration (incl. named dispatch) and a dydt forcing callback, exported via the rxode2 function-pointer table.
- Adds R-level “sticky” per-model features:
rxForcedPars()(solve-time parameter overrides) andrxParLoader()(opt-in to a named par-loader), and persists injected parameters on solved objects. - Adds tests + documentation for the new APIs and a regression test for IOV-eta handling.
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/rxData.cpp |
Implements global registries for par-loaders, forced parameters, injected-parameter capture, and dydt forcing dispatch. |
src/init.c |
Extends the function-pointer table and registers new .Call entry points / C-callables. |
src/codegen.c |
Emits a call to the dydt forcing callback at the end of generated RHS evaluation. |
R/rxsolve.R |
Adds rxForcedPars, rxParLoader, rxInjectedPars, and solve-time plumbing to set/clear C-side buffers and restore injected params on re-solve. |
R/err.R |
Refreshes .env$eta from iniDf post-UDF parsing and excludes IOV etas. |
tests/testthat/*.R |
Adds coverage for par-loader ordering, persistence/restoration of injected params, forcedPars behavior, and IOV-eta regression. |
vignettes/articles/rxode2-solve-hooks.Rmd & man-roxygen/rmdhunks/covariates.Rmd |
Documents new solve-time hooks and forced parameter usage. |
inst/include/*.h |
Exposes new typedefs and function-pointer table entries to downstream packages. |
NAMESPACE |
Exports new R APIs/setters/getters. |
Files not reviewed (2)
- man/rxForcedPars.Rd: Generated file
- man/rxInjectedPars.Rd: Generated file
Comment on lines
+1369
to
+1374
| .etaDf <- .env$df[!is.na(.env$df$neta1) & | ||
| !(.env$df$name %in% .env$level), , drop=FALSE] | ||
| if (nrow(.etaDf) > 0L) { | ||
| .etaDf <- .etaDf[order(.etaDf$neta1), , drop=FALSE] | ||
| .env$eta <- unique(.etaDf$name) | ||
| } |
Comment on lines
3357
to
+3358
| if (is.null(params)) params <- object$.params.single | ||
| params <- .rxApplyInjectedPars(params, object) |
| } | ||
| .model <- as.character(.model) | ||
| if (is.null(params)) params <- object$.params.single | ||
| params <- .rxApplyInjectedPars(params, object) |
Comment on lines
+3375
to
+3382
| .rxApplyInjectedPars <- function(params, object) { | ||
| .inj <- rxInjectedPars(object) | ||
| if (is.null(.inj) || length(.inj) == 0L) return(params) | ||
| if (is.numeric(params) && is.null(dim(params)) && !is.null(names(params))) { | ||
| params[names(.inj)] <- .inj | ||
| } | ||
| params | ||
| } |
Comment on lines
+4528
to
+4535
| static void rxRegisterParLoaderImpl(const char* name, t_rxParLoader cb) { | ||
| if (cb == NULL) return; | ||
| for (int i = 0; i < _rxNParLoaders; ++i) if (_rxParLoaders[i] == cb) return; | ||
| if (_rxNParLoaders < RX_MAX_PAR_LOADERS) { | ||
| _rxParLoaderNames[_rxNParLoaders] = (name == NULL) ? std::string() : std::string(name); | ||
| _rxParLoaders[_rxNParLoaders++] = cb; | ||
| } | ||
| } |
Comment on lines
+4561
to
+4563
| // The active injector flag for the next solve (set from the model that is about | ||
| // to be solved), consumed by rxCallParLoaders and then cleared. | ||
| extern "C" SEXP _rxode2_rxSetActiveParLoader(SEXP nameSxp) { |
Comment on lines
+3649
to
+3655
| if (is.null(names(value)) || any(names(value) == "")) { | ||
| stop("forcedPars must be a fully-named numeric vector", call. = FALSE) | ||
| } | ||
| ## store on the ui env (hidden -- not the printed `meta` block) and mark sticky | ||
| ## so it survives model piping. | ||
| assign("forcedPars", stats::setNames(as.numeric(value), names(value)), | ||
| envir = .ui) |
- err.R: make the post-UDF eta refresh authoritative -- clear .env$eta (to NULL, matching the init) when only IOV/no etas remain, so a stale omega-derived name cannot leak. - rxsolve.R: .rxApplyInjectedPars now restores injected params onto a parameter data.frame or matrix (not just a single named vector) and falls back to the solved object's stored .params.dat when there is no single named vector. - rxForcedPars<-: reject non-numeric input up front instead of letting as.numeric() silently coerce characters/factors to NA. - rxData.cpp: warn when the par-loader registry is full (registration usually runs in .onLoad, so a dropped loader must be diagnosable); correct the active-flag comment to describe the real lifecycle (cleared by the R on-exit hook, not by rxCallParLoaders). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
… solve rxRegisterUiPrep(name, fn) / rxRemoveUiPrep(name) register functions rxode2 calls with the ui at the start of rxSolve.rxUi (before parameter loaders run). A package uses this to rebuild transient C-side state from serializable ui slots after a ui has been saved to disk and reloaded in a fresh session -- e.g. re-registering neural-network shapes so the weight values already carried in rxForcedPars() land in a network that can stride them. Hook errors are downgraded to a warning so a buggy plugin cannot break unrelated solves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Explain the ui-prep hook as the reload-safety companion to the C par-loader: how to rehydrate transient C state from serializable ui slots, resolve positions by name, and the nlmixr2nn weight-persistence pattern (weights in rxForcedPars, shapes in a sticky nnMeta slot re-registered by the prep hook). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 19 changed files in this pull request and generated 8 comments.
Files not reviewed (4)
- man/rxForcedPars.Rd: Generated file
- man/rxInjectedPars.Rd: Generated file
- man/rxParLoader.Rd: Generated file
- man/rxRegisterUiPrep.Rd: Generated file
| } | ||
| } | ||
| if (show_ode == ode_dydt){ | ||
| sAppendN(&sbOut, " if (_rxDydtForceCb != NULL) _rxDydtForceCb(_neq, __t, __zzStateVar__, __DDtStateVar__);\n", 90); |
| // before the parallel per-subject solve, so callbacks writing gpars are safe. | ||
| // gpars is laid out npars per column with `ncols` columns (>= nsub*nsim and | ||
| // nPopPar); a population-constant block is written to every column. | ||
| typedef void (*t_rxParLoader)(rx_solve* rx, double* gpars, int npars, int ncols); |
| // already produces. Called inside the per-subject solve (possibly parallel), so | ||
| // callbacks must be thread-safe (read-only shared state, write only their own | ||
| // dydt slots). neq[0]=nstate, neq[1]=cSub. | ||
| typedef void (*t_rxDydtForce)(int *neq, double t, double *y, double *dydt); |
Comment on lines
+4671
to
+4676
| for (int k = 0; k < npars; ++k) { | ||
| if (_globals.gpars[k] != pre[k]) { | ||
| _rxInjIdx.push_back(k); | ||
| _rxInjVal.push_back(_globals.gpars[k]); | ||
| } | ||
| } |
Comment on lines
+4656
to
+4663
| // ui-driven forced parameters: write to every subject/sim column first, so a | ||
| // registered loader (if any) can still override them. | ||
| for (size_t f = 0; f < _rxForcedIdx.size(); ++f) { | ||
| int k = _rxForcedIdx[f]; | ||
| if (k < 0 || k >= npars) continue; | ||
| double v = _rxForcedVal[f]; | ||
| for (int c = 0; c < ncols; ++c) _globals.gpars[(size_t)c * npars + k] = v; | ||
| } |
| ## force WT = 2 -> cl doubles -> faster decay -> strictly lower cp at every t>0 | ||
| rxForcedPars(ui) <- c(WT = 2) | ||
| s1 <- rxSolve(ui, ev) | ||
| expect_true(all(s1$cp < s0$cp)) |
| ## solve setup, the par-loader injection point). This is the plugin-free forcing | ||
| ## mechanism that lets a model/fit carry externally-owned values (e.g. trained | ||
| ## neural-network weights) and stay self-contained. | ||
|
|
| --- | ||
| title: "Solve-time hooks for package developers" | ||
| --- | ||
|
|
# Conflicts: # inst/include/rxode2.h # src/init.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the solve-time developer hooks that let an external package inject
parameters and force state derivatives at solve time without patching rxode2.
This is the rxode2 half of the NN-in-ODE work (
nlmixr2nn); the hooks aregeneral-purpose and documented for any package developer.
What's new
Parameter-block loader hooks
rxRegisterParLoader()/ remove -- register a C callback that rewrites theparameter block at solve time; multiple loaders are applied in series.
rxParLoader()flag so a model only fires theinjector it opted into (prevents one model's loader from clobbering another's
parameters).
rxode2ptr.h).forcedPars-- plugin-free parameter forcingrxForcedPars()stores a per-ui parameter override as a hidden, sticky ui slot(not printed), applied at solve time.
Derivative forcing
dydtforcing hook for plugin state-derivative forcing.UI fixes
iniDfafter UDF parsing, and keep IOV etas outof that post-UDF eta refresh.
Docs
API and the
rxParLoaderflag.rxForcedParsdocumented in the covariates vignette.Notes
experimental
etaFD()directive; the net API surface is the loader/forcedParshooks above.
feat/nn-sens-contrib(likelihood-contribution +estimation-interceptor API).
nlmixr2nndepends on both.🤖 Generated with Claude Code
https://claude.ai/code/session_01MmVREEBZmrR1nfv3cFUd39