Segfault (signal 11, GC heap corruption): reverse-mode over Lux dense RHS repeated in an in-place ODE adjoint (Julia 1.10/1.12, Enzyme 0.13.171)

[ChrisRackauckas-Claude]

Summary

Reverse-mode Enzyme.autodiff over an in-place ODE RHS that calls a Lux.Chain(Dense(2,50,tanh), Dense(50,2)) (i.e. reverse over LuxLib's fused_dense/dense layers), invoked repeatedly inside a SciMLSensitivity adjoint solve (BacksolveAdjoint/QuadratureAdjoint, EnzymeVJP), segfaults with signal 11 (GC heap corruption) on Julia 1.10 and 1.12 (1.11 is fine).

Versions

Julia 1.10 / 1.12, Enzyme v0.13.171, Lux v1.31.4, LuxLib, ComponentArrays. (From SciML/SciMLSensitivity.jl CI — Core5.)

CI crash signatures (signal 11)

• Julia 1.12, BacksolveAdjoint(): crash in gc_mark_outrefs (GC) during an Enzyme CombinedAdjointThunk for the Lux RHS, called repeatedly from _vecjacobian! per reverse step.
• Julia 1.10, QuadratureAdjoint(): Enzyme PrimalErrorThunk in vec_pjac!.

Reproduction status — NONDETERMINISTIC

The corruption is cumulative across many in-place reverse passes; a single reverse pass is always fine. I could not reproduce a deterministic local segfault (100 Zygote.gradient iterations × 4 sensealgs survived on both 1.10.11 and 1.12.6), but it crashes reliably on the SciMLSensitivity CI runners (heavier OptimizationProblem+Adam 200-iter training path, and repeated hybrid-callback adjoints). This is the classic "GC heap corruption that depends on allocation timing/pressure" signature.

Repro scaffold (exercises the exact path; segfault is environment-dependent)

using SciMLSensitivity, OrdinaryDiffEq, Lux, ComponentArrays, Zygote, Random
dudt2 = Chain(Dense(2,50,tanh), Dense(50,2))
ps, st = Lux.setup(Xoshiro(0), dudt2); psc = ComponentArray{Float32}(ps)
dudt(du,u,p,t) = (du .= first(dudt2(u,p,st)); nothing)
z0 = Float32[2.0,0.0]; tspan=(0.0f0,10.0f0); ts=range(tspan...,length=101)
prob = ODEProblem(dudt, z0, tspan)
loss(p) = sum(abs2, Array(solve(remake(prob;p=p,u0=z0), Tsit5();
            saveat=ts, sensealg=BacksolveAdjoint(autojacvec=SciMLSensitivity.EnzymeVJP()))))
for i in 1:200; Zygote.gradient(loss, psc); end   # CI: signal 11 on 1.10/1.12

The differentiated unit is Enzyme reverse over the Lux dense layers in an in-place RHS, run once per reverse-integration step. Happy to provide the exact failing SciMLSensitivity tests (test/Core5/HybridNODE.jl::test_hybridNODE2/3) and a Manifest. Filing primarily so there's an upstream tracking point; I realize the nondeterminism makes this hard — suggestions for getting a deterministic repro (e.g. forcing GC stress) welcome.

[wsmoses]

@ChrisRackauckas-Claude please reduce this to an isolated mwe

[wsmoses]

@ChrisRackauckas can't reproduce this atm, see below.

Reproduction Attempt & Isolation Summary

1. Test Execution Results
   Full SciMLSensitivity ODE pipeline (repro_lux.jl):
   Ran 500 full Zygote.gradient iterations over BacksolveAdjoint(autojacvec=EnzymeVJP()) on an ODE problem with Lux.Chain(Dense(2,50,tanh), Dense(50,2)) in-place RHS, with forced GC.gc() every iteration.
   Result: Completed all 500 iterations cleanly without segfault or heap corruption on Julia 1.12.6.
   Direct Lux Dense layer AD (repro_direct_lux.jl):
   Ran 10,000 iterations of Enzyme.autodiff(Reverse, f, Const, Duplicated(du, d_du), Duplicated(u, d_u), Duplicated(psc, d_p)) directly on the Lux Dense layer under GC stress.
   Result: Completed 10,000 iterations cleanly without memory corruption.
   Multi-Sensealg Comparison (repro_sensealgs.jl):
   BacksolveAdjoint(EnzymeVJP): Passed 100/100 iterations with forced full GC sweeps (GC.gc(true)).
   QuadratureAdjoint(EnzymeVJP): Triggered an EnzymeRuntimeActivityError:
   text
   EnzymeRuntimeActivityError: Detected potential need for runtime activity.
   Constant memory is stored (or returned) to a differentiable variable and correctness cannot be guaranteed with static activity analysis.
   Failure within method: Lux Dense layer internal call (make_abstract_matrix / reshape on SubArray)
2. Findings
   Environmental Dependency: In local isolated testing (Julia 1.12.6, Enzyme v0.13.173), repeated BacksolveAdjoint passes over Lux dense layers do not deterministically segfault locally (confirming the user's observation that GC heap corruption requires specific allocation timing/pressure from heavier training loops or CI runners).
   Activity Mismatch in QuadratureAdjoint: The QuadratureAdjoint path constructs SubArray views over parameters passed to Lux's make_abstract_matrix / reshape, resulting in an EnzymeRuntimeActivityError due to constant-vs-active memory pointer aliasing.

[ChrisRackauckas-Claude]

Got a deterministic, minimal reproducer and a root cause. The nondeterministic segfault has a deterministic error underneath it.

TL;DR

The trigger is a runtime-activity condition in the QuadratureAdjoint + EnzymeVJP path when the ODE carries a callback and the parameters are a ComponentArray. The parameter reaches the differentiated Lux.Chain as a ComponentVector backed by a SubArray view into a plain Vector{Float32}, and Enzyme sees the underlying Memory{Float32} buffer aliased between constant and active use.

• Julia 1.11 / 1.12 (Enzyme 0.13.171): Enzyme raises a clean, catchable EnzymeRuntimeActivityError.
• Julia 1.10 (Enzyme 0.13.171): Enzyme instead compiles a PrimalErrorThunk that calls the primal with the bare Vector{Float32} parameter → applychain does p.layer_1 on an Array → type Array has no field layer_1, and under allocation pressure that bad field access reads unmapped memory → signal 11. This is exactly the reported 1.10 signature (PrimalErrorThunk in vec_pjac!).

So "GC heap corruption that depends on allocation timing" = the same out-of-type field access, sometimes hitting the type check (clean throw) and sometimes reading garbage (SIGSEGV).

Minimal reproducer (deterministic)

Pinned to the issue's versions: Julia 1.10.11, Enzyme v0.13.171, Lux v1.31.4, SciMLSensitivity v7.112.2.

using SciMLSensitivity, OrdinaryDiffEq, DiffEqCallbacks, Lux, ComponentArrays, Zygote, Random

u0    = Float32[2.0, 0.0, 0.0, 0.0]
tspan = (0.0f0, 1.0f0)
net   = Chain(Dense(4, 50, tanh), Dense(50, 2))
ps, st = Lux.setup(Xoshiro(0), net)
p0    = ComponentArray{Float32}(ps)

function rhs!(dx, x, p, t)
    dx[1:2] .= first(net(x, p, st))
    dx[3:4] .= 0.0f0
    return nothing
end
affect!(int) = (int.u[3:4] = -3 * int.u[1:2])
cb   = PeriodicCallback(affect!, 0.1f0, save_positions = (true, true), initial_affect = true)
prob = ODEProblem(rhs!, u0, tspan)

sensealg = QuadratureAdjoint(autojacvec = SciMLSensitivity.EnzymeVJP())
loss(p) = sum(abs2, Array(solve(prob, Tsit5(); u0, p, callback = cb,
    save_everystep = false, save_start = true, sensealg))[1:2, :])

Zygote.gradient(loss, p0)   # 1.10: `type Array has no field layer_1`; 1.11/1.12: EnzymeRuntimeActivityError

• The callback is required — remove it and QuadratureAdjoint completes cleanly. (Backsolve/Interpolating with EnzymeVJP complete fine here; they are the separate, still-nondeterministic 1.12 gc_mark_outrefs path, which I did not manage to reduce.)
• To get signal 11 deterministically on 1.10, run the near-verbatim test_hybridNODE2 form (Optimization + Adam, loss = sum(abs2, ode_data .- pred')) — I reproduced signal 11 on every run of that.

Version behavior (identical MWE above)

Julia	Enzyme 0.13.171 result
1.10.11	type Array has no field layer_1 (throw) / signal 11 under pressure
1.11.9	clean EnzymeRuntimeActivityError
1.12.6	clean EnzymeRuntimeActivityError

(Note on "1.11 is fine" from the OP: with EnzymeVJP forced, 1.11/1.12 also fail on this path — but with a clean throw, never a segfault or memory corruption. So 1.11 never showed the crash; the underlying activity problem is version-independent, only the failure mode (clean error vs UB) is version-dependent.)

The 1.11/1.12 error text:

EnzymeRuntimeActivityError: Detected potential need for runtime activity.
Constant memory is stored (or returned) to a differentiable variable ...
Failure within method:
  d::Dense internal(::Dense{typeof(tanh),...},
     ::Vector{Float32},
     ::ComponentVector{Float32, SubArray{Float32,1,Vector{Float32},Tuple{UnitRange{Int64}},true}, ...},
     ::@NamedTuple{})
  @ Lux .../layers/basic.jl:365
Julia value causing error: Unknown object of type Memory{Float32}

1.10 backtrace at the crash:

type Array has no field layer_1
 [2] applychain           @ Lux/src/layers/containers.jl:577
 [8] PrimalErrorThunk     @ Enzyme/src/compiler.jl:6224
[10] runtime_generic_augfwd(..., ::Chain{...}, ::Nothing, ::Vector{Float32}, ...)  @ jitrules.jl:600
[11] rhs!                 @ mwe.jl
 ... Void @ SciMLBase/utils.jl:513
 ... vec_pjac!            @ SciMLSensitivity/quadrature_adjoint.jl:453
 ... quadgk / AdjointSensitivityIntegrand

Two things fall out

1. Enzyme (1.10): the runtime-activity condition that 1.11/1.12 report cleanly is, on 1.10, turned into a PrimalErrorThunk that runs the primal with a wrongly-typed (Vector vs ComponentVector) argument and then segfaults instead of raising. The 1.10 path should raise the same EnzymeRuntimeActivityError rather than producing UB.
2. SciMLSensitivity: QuadratureAdjoint's vec_pjac! feeds the parameter into the Enzyme VJP as a SubArray-backed ComponentVector that trips static activity analysis; enabling set_runtime_activity(Reverse) on that VJP (workaround (b) from the error message) would side-step it on all versions.

[ChrisRackauckas-Claude]

Now reproduced the other signature too — the Julia 1.12 BacksolveAdjoint → gc_mark_outrefs GC-heap-corruption crash. This one is genuinely a memory-corruption bug (distinct from the QuadratureAdjoint activity issue above), and I found the knob that makes it reliable.

The knob: force constant GC with a small heap hint

Running the near-verbatim test_hybridNODE2 training loop (Optimization + Adam, BacksolveAdjoint(autojacvec=EnzymeVJP())) under a small --heap-size-hint on Julia 1.12.6 crashes within the first 1–3 Adam solves:

• SIGSEGV in gc_mark_outrefs, or
• SIGABRT with GC error (probable corruption) (gc_dump_queue_and_abort).

Crash rate vs. the heap hint (fresh single processes, verified locally):

--heap-size-hint	crashes
150M	3/3 (solo)
100M	2/3 (solo)
300M	1/3 solo, but 4/4 when ~4+ copies run concurrently

Two controls that pin down the trigger:

• Plain repeated Zygote.gradient (no optimizer) survives 4000 iterations, 0/6 crashes, even with GC.gc(true) every step and the same heap hint. The Adam/optimizer allocation churn under heap pressure is required — matching the OP's "heavier OptimizationProblem+Adam path". The --heap-size-hint just removes the need to wait for CI-level memory pressure.
• Smaller heap hint (more frequent GC) ⇒ higher crash probability, consistent with a GC that lands on an object left in a bad state by the reverse pass.

Backtrace (SIGSEGV form) — exact match to the reported signature

signal 11: Segmentation fault
jl_datatype_layout        julia.h:1523            <- reading a GARBAGE type tag
gc_mark_outrefs           gc-stock.c:2356
gc_mark_loop_serial ...
_jl_gc_collect            gc-stock.c:3074
ijl_gc_small_alloc        gc-stock.c:776          <- GC triggered by an allocation...
copy                      ./array.jl:356
augmented_primal          LuxLib/src/impl/matmul.jl:276   <- ...inside Enzyme's aug-primal for matmuladd
matmuladd / fused_dense / Dense / applychain / Chain
ODEfunc                   (Lux dense RHS)
Void / diffejulia_Void_*  SciMLBase/utils.jl
CombinedAdjointThunk      Enzyme/compiler.jl:6240
autodiff                  Enzyme/Enzyme.jl:536
_vecjacobian!             SciMLSensitivity/derivative_wrappers.jl:1079
ODEBacksolveSensitivityFunction   backsolve_adjoint.jl:54
perform_step! (Tsit5)     reverse integration

jl_datatype_layout faulting inside gc_mark_outrefs means the mark phase read a corrupted type tag on an object it was scanning — i.e. it walked into a freed/garbage object. The triggering allocation is the Array copy in LuxLib's augmented_primal for matmuladd, run inside the Enzyme CombinedAdjointThunk that _vecjacobian! invokes on every reverse Tsit5 step. So an object produced/rooted by the generated augmented-primal code is being seen by GC with a bad tag — a GC-rooting/liveness bug in the Backsolve + Enzyme reverse path, surfaced when a GC lands in the middle of that region.

Self-contained reproducer

Same versions as above (Julia 1.12.6, Enzyme 0.13.171, Lux 1.31.4, SciMLSensitivity 7.112.2). Run with:

julia --heap-size-hint=150M repro.jl

using SciMLSensitivity, OrdinaryDiffEq, DiffEqCallbacks, Lux, ComponentArrays, Random
using Optimization, OptimizationOptimisers

Random.seed!(12345)
u0 = Float32[2.0, 0.0, 0.0, 0.0]; tspan = (0.0f0, 1.0f0)

# ground-truth trajectory from a known ODE (gives a nonzero training loss)
function trueODEfunc(dx, x, p, t)
    @views dx[1:2] .= x[1:2] + x[3:4]
    dx[1] += x[2]; dx[2] += x[1]; dx[3:4] .= 0.0f0; return nothing
end
trueaffect!(int) = (int.u[3:4] = -3 * int.u[1:2])
cb_ = PeriodicCallback(trueaffect!, 0.1f0, save_positions=(true,true), initial_affect=true)
ode_data = Array(solve(ODEProblem(trueODEfunc, u0, tspan), Tsit5();
    callback=cb_, save_everystep=false, save_start=true))[1:2, :]

net = Chain(Dense(4, 50, tanh), Dense(50, 2))
ps, st = Lux.setup(Xoshiro(0), net); p0 = ComponentArray{Float32}(ps)
function rhs!(dx, x, p, t)
    dx[1:2] .= first(net(x, p, st)); dx[3:4] .= 0.0f0; return nothing
end
affect!(int) = (int.u[3:4] = -3 * int.u[1:2])
cb = PeriodicCallback(affect!, 0.1f0, save_positions=(true,true), initial_affect=true)
prob = ODEProblem(rhs!, u0, tspan)

sensealg = BacksolveAdjoint(autojacvec = SciMLSensitivity.EnzymeVJP())
predict(p) = Array(solve(prob, Tsit5(); u0=u0, p=p, callback=cb,
    save_everystep=false, save_start=true, sensealg=sensealg))[1:2, :]
loss(p,_) = sum(abs2, ode_data .- predict(p))

optf = OptimizationFunction(loss, AutoZygote())
for rep in 1:60          # crashes within the first few reps under --heap-size-hint=150M
    solve(OptimizationProblem(optf, p0), Adam(0.0025); maxiters = 200)
    GC.gc(true)
    @info "rep $rep survived"
end

If the first process happens to survive, launch a few concurrently (for i in 1..4; julia --heap-size-hint=150M repro.jl & done) — that was 4/4. Happy to attach the full crash logs / a Manifest. Between this and the QuadratureAdjoint MWE above, both CI signatures now reproduce locally on demand.

[ChrisRackauckas-Claude]

Scoping data point: the Backsolve gc_mark_outrefs corruption does not reproduce on Julia 1.10 — it appears specific to the 1.12 GC.

I ran the exact recipe that crashes 1.12 (Optimization + Adam, BacksolveAdjoint(autojacvec=EnzymeVJP()), the test_hybridNODE2 callback loop) on Julia 1.10.11 under progressively harsher GC pressure:

Config (Julia 1.10.11)	Result
8 concurrent, --heap-size-hint=200M	0/8 crash
solo, --heap-size-hint=100M	0/3
solo, --heap-size-hint=150M	0/3
12 concurrent, --heap-size-hint=50M (hint below the live set ⇒ GC on nearly every allocation)	0/12

26/26 clean. Each process ran 60 × 200 Adam iterations (~12k Enzyme reverse passes) to completion; EnzymeVJP was forced (no VJP fallback — the identical code crashes inside Enzyme's CombinedAdjointThunk on 1.12), so the reverse path is genuinely exercised, just without corruption. For contrast, on 1.12.6 this recipe crashes 4/4 concurrent and 3/3 solo at 150M.

So the two signatures split cleanly by version/sensealg:

• 1.10 → the QuadratureAdjoint PrimalErrorThunk-in-vec_pjac! activity bug (deterministic MWE, first comment).
• 1.12 → the BacksolveAdjoint gc_mark_outrefs GC-heap corruption (second comment).

The Backsolve crash landing inside 1.12's rewritten stock GC (gc-stock.c) and not reproducing at all on 1.10's older GC points at a bad interaction between the object rooting/liveness in the generated Backsolve+Enzyme reverse code and the 1.12 GC specifically — likely the most fruitful place to bisect that one.

[ChrisRackauckas-Claude]

Correction and isolation results on the QuadratureAdjoint failure — importantly, my earlier claim that set_runtime_activity fixes it was wrong, and I could not reduce it to a standalone Enzyme call. Everything below verified locally (Julia 1.10.11 / 1.12.6, Enzyme 0.13.171, Lux 1.31.4, SciMLSensitivity 7.112.2).

set_runtime_activity does NOT fix it

I said above that SciMLSensitivity could set_runtime_activity(Reverse) "to fix all versions." That's incorrect — RTA only unmasks the real failure. Building EnzymeVJP(mode = Enzyme.set_runtime_activity(Enzyme.Reverse)) and differentiating the QuadratureAdjoint+callback MWE:

• 1.11 / 1.12 without RTA: clean EnzymeRuntimeActivityError.
• 1.10 / 1.11 / 1.12 with RTA: the activity check is suppressed and execution runs straight into FieldError: type Array has no field layer_1 — the parameter reaches the Lux Chain as its bare backing Vector instead of the ComponentVector.

So the activity error was firing first; the underlying bug is the ComponentArray wrapper being dropped, and RTA just lets you reach it. A reference ReverseDiffVJP computes the correct gradient (‖g‖≈363.7) on the identical problem.

The SciMLSensitivity call site is correctly typed

Instrumenting quadrature_adjoint.jl vec_pjac! at the failing Enzyme.autodiff, both the primal p and shadow out are ComponentVector{Float32, Vector{Float32}, …} — they match. Marking the state active (Duplicated) instead of Const, to mirror the working BacksolveAdjoint _vecjacobian! path, does not fix it either.

It does not reduce to a standalone Enzyme call

I tried to reproduce the inner Enzyme.autodiff in isolation — same in-place Lux-dense RHS, same ComponentArray parameter, same annotations. 8 configurations, all compute the correct gradient (‖dp‖≈8.46), none crash:

standalone Enzyme.autodiff(Reverse, …) config	result
Const(rhs!), state Const, Duplicated(p, dp)	✅ correct
Const(rhs!), state Duplicated	✅ correct
Duplicated(Void(ODEFunction(rhs!)), …), state Const	✅ correct
Duplicated(Void(prob.f)) (FunctionWrappersWrapper), state Const	✅ correct
all four again with Chain/st closure-captured (as in test_hybridNODE2)	✅ correct

So the failure is context-dependent: it only appears when the vec_pjac! Enzyme.autodiff runs inside the full pipeline (QuadratureAdjoint's quadgk integrand → adjoint ODE solve → outer Zygote pass). The same inner call, reproduced directly, always succeeds. The backtrace shows Enzyme entering runtime_generic_augfwd (its type-unstable generic fallback) and passing p there as Vector{Float32} rather than the ComponentVector — but I couldn't force that generic-dispatch path outside the nested-adjoint context.

Net

• No standalone Enzyme MWE — the minimal reproducer remains the SciMLSensitivity one in my earlier comment (QuadratureAdjoint + EnzymeVJP + PeriodicCallback).
• Ruled out as the cause: the set_runtime_activity toggle, the call-site primal/shadow types, the state activity annotation, the Void/ODEFunction/FunctionWrapper wrapping, and closure-capture of the network.
• What's left pointing at Enzyme: the ComponentArray wrapper is dropped only when the differentiation happens through the generic fallback that the nested adjoint context induces. Happy to hand over the full instrumented repro/logs if that helps whoever digs into the Enzyme side.

[wsmoses]

@ChrisRackauckas-Claude can you re-evaluate on current main