Major performance regression after opting RFLUFactorization out of split dual AD path

[hersle]

@ChrisRackauckas I think something is wrong with #1041. Found when solving ODEs:

using OrdinaryDiffEqRosenbrock, LinearSolve, ForwardDiff, RecursiveFactorization

f!(du, u, p, _) = du .= u.^2 .+ p.*u
prob = ODEProblem(f!, randn(500), (0.0, 0.1), [1.0])
alg = Rodas5P(; linsolve = LinearSolve.RFLUFactorization())
loss(p) = solve(remake(prob; p), alg; save_everystep=false, abstol=1e-6, reltol=1e-6).u[end]

@time ForwardDiff.derivative(loss, 1.0);

Before c0544d2 (i.e. LinearSolve 3.85.1):

  0.035590 seconds (268 allocations: 15.641 MiB)

After c0544d2 (i.e. LinearSolve 3.85.2):

  1.574712 seconds (1.05 k allocations: 169.178 MiB, 1.27% gc time)

This is with ] dev-ed LinearSolve and git checkout c0544d29 vs. git checkout c0544d29~.

[hersle]

Profiling after c0544d2:

@profview ForwardDiff.derivative(loss, 1.0);

[hersle]

Also worried that PureKLUFactorization may be affected, which was also opted out in the same PR. Could explain why it performs so bad for the example in #1048.

[ChrisRackauckas]

I cannot reproduce it being slower in isolation. There must be some other explanation for your case.

[hersle]

Thanks for checking! I will see if I am able to narrow it down.

[oscardssmith]

what cpu do you have?

[hersle]

@ChrisRackauckas @oscardssmith See #1064 with an actual reproducible example using PureKLU. CPU is listed there.