Simulation for bluff erosion results in garbage solution after nonlinear solver for mechanical problem fails to converge

[ebayats]

This behavior emerges after ~990 timesteps on a full bluff simulation problem. I can try to recreate it on a smaller problem but it might take some time to create something that will force the behavior I am seeing.

The odd behavior I’m seeing is related to the nonlinear solver for the mechanical problem. The behavior is as follows:

• The nonlinear solver fails to converge
• The time-step is cut
• On the next solve attempt, the nonlinear “converges”, but it converges with a garbage solution (the error is still very high).
• At that point, the entire mesh gets eroded permanently, but I think that’s because of the garbage solution of super high nonphysical displacements/stresses etc.

I observed this running the problem on 4 processors. In serial, it behaves differently (which is perhaps another issue…). In serial, the timestep is continually cut until it reaches the minimum allowable value, and then converges, but to me the “converged” solution still looks quite non-physical.

Attached is a tar file with all the files needed to run the simulation to try and reproduce the behavior. The .yaml files are in the "sim" subdirectory. Also in the "sim" directory are the logfile from running the problem on 4 processors, and an excerpt of that logfile ("logfile-excerpt.txt") which highlights the problematic behavior.

reproduce_bluff_bug.tar.gz

[ebayats]

@lxmota

I have some updated information on this in case it is helpful.

After pulling and rebuilding Trilinos (develop) and LCM (main), up to date as of 6/23/26 PM), and adding the suggested "Static Equilibrium Initialization: true" to coupled.yaml. I can of course rebuild today and test again, but it takes a bit of time for these simulations to run since the problematic behavior doesn't appear until ~992 timesteps.

The simulation for the bluff in parallel on 4 cpus still produces the behavior where the mechanical problem solver first fails, then "converges" to a garbage solution, and then erodes the entire mesh. This happens for time step 992.

Interestingly, the issue did not reproduce from a restart -- I tried running a restart from output idx 985, and the solver proceeds to pass timestep 992 without issue and proceed without ever showing this behavior.

I am including a tar ball of the files that produced this behavior in case it is helpful.

I am also including excerpts of the logfiles from the simulations in both serial and parallel around the timesteps of interest, in case that sheds further light on the issue. The solver demonstrates this problematic behavior in parallel on 4 cpus but does not demonstrate this behavior in serial.

sim-test-bluff-restart.tar.gz

logfile-excerpt-serial.txt

logfile-excerpt-parallel.txt

[lxmota]

@ebayats This is very helpful, thank you, especially that a restart from idx 985 sails past 992. That (plus the residual being acceleration-driven from a normal velocity) shows it's accumulated in-memory state, not the configuration, which narrows the root a lot. Digging in now.