Add Reactant correctness check for AtmosphereModel.#687
Conversation
There was a problem hiding this comment.
Shouldn't run only on the GPU jobs? Also, it may be easier to move all these tests in a dedicate directory (reactant/?) and then in
Lines 13 to 20 in d978c46
reactant/. Edit: I went on and did that.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ParallelTestRunner fixes the seed anyway.
|
I believe that the errors here https://github.com/NumericalEarth/Breeze.jl/actions/runs/25581771362/job/75102082127?pr=687#step:11:536 are due primarily to EnzymeAD/Reactant.jl#2870. Basically, when we compile with Reactant, we alias the initial state rather than storing it as a copy, causing the RK3 substeps to be incorrect. This enters here Breeze.jl/src/TimeSteppers/ssp_runge_kutta_3.jl Lines 150 to 156 in 10c0d0b My workaround was instead to use a kernel to copy each of the prognostic fields, which I can implement now for the time being if that sounds good There may also be an issue with |
|
I'm temporarily checking out EnzymeAD/Reactant.jl#2984 by @Pangoraw to see how it goes here. |
|
There are still correctness errors: https://github.com/NumericalEarth/Breeze.jl/actions/runs/27677328615/job/81855741446?pr=687#step:14:1076 |
|
I think there is an issue with |
should we call first_time_step! in the correctness test so that initialization is the same for Reactant and vanilla models? |
|
Where are you looking at? |
|
Reactant tests are timing out again, the way they did before #782 |
|
Given that also #774 is stuck in Reactant tests, I take something is up in post v0.2.266? |
| # concurrent jobs, some of them reaching a large memory usage. We reduce the | ||
| # number of jobs to avoid memory issues. | ||
| TEST_ARGS+=(--jobs=$(($(nproc) - 1))) | ||
| TEST_ARGS+=(--quickfail) |
There was a problem hiding this comment.
I've been thinking for a while of setting --quickfail for all jobs, maybe I'll open a separate PR later to just do that
There was a problem hiding this comment.
I suspect that we could learn more from the failure with it since the timeout seems to prevent the buffered results from being shown.
There was a problem hiding this comment.
In this case were jobs even time out that's really necessary, but in general it's a bit frustrating having to wait long time to find what's failing, which was my motivation for doing this.
I opened #798, elaborating the downside and its mitigation.
There was a problem hiding this comment.
Ok with quickfail, we actually catch the error (not a timeout)
2026-06-23T21:53:36.4406195Z UNKNOWN: <unknown>:0: error: 'math.ctlz' op unsupported op for export to XLA
2026-06-23T21:53:36.4406485Z <unknown>:0: note: see current operation: %1013 = "math.ctlz"(%1003) : (tensor<8x8x8xi64>) -> tensor<8x8x8xi64>
which should be solved once EnzymeAD/Enzyme-JAX#2578 hits a new Reactant_jll
There was a problem hiding this comment.
I'm more concerned as to who emits ctlz. I'm assuming its not coming from you guys, which implies it may be the defn of some libdevice implementation for sinpi or something. Obviously we shouldn't import that version but sinpi [or whatever is the actual cause] directly
There was a problem hiding this comment.
Pinpointed it to fma_emulated used in ConvertingDivision{Float32}. It itself calls _exponent_finite_nonzero . Maybe we it could makes sense to use NativeDivision for Reactant since it does not have fma?
|
This looks good to me? I think we should also add a timestepping loop but that can be a subsequent PR imo, anticipating other correctness issues |
Co-authored-by: Mosè Giordano <765740+giordano@users.noreply.github.com>
Yes! But we need to wait for EnzymeAD/Reactant.jl#2984 making it into a proper release |
Closes #683. Checking field equality before and after a single timestep.