SIMD/SLEEF optimizations cause crashes for old hardware

[bhaller]

@andrewkern we weren't expecting this to come up, but here it is. :-> From https://groups.google.com/g/slim-discuss/c/S4DF_5vh-KE/m/X-NJPf7CAAAJ:

I have been running sims on a distributed cluster and have noticed crashes on jobs that were specifically sent to clusters with older hardware. Of those, most exit with some kind of opaque SIGILL error. I didn't notice this with v5.0 (haven't tried v5.1). My hypothesis is that the updated vector operations are not compatible with x86_64-v2 hardware. Those 2012 cpus are pretty old now, but there are still plenty of HPCs that still have them in use! I was just wondering if this is a known thing on your end or if there is something I could fix on my end. Either way I was hoping I could get some clarity on hardware requirements from you if possible.

For now, users on hardware this old can build SLiM themselves with -D USE_SIMD=OFF in their cmake build command, IIRC. But given that this has actually bitten a user, I guess it'd be good to get a solution to this that doesn't require the user to set build flags manually. How difficult is that to detect at compile time? There must be a way to do this within CMake...

[SolomonAPS]

@andrewkern Hey! I ran a quick survey of all the cpu's I connected to that are on x86_64-v2 hardware.

nosimd 0849b57259da Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz avx2=n OK
nosimd 566166e6b99d Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz avx2=n OK
nosimd a280efab74af Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz avx2=n OK
nosimd a5b766b05a84 Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz avx2=n OK
nosimd cb60457b4be9 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz avx2=n OK
portable 52172e8bcfd2 Intel(R) Xeon(R) CPU X5667 @ 3.07GHz avx2=n SIGILL
portable 924a6faf4da4 Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz avx2=n SIGILL
portable a280efab74af Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz avx2=n SIGILL
portable b4ae131cc887 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz avx2=n SIGILL
portable c004.pile.uchicago.edu Intel(R) Xeon(R) CPU X5660 @ 2.80GHz avx2=n SIGILL
portable c014.pile.uchicago.edu Intel(R) Xeon(R) CPU X5660 @ 2.80GHz avx2=n SIGILL
portable c019.pile.uchicago.edu Intel(R) Xeon(R) CPU X5660 @ 2.80GHz avx2=n SIGILL
portable c024.pile.uchicago.edu Intel(R) Xeon(R) CPU X5660 @ 2.80GHz avx2=n SIGILL
portable d2c0806af7a1 Intel(R) Xeon(R) CPU E5520 @ 2.27GHz avx2=n SIGILL

Here, 'nosimd' is SLiM5.2 built with -D USE_SIMD=OFF as @bhaller suggested (thanks Ben!) and 'portable' is just the stock build. The second column is not intelligible but those what clusters I was connected to. The final column is whether they ran or exited with a SIGILL code where all the nosimd runs succeeded and all the portable runs failed.

I think have a very niche use case where I am using a distributed cluster and can take advantage of these old cpus that generally are not in use. I will say I am seeing a lot of major universities do have these cpus still in service although it is not clear how many are out there or if I am just churning through a lot of the same. For my use case I am compiling within a container and shipping that out to different clusters so auto detecting the cpu type at the time of compilation wouldn't help and I imagine there are essentially no users that are running slim on an all x86_64-v2 cluster. So I am not sure this is something that needs 'fixing' but it would be helpful for people to know that SLiM can run on older hardware and that you can take advantage of such things (2x slower per job doesn't matter if you have access to thousands more cores). However, from what have gathered it is possible to check if the cpu supports simd operations, see https://gcc.gnu.org/onlinedocs/gcc/x86-Built-in-Functions.html It looks like you can check with __builtin_cpu_supports("avx2") and __builtin_cpu_supports("fma") although I am not super familiar with this stuff.

[andrewkern]

Hi @SolomonAPS -

Thanks for this.

The problem is in our CMake setup: the SIMD detection used check_cxx_compiler_flag("-mavx2" ...), which only asks "does the compiler accept this flag?" — not "does the target CPU support AVX2?" Any modern gcc/clang accepts -mavx2, so the "portable" build was unconditionally adding -mavx2 -mfma to the global compile flags. That's why every one of the
pre-Haswell CPUs in your list (X5570, X5660, X5667, E5520, E5-2665, E5-2640 v2) SIGILL'd: none of them have AVX2.

There are two options for a PR to fix this @bhaller--

The smaller option would be to turns USE_SIMD into a level rather than a boolean:

• USE_SIMD=OFF — no -m flags, runs on any x86_64
• USE_SIMD=SSE42 — -msse4.2, runs on x86_64-v2 (Nehalem/K10 and later, ~2009+) — should cover every CPU in your survey
• USE_SIMD=AVX2 (default) — current behavior, requires Haswell+ / Excavator+
• USE_SIMD=NATIVE — -march=native for local builds

The bigger option would be to us __builtin_cpu_supports for runtime dispatch. Which way would you like to go @bhaller -- let me know and I can work on a PR

[bhaller]

Hi @SolomonAPS -

Thanks for this.

The problem is in our CMake setup: the SIMD detection used check_cxx_compiler_flag("-mavx2" ...), which only asks "does the compiler accept this flag?" — not "does the target CPU support AVX2?" Any modern gcc/clang accepts -mavx2, so the "portable" build was unconditionally adding -mavx2 -mfma to the global compile flags. That's why every one of the pre-Haswell CPUs in your list (X5570, X5660, X5667, E5520, E5-2665, E5-2640 v2) SIGILL'd: none of them have AVX2.

There are two options for a PR to fix this @bhaller--

The smaller option would be to turns USE_SIMD into a level rather than a boolean:

• USE_SIMD=OFF — no -m flags, runs on any x86_64
• USE_SIMD=SSE42 — -msse4.2, runs on x86_64-v2 (Nehalem/K10 and later, ~2009+) — should cover every CPU in your survey
• USE_SIMD=AVX2 (default) — current behavior, requires Haswell+ / Excavator+
• USE_SIMD=NATIVE — -march=native for local builds

The bigger option would be to us __builtin_cpu_supports for runtime dispatch. Which way would you like to go @bhaller -- let me know and I can work on a PR

Hi @andrewkern! Hmm, so with alternative #1, the USE_SIMD solution, the default would be USE_SIMD=NATIVE, which would build using whatever SIMD support is present on the machine being built upon. If the user is doing cross-compilation – compiling a binary on machine A that will be run on machine B – then they could change that setting to OFF, SSE42, or AVX2 to choose the SIMD instruction set that they want to target. Am I understanding correctly? This seems like the right design.

Alternative #2 would, as I understand it, produce a binary that would detect, at runtime, the hardware architecture it is being run on, and dispatch to different versions depending upon that; so the binary would contain code for all four alternatives. That seems like overkill, and it would also introduce a (presumably small) runtime overhead each time it needed to decide between implementations. I don't think we want to do that; I don't see much benefit. I guess it would let someone who is managing a heterogeneous cluster, where some machines have SSE42 and some have AVX2, build the binary once and copy it over to all machines, and all machines would use the optimal SIMD version for themselves automatically. But I'm not finding that to be a compelling use case, I think; they can just build locally on each machine if that's what they want, or build to the lowest common denominator.

I dunno. I guess this also affects conda and such, doesn't it? I think (I'm not sure) conda, and probably some other package managers, build a reference binary for a given platform once, and then just install that same binary for everyone. In that case, alternative #1 would likely produce an AVX2 build (because the machine the build happens on is probably modern), and then the installed binary would fail on old machines. Alternative #2 would produce a binary that would work on SSE42 machines even if the build machine is AVX2, which is nice. Hmm. Well, what do you think @andrewkern? Am I thinking about this right? Initially we were figuring that nobody would be using non-AVX2 machines any more so it didn't matter; but apparently that is not the case, so if we're really going to fix this issue properly , maybe alternative #2 is actually the way to go, otherwise we'll have people installing with conda and other package managers on old systems and then complaining that it crashes. So, maybe alternative #2. Thoughts? (I wish I had a sense of how common these non-AVX2 machines actually are in our user base!)

[andrewkern]

Hi @bhaller — yeah you are thinking about this right. Conda is the case that breaks #1 -- I hadn't thought about this. conda-forge runs builds on modern hardware, so whether the default is NATIVE or AVX2, the binary they ship to everyone is going to require AVX2. Anyone on old hardware who conda installs slim might get a crash

I think that #2 is the way to go:

• Dispatch overhead is essentially zero. The way you'd implement this, i think, is at program startup, read __builtin_cpu_supports("avx2") once, populate a small table of function pointers (or set
  up an ifunc resolver), and from then on every call is just an indirect call and these are batch vector opps, so one indirect call per batch won't add much in time.

• It's not a huge refactor. eidos_simd.h is header-only inline functions guarded on EIDOS_HAS_AVX2 etc. To do runtime dispatch cleanly, we move the SIMD bodies out of the header into two source files — eidos_simd_baseline.cpp (compiled at baseline x86-64, scalar/SSE2) and eidos_simd_avx2.cpp (compiled with -mavx2 -mfma). A thin eidos_simd.cpp does the startup detection and exposes function pointers. The rest of the codebase calls through those pointers.

I think we could ship -msse4.2 as the baseline (covers x86_64-v2, i.e., every CPU in Solomon's table) + runtime AVX2 dispatch for the hot kernels. That way even users who somehow end up not hitting the AVX2 path still get SSE4.2 vectorization, and the binary is portable to ~15-year-old CPUs.

Happy to take a swing at the PR. Want me to do a quick benchmark first to confirm the autovec loss isn't significant before committing to the refactor?

[bhaller]

Hi @andrewkern. What you propose above sounds great. No need for a benchmark, I'm sure that the approach you describe will have low enough overhead that it won't matter. One question, though. It sounds like you're proposing that SSE4.2 would be assumed to be present if AVX2 is not present. Can we just make it fully backward compatible by being able to fall back all the way to "none" if even SSE4.2 is not present? It sounds like SSE4.2 gets us back to 15 years ago; but we already got bitten once by assuming that nobody is using old CPUs, so maybe we should just shift off of that assumption altogether, if we can. Is it easy to detect SSE4.2 at runtime as well, with another builtin, and fall back to "none" if it isn't? If that's easy, then let's do that; if it isn't, then let's do what you proposed. Full steam ahead. Thanks! :->