Progress on a SciPy that is free from a Fortran-to-WASM compiler dependency and upstreaming WASM patches

[agriyakhetarpal]

At the time of writing, SciPy's main branch has dropped scipy.odr, which was already deprecated and was the last piece of Fortran code present in SciPy. SciPy v1.19.0, expected to arrive in December 2026/January 2027, will be the first version of a Fortran-free SciPy. Note that it was already possible to build SciPy v1.18.0 with scipy.odr excluded via the without_fortran Meson option, which I had tried locally and succeeded with, but didn't end up including it in #586 as we ought to match what the SciPy v1.18 PyPI wheels ship to users.

Building on top of #586 locally in a separate branch and excluding scipy.odr from the build, I have recently been at work compiling SciPy to WebAssembly and establishing a recipe that removes most of the f2c-related and SciPy-related hacks in pyodide-build, drops the use of a Fortran compiler and of libf2c across the stack:

I posted about this in this comment in the Scientific Python Discourse forum last week: https://discuss.scientific-python.org/t/ann-scipy-1-18-0-release/2394/2, and I will use this issue to post a longer comment about the current progress I have:

• While OpenBLAS enables a NO_FORTRAN build option to only build the BLAS parts and get the LAPACK parts from a different LAPACK library, I found trouble using it for our needs, because it's written in a manner that still relies on Fortran-like calling conventions, even though we don't compile any Fortran-specific parts with its C codebase. This makes upstreaming tricky because of the large number of function signatures that would not match (see below).
• As a result, I started looking for other BLAS libraries that do not rely on Fortran, and I arrived at BLIS. It looked promising to me because it is entirely written in C.
  • I was able to compile it to WASM, albeit with a few differences, as it requires CBLAS interface for crotg, zrotg, zdrot and csrot flame/blis#916, which is being added by CBLAS interface for crotg, zrotg, zdrot and csrot flame/blis#926 (which I'm using as a patch). I hit this issue when running the SciPy test suite, where I noticed null function signature mismatches where these symbols representing Level 1 plane rotation routines were not exported.
  • Also, I am currently using the generic architecture/kernel for BLIS, instead of putting in the effort to write an optimised wasm32 kernel.
• In all this, I also fixed an issue with RPATHs showing up in macOS's /private/ folders and SciPy extension modules being non-relocatable (Resolve the runtime path before patching it auditwheel-emscripten#55).
• I am using libsemicolon-lapack (see discussion for reference) for the LAPACK side of the implementation. While it is not used upstream in SciPy yet, my primary reason for its viability is somewhat validated by RFC: 64-bit LAPACK and cython_lapack scipy/scipy#21889 (comment), i.e., it was able to pass the test suites for NumPy and SciPy when used with OpenBLAS.
• I have not yet thought about the performance situation here. It is a miracle that SciPy works in WASM anyway, and while it's nice that we get the extra performance boost from OpenBLAS being compiled with WASM SIMD intrinsics via Feat/openblas wasm simd pyodide#5960, switching to a non-SIMD version isn't that big of an issue, as most of our usage is not performance-oriented in nature, and such lost performance-related gains can be brought back in future.
• Currently, SciPy compiled with BLIS and semicolon-lapack and various function signature mismatches resolved is able to run up to 57% of the test suite without encountering a crash due to function signature mismatches. I made incremental progress on this, as expected – going from not being able to import scipy after compiling successfully, to running 2% of the test suite, to 13%, to 57%. However, there are a bunch of numerical issues all across the test suite, where various scipy.linalg and scipy.optimize tests just straight up return entirely incorrect results, leading to at least a thousand test failures across ~50,000 tests. This may be fixed if we spend some extra time finding out where this is coming from in BLIS – I have a strong hunch it's not a semicolon-lapack problem but rather an issue with miscompilation of BLIS (maybe the wrong compiler flags were used, or something, and that causes bitness issues or whatever) and misinteractions with SciPy's build options.

Next steps on my mind:

• Upstream the WASM recipes for BLIS and semicolon-lapack into pyodide-recipes here, as I was able to compile both (the latter was a bit easier). Even though we don't end up using them, these seem unlikely to require much maintenance, and someone else might find their presence useful for their own projects. Also, BLIS's own test suite in WASM did pass when I tried it, if I remember correctly.
• Switch to a better-maintained version of BLIS. After having a chat with @rgommers, we settled on trying out AMD's version of BLIS: https://github.com/amd/blis. This is used throughout the AMD stack, as it provides tuned kernels for the AMD Zen (Ryzen/Threadripper) architecture CPU lineup. Compiling it to WASM remains an option, and I haven't attempted this as such yet, and thus cannot speak to how hard it would be.
• Upstream my pyodide-build patches for the f2c-fixes removals and SciPy-related removals.
• Or, retry with OpenBLAS with its NO_FORTRAN option and use semicolon-lapack to go along with it.
• Contribute to Meson to remove its outdated/incorrect note that wasm.ld does not support shared libraries – according to @hoodmane previously, it does and has been doing so for a long time, so this raises configuration errors when compiling. It's falsely flagged.
• libflame might also have been an option, but that is also kind of untested waters for SciPy's usage, so this is on the last rung of my priorities.

I would like to reiterate that the emphasis of this tracker is not only on removing all the Fortran bits from our build pipeline, but also on upstreaming what we achieve here in the form of a WASM CI job for SciPy. While pyemscripten wheels for SciPy might be a far-elusive goal for us and while SciPy does not yet have a dedicated platform support policy, it is reasonable for us to think that, with SciPy's maintainer overlap with NumPy, it would be based on NumPy's policy as defined in NEP 57 – and thus a CI job running the test suite in Pyodide is easier to add and maintain, and easy to disable temporarily if there are larger problems with its running.

As such, we need to optimise by minimising the six (actually five if we merge pyodide/pyodide-build#21) patches we currently have in the recipe to a number SciPy is comfortable with, and minimising the number of sed commands we use to fix function signatures across various BLAS-exported symbols, and various other problems. My estimate is that when these numbers drop to, say, 3-5, and are placed in parts of the SciPy codebase that are not often touched even upstream, this would become an easier task than ever before, and we would be able to convince the SciPy maintainers towards its inclusion. We have already spent quite a lot of time running the entire test suite in Pyodide here, and we have enough information about which tests cause problems and need to be skipped/xfailed/etc., and why. Similarly, CI time isn't a concern, as the test suite completes in roughly 34 minutes, and the non-slow tests would finish even faster.

[hoodmane]

expected to arrive in December 2025/January 2026

That sounds like the past?

[agriyakhetarpal]

Sorry, edited. I meant December 2026/January 2027.

[ryanking13]

As a result, I started looking for other BLAS libraries that do not rely on Fortran, and I arrived at BLIS. It looked promising to me because it is entirely written in C.

Sounds good. We have an upstream issue about it (pyodide/pyodide#227).

Also, I am currently using the generic architecture/kernel for BLIS, instead of putting in the effort to write an optimised wasm32 kernel.

We never had a wasm32 optimized kernel, so I think it is fine.

Upstream my pyodide-build patches for the f2c-fixes removals and SciPy-related removals.

Yeah, it would be great if we can remove them. But note that we might have chances to build old scipy versions for e.g. patch release, etc, so we will probably keep them for a while.

[agriyakhetarpal]

Yes, correct, the pyodide-build patches would be gated on the SciPy version grabbed from meta.yaml when building in-tree, and we won't need to bother for the out-of-tree case as that hasn't happened before.

I have to note that there are WASM kernels for a few specific operations in OpenBLAS now, just that have to update our recipe to update to newer versions for us to make use of them – and fix the SIMD compilation flags and/or builds as might be deemed necessary. OpenBLAS versions 0.3.32 and 0.3.33 have a few enhancements in this area, thanks to @teddygood's work.

[ryanking13]

OpenBLAS versions 0.3.32 and 0.3.33 have a few enhancements in this area, thanks to @teddygood's work.

Wow, I didn't know that. Awesome work @teddygood! Maybe you can work more closely with scientific python people to get more support.

[teddygood]

Wow, this is really impressive work, @agriyakhetarpal. It’s exciting to see SciPy getting this close to a Fortran-free build, and to see a possible path toward upstreaming a WASM CI job for SciPy. Also, thanks for mentioning the OpenBLAS WASM work.

@ryanking13 Thanks for the kind words! That’s an interesting idea. I’d love to learn more about how the Scientific Python community is thinking about this space.

[agriyakhetarpal]

IMO, updating from OpenBLAS 0.3.28 to OpenBLAS 0.3.33 is (for now) a separate task from the BLIS experiment, and during that time I think we can still see performance improvements. There may be some issues. While we haven't noticed any problems in the test suite, I think we should keep our OpenBLAS version the same as what is used in upstream SciPy. Is that something you would like to explore doing here, @teddygood? Since you added the kernels and have done some benchmarking for Pyodide builds in your OpenBLAS PRs, I think it would be great to do this in this repository as well.

[ilayn]

Congratulations for your hard work and your really nice comments elsewhere @agriyakhetarpal . Apologies for not responding timely, I'm having a bit of a break from FOSS stuff lately since I'm running low on free time lately and also to curb the addiction. But allow me to place a few comments about the semicolon thing;

• The reason for the CBLAS requirement in that semicolon library is precisely what you are going through; I did not only want avoid the F77 code itself but also its strange ABI issues across many architectures and strange naming and other .
• But this brings other challenges; namely CBLAS customers are rather small in numbers compared to Fortran API of BLAS. Hence receives less love as you saw in the BLIS repo. I'm waiting also MKL to have ?gemmtr to be added. It should not bother you; just letting you know.
• Unfortunately even if it is so, you would probably need a Fortran shim to handle SciPy scipy.linalg.blas and scipy.linalg.cython_blas modules as they expect the traditional API. I don't have any fight left in me to start that discussion at least for the former one.
• If you clone OpenBLAS and run make NO_LAPACK=1 NO_FORTRAN=1 DYNAMIC_ARCH=1 PREFIX=/tmp/openblas_cblas_fat NO_SHARED=1 it should give you a good CBLAS. MKL CBLAS is also good. You can use all three with semicolon (if that's what you want). I use it quite often.
• I ran a few examples already and I am hoping to write a smaller modernized CBLAS (maybe end of summer, empty repo is here https://github.com/ilayn/blastik). Especially, removing oddball architectures and adding Zen 4 and 5 makes things much faster.
• There are a lot of naked for loops and bunch of scalar operations inside LAPACK functions. Hence SIMD would also help in LAPACK too, which to this day pretty much untouched by anyone. I have some moderate but nonnegligable speedups locally. Hence expect to go faster than OpenBLAS if things are implemented properly.

If you let me know a CI job or some recipe that I can work on, I can check about that 57% crash issue.

Anyways, it's great to see progress nevertheless. So once again congratulations.

[teddygood]

While we haven't noticed any problems in the test suite, I think we should keep our OpenBLAS version the same as what is used in upstream SciPy. Is that something you would like to explore doing here, @teddygood?

I'd be interested in working on this. I checked SciPy’s OpenBLAS, and both SciPy v1.18.0 and current SciPy main use scipy-openblas32==0.3.31.22.0, which corresponds to an OpenBLAS 0.3.31.dev build. Since Pyodide’s libopenblas recipe is currently on 0.3.28, should I target the 0.3.31 line to match upstream SciPy, rather than 0.3.33, @agriyakhetarpal?