Compile a Fortran-free WASM SciPy, with BLIS and semicolon-lapack

[agriyakhetarpal]

Description

See #604 for all the context. This is the last stage of work towards a rendition of SciPy that builds without a Fortran-to-WASM compiler, weird f2c hacks, Fortran calling conventions, function signature mismatches, and general runtime issues that have troubled us over the years. With the changes in this PR and pyodide/pyodide-build#395 in place (used in the submodule in this PR), the stack is entirely based in C, with no Fortran code. OpenBLAS is no longer used with SciPy, as was once integrated with pyodide/pyodide#3331. The stack is now based on https://github.com/flame/blis (see agriyakhetarpal/blis#1) and https://github.com/ilayn/semicolon-lapack.

It is important to note that this is not ready for merging yet, as scipy.odr is still present in SciPy 1.18, and we should keep it. We should wait until SciPy 2.0 later this year to do so, as that subpackage will have been removed in that update (it has already been removed in SciPy main). However, what we have achieved through this effort now paves the way forward for an in-tree CI job and subsequent first-class support for SciPy in Pyodide/WebAssembly.

Note that we still need one patch for SciPy (but we are down from five patches to just one!) and a few for BLIS. I upstreamed the patches semicolon-lapack needed in ilayn/semicolon-lapack#23 and ilayn/semicolon-lapack#24). A 100% of the SciPy test suite is passing in WebAssembly as far as can be made applicable (see scipy-conftest.py). There is one skipped test, but that's more of a test ordering problem, and the test definitely passes in isolation.

This PR follows on from the last updates to the SciPy recipe that I made in #586 and #611.

cc: the listed recipe maintainers @steppi and @lesteve, and also @ilayn and @rgommers. Thank you!

Type of change

• Adding a new package
• Updating an existing package
• Bug fix
• Other (please describe)

---

Note

Considering adding a new package?

With the acceptance of PEP 783, package maintainers can now
build and publish Pyodide-compatible wheels (pyemscripten platform) directly to PyPI from their
own repositories. This is the preferred approach going forward.

Instead of adding a recipe here, we encourage you to:

1. Contact the package maintainers and ask them to build Emscripten/Pyodide wheels in their CI
2. Refer them to the Pyodide documentation on building packages
   and cibuildwheel's Pyodide support

We still accept new recipes, but only when it is not possible to build the package upstream.

[github-actions]

Package Build Results

Total packages built: 29
Total build time: 0:20:36

Package Build Times (click to expand)

Package	Build Time
scipy	8m 52s
libopenblas	7m 44s
numpy	5m 30s
libboost	4m 44s
libblis	3m 34s
libsemilapack	2m 38s
libf2c	34s
cffi	10s
regex	9s
MarkupSafe	4s
pydoc_data	4s
atomicwrites	3s
pycparser	2s
pyparsing	1s
pytest	1s
attrs	1s
setuptools	1s
Jinja2	1s
pytz	1s
more-itertools	1s
micropip	1s
tblib	1s
six	1s
Pygments	1s
exceptiongroup	1s
pytest-asyncio	1s
pluggy	1s
py	0s
iniconfig	0s

Longest build: scipy (8m 52s)
Packages built in more than 10 minutes: 0

[agriyakhetarpal]

The entirety of the test suite is passing in https://github.com/pyodide/pyodide-recipes/actions/runs/28972682213/job/85976430414?pr=619 🎉

(Looks like adding the "do not merge" label triggered the test suite again, oh well, but never mind :P)

My next steps here will be to:

• Upstream the SciPy getrf/getri function signature mismatch (I'm preparing a PR)
• Start figuring out what to do about the gees_ calls patch 0003 – @ilayn, I think you know much better than I do about it!
• Wait for a new semicolon-lapack tag and bring in Reinstate qmax argument to dlasq3 and slasq3 for parity with reference LAPACK and cython_lapack ilayn/semicolon-lapack#24.

The BLIS patches are also somewhat upstreamable. I think it would be nice if we were to host a GitHub release somewhere with a precompiled, patched WASM BLIS build and a semicolon-lapack build that SciPy can use for a CI job. Or, it's useful to have these as submodules or similar under the SciPy org, as the compilation time isn't a lot for either of them and that helps avoid the churn of having to create a new release for every code change, as code can be patched pretty easily. The only concern with a SciPy WASM CI job, IMO, would be how fragile it would be, but that's (i) something we can only tell when we try it, and (ii) we cannot escape that reality (though we can easily disable the job in the Actions tab or xfail tests temporarily if there are issues).

Overall, I think ilayn/semicolon-lapack#22 helped quite a lot here. I haven't really bisected things as such, but I believe I got the numerical issues I was getting fixed for free from that PR, in addition to the test suite crash at 57%. Previously, there were thousands of tests failing due to irreconcilable and absurd values (such as 3.5324234 instead of 2.108471 and so on), but there are none now!

[ilayn]

Super cool that you pulled it off @agriyakhetarpal I just made a dummy release again just to keep things rolling.

For the build process, I think if you skip building the tests (-Dtests=false should do it), it is bearable times to build it from scratch. Not sure about the download times though. So indeed having a prebuilt binary somewhere might be a good idea.

I'll check the patch no. 3.

[rgommers]

libblis + libsemilapack in your build time table is ~6 minutes together, if that's on regular GitHub runners then that's acceptable for a SciPy CI job I'd say.

[agriyakhetarpal]

Yup, that's also when all packages are being built on an oversubscribed CPU, too. So building them with parallel jobs sequentially should definitely reduce that time slightly down from 6 minutes.

[ryanking13]

I guess we will need to think about when this should be merged (when the f2c, scipy-related codes can be dropped from pyodide-build), but amazing work!

[ryanking13]

Are these patches upstreamed?

[agriyakhetarpal]

Nope, not yet, but they live in https://github.com/pyodide/blis. I plan to do this soon-ish and send PRs to BLIS, but they span quite a lot of files, so I'll take it slowly :) I'm waiting for a response from the BLIS maintainers on flame/blis#491 first.

[ryanking13]

yeeees

[ryanking13]

why do we need host meson? doesn't scipy install it in the isolated environment?

[agriyakhetarpal]

This is needed for semicolon-lapack, actually, which uses Meson as its build system. It's just that we haven't had any system-level library recipes so far that were using Meson, and this is the first.

If it were a Python package, then yes, it would have been picked up as a build-time dependency.

[ryanking13]

I see. Thanks for the clarafication!