RFC: Providing libraries as Emscripten external ports

[ryanking13]

This is a kind of reviving my proposal in pyodide/pyodide#2805.

As people will (hopefully) start to build and publish their python packages to PyPI after PEP 783 adoption. I think it would be useful for us to provide a way to build shared library dependencies easily.

When I proposed pyodide/pyodide#2805, there wasn't a official way to do this. But now emscripten has a concept of "External Ports", where people can specify how to build libraries from remote path.

So I think what we can do is to

1. Create a new repository that stores all the shared libraries and the build script.
2. Add the "external port" script for each library in pyodide-build
3. Provide a new sub command in pyodide-build that create a CLI flag that builds each port

WDYT?

[agriyakhetarpal]

Thanks for linking to the documentation about external ports. I like this idea! However, one thing I would change about this proposal would be to avoid creating a new repository; that just adds churn when we have to sync repositories and submodules between one another when updating the scripts. I think it'd be nice to leave it in a ports/ folder at the top level in the pyodide-recipes repository itself, just like packages/. That way, it's easier for us to use the same pyodide-build submodule arrangement we currently have to test both packages and ports, and apply any necessary patches to ports. It looks like Emscripten can use --use-port from a path in another directory just fine.

One of the other reasons why I like this idea is that we can also do some namespacing for the ports; i.e., we can then figure out a way to add multiple ports for a library, say, if a package X needs a library Y(a) with no patches, but package Z needs a library Y(b) with custom patch(es) or a different version of it applied to make it work.

One thing I'd like to make our process easier around, at the same time as this proposal, would be how to make such a feature available for out-of-tree builds. Is it possible to do so? For example, there are packages like python-flint that need to build libraries like GNU MPFR and the like, and it'd be nice if we could do something similar to pyodide build-port mpfr. Or that could be OpenBLAS for SciPy or whatever. But maybe that should still live in their own build pipelines/repositories?

[ryanking13]

I think it'd be nice to leave it in a ports/ folder at the top level in the pyodide-recipes repository itself, just like packages/.

Sounds good to me. One minor concern is we should have all those libraries in a single repository as a submodule (or as a source) which will make the repository a lot bigger, but we are downloading and building those libraries anyways so it might not be a big concern.

One thing I'd like to make our process easier around, at the same time as this proposal, would be how to make such a feature available for out-of-tree builds.

We can discuss more about that (providing a pyodide CLI for buildings ports). I don't want to maintain the libraries in a long term, and do not want to guarantee that it will be keep working as it is not our responsibility.

[agriyakhetarpal]

I think it'd be nice to leave it in a ports/ folder at the top level in the pyodide-recipes repository itself, just like packages/.

Sounds good to me. One minor concern is we should have all those libraries in a single repository as a submodule (or as a source) which will make the repository a lot bigger, but we are downloading and building those libraries anyways so it might not be a big concern.

Submodules make sense to me. Yes, having lots of submodules isn't too bad I think – we can always recommend doing git clone --recurse-submodules --shallow-submodules <repository_url>. Even the .gitmodules file has a shallow key, so it can't be that worse off than the status quo, I hope 😅

One thing I'd like to make our process easier around, at the same time as this proposal, would be how to make such a feature available for out-of-tree builds.

We can discuss more about that (providing a pyodide CLI for buildings ports). I don't want to maintain the libraries in a long term, and do not want to guarantee that it will be keep working as it is not our responsibility.

I guess so, but my concern is that you mentioned the following:

As people will (hopefully) start to build and publish their python packages to PyPI after PEP 783 adoption. I think it would be useful for us to provide a way to build shared library dependencies easily.

This would all happen in out-of-tree builds anyway, and not through pyodide-recipes? So our idea could be that we do pyodide build-ports <library> and it'd build the port by fetching the recipe from the pyodide-recipes repository's packages/ folder, like a library zoo of sorts? Somewhat similar to Meson's WrapDB: https://github.com/mesonbuild/wrapdb.

i.e., what I don't get from this RFC is how we can better assist downstream package builders in this scenario. I might be missing the overall plan of action here (sorry), so it'd be great if you could elaborate and clarify if I'm misunderstanding something!

[ryanking13]

i.e., what I don't get from this RFC is how we can better assist downstream package builders in this scenario. I might be missing the overall plan of action here (sorry), so it'd be great if you could elaborate and clarify if I'm misunderstanding something!

No worries, this is just an RFC so I don't have a concrete plan in my head either. My rough idea was to 1) snapshot all the libraries in pyodide-recipes at this moment, 2) provide them as an external port so that we can facilitate out-of-tree builds of recipes that we have in this repository.

So roughly what I thought was it would look like

# in downstream repository
wget https://github.com/pyodide-recipes/port-script-for-libxxx.py -o port-script-for-libxxx.py
export CFLAGS="--use-port=/port-script-for-libxxx.py"
pyodide build

But basically we don't keep this port-script updated / do not actively add new script. If a new package maintainer needs a library for their package, it is their responsibility not us.

However, now that I think about it, this might not be such a good idea. It might be a better approach to just upstream the library along with the package from the beginning.