pip fails with `No module named 'encodings'` in pyodide venv created from uv python

[oscarbenjamin]

This is coming from https://discuss.python.org/t/pep-776-emscripten-support/84996/20

I see an error when following the instructions from the docs to create venv and use pip:
https://pyodide.org/en/stable/development/building-and-testing-packages.html

It is possible that this is due to using a uv installed Python.

Specifically I run these commands that seem to complete successfully until the last:

uv python install 3.12
uv venv -p 3.12 .venv
source .venv/bin/activate
uv pip install pip
pip install pyodide-build
pyodide venv .venv_pyodide
source .venv_pyodide/bin/activate
pip # error

The error I see is (I have redacted two paths):

$ pip
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '.../.venv_pyodide/bin/python3.12-host'
  isolated = 0
  environment = 1
  user site = 0
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/home/oscar/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '.../.venv_pyodide/bin/python3.12-host'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007b7352aeb740 (most recent call first):
  <no Python frame>

I have not tested this yet with a non-uv Python.

[agriyakhetarpal]

Hi @oscarbenjamin, thanks for the report! Yes, uv-managed Python environments don't work yet; see #58 for a related issue.

There are two things at play here:

• the virtual environment set up by uv is special in the way that uv adds barriers to the use of pip instead of uv pip in the environment;
  • uv pip install pip will install pip, but pip will still point to other pips on one's system instead of the one that gets installed – the pip CLI entry point isn't added to the virtual environment.
  • Thus, the only way to access the pip installed by uv pip is to use the python -m pip invocation.
• the pyodide venv command patches pip to not require SSL for installing packages and configures it to look for Emscripten wheels in Pyodide's jsDelivr CDN/index, and it relies on virtualenv's API to set up a virtual environment – which isn't compatible with uv venv out of the box.

Given that uv does not plan to include a programmatic means of usage through a Python API and third-party Python bindings are not present (that I know of), I'm not sure if pyodide venv can use uv for the environment creation/management. The fact that pip is pure Python helps with the fact that it can be monkeypatched, but uv is a binary package – it would need upstream support to handle Pyodide-specific behaviour.

[agriyakhetarpal]

In the meantime, @ryanking13 has been progressing on uv support across other areas for pyodide-build in #132. In #117, I exposed some more arguments to pyodide venv from virtualenv's supported arguments, and also documented that pyodide venv uses virtualenv under the hood with the pyodide venv --help command. Here's an excerpt, adding for ease of access:

pyodide venv --help
                                                                                                                                                                      
Usage: pyodide venv [OPTIONS] DEST                                                                                                                                     
                                                                                                                                                                      
Create a Pyodide virtual environment. Additionally, this interface supports a subset of the
arguments that virtualenv supports, with some minor differences for Pyodide compatibility.
Please note that passing extra options is experimental and may be subject to change.                                                            
Registered by: pyodide-build                              

It hasn't been released yet, however. I think we should cut a release soon.

[hoodmane]

I think this problem is not actually directly related to pip. The problem is related to the symlink from .venv_pyodide/bin/python3.12-host to .venv/bin/python3. The uv Python seems to not handle this the way that normal Python does.

[hoodmane]

Compare:

python3.12 -m venv venv-normal
source ./venv-normal/bin/activate
pip install pyodide-build
pyodide venv venv-normal-pyodide
./venv-normal-pyodide/bin/python-host -c 'import sys; print(sys.base_prefix)'

to:

uv venv -p 3.12 venv-uv
source venv-uv/bin/activate
uv pip install pip

pip install pyodide-build
pyodide venv venv-uv-pyodide
venv-uv-pyodide/bin/python-host -c 'import sys; print(sys.base_prefix)'

[hoodmane]

If we set PYTHONHOME, it works:

PYTHONHOME=/usr/ ./venv-uv-pyodide/bin/pip

or:

PYTHONHOME=/home/rchatham/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu ./venv-uv-pyodide/bin/pip

[hoodmane]

I think the easiest fix is instead of symlinking python-host to sys.executable we can make it a script with:

#!/bin/bash
exec {sys.executable} $@

[agriyakhetarpal]

I'm reopening this as we had to revert the solution added in #144 – it didn't fully work, unfortunately.