Lazily installing cross-build packages is currently not reliable

[agriyakhetarpal]

With #297 and #302, we added a feature to install cross-build packages only when we require them. However, as part of debugging the implementation devised in #21 and testing it in pyodide/pyodide-recipes#591, I think I have exposed a latent bug that has existed for a while, but our infrastructure has kept it from manifesting in CI builds.

To reproduce:

1. Set up the pyodide-recipes repository
2. Set up submodule to point to Install unisolated packages into the isolated build venv and overlay their cross-build files #21 (it is what exposes the bug right now, but the changes are not related)
3. Run pyodide build-recipes RobotRaconteur in the pyodide-recipes repository, fresh, before building anything else.

Right now, ensure_cross_build_packages_installed() on main is called inside _build_in_isolated_env():

pyodide-build/pyodide_build/pypabuild.py

Lines 122 to 123 in 2946127

	if in_xbuildenv() and needs_cross_build_install:
	get_current_xbuildenv_manager().ensure_cross_build_packages_installed()

pyodide-build/pyodide_build/pypabuild.py

Lines 183 to 184 in 2946127

	# first install the build dependencies
	symlink_unisolated_packages(env, builder.build_system_requires)

Which is part of _compile(). Now, looking at _build_package at

pyodide-build/pyodide_build/recipe/builder.py

Lines 564 to 584 in 2946127

	def _build_package(self, bash_runner: BashRunnerWithSharedEnvironment) -> None:
	if self.recipe.is_rust_package():
	bash_runner.run(
	RUST_BUILD_PRELUDE,
	script_name="rust build prelude",
	cwd=self.src_extract_dir,
	)
	bash_runner.run(
	self.build_metadata.script,
	script_name="build script",
	cwd=self.src_extract_dir,
	)
	url = self.source_metadata.url
	prebuilt_wheel = url and url.endswith(".whl")
	if not prebuilt_wheel:
	self._compile(bash_runner)
	self._package_wheel(bash_runner)
	shutil.copytree(self.src_dist_dir, self.dist_dir, dirs_exist_ok=True)

The order is bash_runner.run(self.build_metadata.script, ...), which runs the CMake configuration, and then this is followed by self._compile(bash_runner), which is where we lazily load cross-build packages.

In this case, the RobotRaconteur recipe has a script section that needs host_site_packages populated before CMake runs, but host_site_packages/numpy/_core/include is still empty at execution time, so headers such as ndarrayobject.h are not found because they do not exist yet. There is no .cross-build-packages-installed marker either. Slightly before the last few commits I pushed in #21, we were not installing the cross-build packages; instead, we were only installing the cross-build files. ndarrayobject.h and other similar headers are not included in our cross-build files.

Why this has never showed up for us is because by the time the build order reaches RobotRaconteur, some earlier package that lists numpy as a host dependency (could be scipy, pandas, or any other scientific package, etc.), has already necessitated the existence of the _compile() step and warmed up the host_site_packages directory, which is then used by all the packages.

Building RobotRaconteur in isolation (just the recipe and its direct dependencies) does not trigger the lazy installation, so nothing is present in the cross-build packages.

Therefore, a fix could be to change when the lazy trigger runs, so that we detect the need to install, and install, the cross-build packages at the start of a package build rather than at a different event (if the package's recipe lists them in the host: section, of course).

[ryanking13]

Thanks for the analysis. However, I don't think this is a bug, rather it is a problem in the RobotRaconteur build setup in this case.cmake in RobotRaconteur recipe runs "before" the python build backend runs, so we can't have build dependencies installed in the isolated environments.

I would say RobotRaconteur should change how they locate numpy when running CMake (e.g. using unisolated build), instead of us changing how to install build dependencies (cc: @johnwason).

[johnwason]

Hi @ryanking13 I have this package building in my CI pipeline and it is on PyPi with PEP 783 so if it is causing a problem it can probably be removed. You can also remove the boost package too if no other packages use it.

[ryanking13]

That's awesome. Thanks for letting us know.

[agriyakhetarpal]

Thanks for letting us know @johnwason! I inspected the build pipeline in https://github.com/robotraconteur/robotraconteur/actions/runs/26373485872/job/77629652108 and tried it with #21 in agriyakhetarpal/robotraconteur#1, and it looks like it's working, and I am unable to reproduce what I saw locally. I'll keep this issue open but will revisit it if I come across it again.