What happened?
fckit_install_venv.cmake pip-installs fckit_yaml_reader (and its deps ruamel.yaml==0.18.6, ruamel.yaml.clib, fypp) into the per-build venv with OUTPUT_QUIET and no return-code check:
https://github.com/ecmwf/fckit/blob/develop/cmake/fckit_install_venv.cmake
execute_process( COMMAND ${Python3_EXECUTABLE} -m pip
install ${PIP_OPTIONS} ${CMAKE_CURRENT_SOURCE_DIR}/src/fckit/${_pkg_name} OUTPUT_QUIET )If this install fails transiently (e.g. many CI jobs hitting PyPI concurrently for ruamel.yaml/ruamel.yaml.clib/fypp), the failure is silently swallowed. The configure/build proceeds and only dies much later during downstream Fypp preprocessing with a misleading error:
error: No module named 'fckit_yaml_reader' [ModuleNotFoundError]
error: failed to import module 'field_config' [FyppFatalError]
This is hard to diagnose because the real cause (a failed pip install) is invisible, and the reported error points at Fypp/consumer code instead.
Observed in ECMWF IFS (CY49R2) CI: the failure is intermittent across matrix jobs on identical commits — some jobs build fine, others fail at the Fypp step. A clean local rebuild of the same commit succeeds. Manually running the exact pip install into the venv also succeeds. This points to a transient install failure being masked, rather than a missing/incompatible dependency.
Related but distinct from #63 (which addressed dependency resolution in 0.13.3). Here the install mechanism is correct; the problem is that failures are silent and non-deterministic.
What should happen instead?
The venv bootstrap should fail loudly and be resilient to transient errors:
- Check the result of the pip install (
RESULT_VARIABLE) and emit a clear FATAL_ERROR on non-zero, so the true cause surfaces immediately instead of a downstream Fypp ModuleNotFoundError. Drop or soften OUTPUT_QUIET at least on failure so pip's error is visible.
- Retry the pip install a bounded number of times (or use
pip --retries N) to absorb transient PyPI/network hiccups.
- Optionally, document/support fully-offline installs via
ARTIFACTS_DIR (pre-staged wheels, --no-index --find-links) for deterministic CI.
Version
0.14.2 (also relevant to 0.14.x / develop; the OUTPUT_QUIET install pattern is unchanged)
Platform (OS and architecture)
ECMWF ATOS (hpc2020), Linux x86_64; Intel and GNU toolchains
What happened?
fckit_install_venv.cmakepip-installsfckit_yaml_reader(and its depsruamel.yaml==0.18.6,ruamel.yaml.clib,fypp) into the per-build venv withOUTPUT_QUIETand no return-code check:https://github.com/ecmwf/fckit/blob/develop/cmake/fckit_install_venv.cmake
If this install fails transiently (e.g. many CI jobs hitting PyPI concurrently for
ruamel.yaml/ruamel.yaml.clib/fypp), the failure is silently swallowed. The configure/build proceeds and only dies much later during downstream Fypp preprocessing with a misleading error:This is hard to diagnose because the real cause (a failed pip install) is invisible, and the reported error points at Fypp/consumer code instead.
Observed in ECMWF IFS (CY49R2) CI: the failure is intermittent across matrix jobs on identical commits — some jobs build fine, others fail at the Fypp step. A clean local rebuild of the same commit succeeds. Manually running the exact pip install into the venv also succeeds. This points to a transient install failure being masked, rather than a missing/incompatible dependency.
Related but distinct from #63 (which addressed dependency resolution in 0.13.3). Here the install mechanism is correct; the problem is that failures are silent and non-deterministic.
What should happen instead?
The venv bootstrap should fail loudly and be resilient to transient errors:
RESULT_VARIABLE) and emit a clearFATAL_ERRORon non-zero, so the true cause surfaces immediately instead of a downstream FyppModuleNotFoundError. Drop or softenOUTPUT_QUIETat least on failure so pip's error is visible.pip --retries N) to absorb transient PyPI/network hiccups.ARTIFACTS_DIR(pre-staged wheels,--no-index --find-links) for deterministic CI.Version
0.14.2 (also relevant to 0.14.x / develop; the
OUTPUT_QUIETinstall pattern is unchanged)Platform (OS and architecture)
ECMWF ATOS (hpc2020), Linux x86_64; Intel and GNU toolchains