Add separate uv cache image stage to CI that only changes on uv.lock changes

[msimberg]

Currently the uv cache is populated in the same step as the checkout stage. However, while the source code changes every commit, uv cache doesn't need to be repopulated every new commit, only when uv.lock changes. This introduces an intermediate image that is rebuilt only on uv.lock changes.

[msimberg]

cscs-ci run default

[Copilot]

Pull request overview

This PR refactors the GitLab CI container build pipeline by introducing a separate “venv” image stage intended to cache the uv sync environment across commits, so the expensive dependency install only needs to be rebuilt when uv.lock (or other venv inputs) change.

Changes:

• Adds a new venv CI stage that builds and publishes an icon4py-venv image keyed off uv.lock (and selected build inputs).
• Updates the checkout/build image stage to be based on the new venv image rather than running uv sync on every commit.
• Tightens gt4py-cache.sh by requiring BASE_IMAGE to be present (but currently needs a small fix to work with set -u).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ci/scripts/gt4py-cache.sh	Adds a BASE_IMAGE validation check for consistent cache directory hashing.
ci/docker/venv.Dockerfile	New Dockerfile to build a reusable uv/venv layer intended to change only when dependency inputs change.
ci/docker/checkout.Dockerfile	Switches build to start from the new venv image instead of the base image and removing the uv sync step.
ci/base.yml	Introduces the new venv stage and wiring between baseimage → venv → checkout image builds.
ci/default.yml	Adds the build_venv_aarch64 job to the default pipeline.
ci/all.yml	Adds the build_venv_aarch64 job to the “all” pipeline.
ci/merge.yml	Adds the build_venv_aarch64 job to the merge-gating pipeline.
ci/benchmark_bencher_common.yml	Adds the build_venv_aarch64 job to the benchmark pipeline.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[msimberg]

cscs-ci run default

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[msimberg]

cscs-ci run default

[msimberg]

cscs-ci run default

[github-actions]

Mandatory Tests

Before merging, run the merge pipeline with cscs-ci run merge. Merging is blocked unless this pipeline passes.

When developing, you can test your changes on CSCS CI before merge with the default pipeline: cscs-ci run default. This will run a default subset of tests.

You can pass options to override pipeline variables, for example:

• cscs-ci run default;BACKENDS=gtfn_cpu;LEVELS=unit
• cscs-ci run default;MODEL_SUBPACKAGES=common:driver;SESSIONS=model
  Avoid running the pipeline for all tests when you are developing.

Available options are:

• SESSIONS: model, model_mpi, or tools (correspond to nox sessions)
• MODEL_SUBSETS: datatest, basic, or stencils (correspond to nox session selections)
• MODEL_SUBPACKAGES: subpackages for non-MPI tests (last component, e.g. diffusion, standalone_driver)
• MODEL_MPI_SUBPACKAGES: subpackages for MPI tests (as above)
• BACKENDS: backends
• GRIDS: grids for stencil tests (simple, icon_regional, or icon_global)
• LEVELS: testing level for non-stencil tests (any, unit, or integration)

See scripts/python/generate_ci_pipeline.py and noxfile.py for available values for each option.

The all pipeline can be run with cscs-ci run all. This will run all icon4py tests in CSCS CI which can be expensive. This pipeline runs on a schedule on main, and can be run when extensive validation is needed (e.g. before releases).

Optional Tests

To run benchmarks you can use:

• cscs-ci run benchmark-bencher

For more detailed information please look at CI in the EXCLAIM universe.

[msimberg]

cscs-ci run default

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

[msimberg]

cscs-ci run merge

[msimberg]

The CI failure is "fixed" by #1352.