Extended standalone driver test#1303
Conversation
Make cscs ci test jobs correspond more closely to nox sessions.
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;common;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=unit:integration:validation;MODEL_MPI_SUBSETS=datatest |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver:common;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=unit:integration:validation;MODEL_MPI_SUBSETS=datatest |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=integration:validation;MODEL_MPI_SUBSETS=datatest |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=integration:validation;MODEL_MPI_SUBSETS=datatest |
This PR unifies the default, extra, distributed, dace pipelines into one pipeline that can be parameterized to run different subsets of tests. The test jobs are generated by a new script `generate_ci_pipeline.py` because we can't use variables to dynamically filter gitlab matrix jobs. The test jobs are triggered as a gitlab child pipeline (https://docs.gitlab.com/ci/pipelines/downstream_pipelines/#parent-child-pipelines). There are now three main CI pipelines (in addition to the benchmark pipelines which are mostly unchanged, except to adapt to the changes in `base.yml`): - `default` still exists, and should be the "default" pipeline to run when you want to check if the PR is roughly in shape. This runs serial and MPI tests with one GPU backend. Bikeshedding on what belongs here welcome, but in my opinion this should be relatively small. `default` can and should be customized to run a subset of tests that are relevant for your changes (see below). A future extension would be to automatically detect which tests actually need to run based on the changes in the PR (see e.g. #1291). This will not be implemented in this PR. - `merge` is new, and will be made gating for PR merges. `default` will not be required for merges since it's a subset of `merge`. `merge` is roughly like the current `default` pipeline in that it runs most tests with all backends. This can be triggered when a PR has been approved and auto-merge has been enabled. A future extension is to make this pipeline run using merge queues (https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue). - `all` is also new, and runs _all_ tests, including future "extended" or "validation" tests (#1303). This is meant to be run e.g. once a week on `main`. I've currently set up a run that runs daily at 2 AM on this branch (first run here: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/5125340235196978/2255149825504677/-/pipelines/2624464232). I'll update this to `main` and weekly when this is merged. As mentioned above, the `default` pipeline is intended to be customized with CI variables, e.g. > cscs-ci run default;SESSIONS=model;MODEL_SUBPACKAGES=dycore All variables can be overwritten. Otherwise they take the default values from the `default` pipeline definition. The test reminder has been updated with some hints and reminders about which pipelines to run and how to customize them. The `dace`, `extra`, and `distributed` pipelines have been removed since they're covered by the other pipelines now. Slightly out of scope, I've changed some of the scripts in `scripts/` to use `--only-group scripts` to not pull in unnecessary dependencies. The new `generate_ci_pipeline.py` script is run directly with `python scripts/python/generate_ci_pipeline.py` instead of through `scripts/run` because the latter pulls in many unnecessary dependencies. By running it directly with `--only-group scripts`, `generate_ci_pipeline.py` can be run in a very slim container which is much faster to pull. Since the `all` pipeline now ran tests that were previously not run in CI, it exposed some numpy/cupy mismatches in the RBF tests. I've fixed those at the same time. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| stage: test | ||
| timeout: 8h | ||
| variables: | ||
| ICON4PY_DALLCLOSE_PRINT_INSTEAD_OF_FAIL: true |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=integration:validation;MODEL_MPI_SUBSETS=datatest |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=integration:validation;MODEL_MPI_SUBSETS=datatest |
|
Mandatory Tests Before merging, run the When developing, you can test your changes on CSCS CI before merge with the You can pass options to override pipeline variables, for example:
Available options are:
See The Optional Tests To run benchmarks you can use:
For more detailed information please look at CI in the EXCLAIM universe. |
|
cscs-ci run default;SESSIONS=model_mpi;MODEL_MPI_SUBPACKAGES=standalone_driver;BACKENDS=gtfn_cpu:dace_cpu:gtfn_gpu;LEVELS=integration:validation;MODEL_MPI_SUBSETS=datatest |
1 participant
Nothing to see here. Still depends on #1295 (for parameterizable standalone driver tests) and #1294 (for easier CI parameterization), but opening to experiment with compilation options for bitwise reproducible results between single- and multi-rank runs.
Adds a new testing level (
extended) and one test that uses it (a seven-day standalone driver test, we can bikeshed about how "extended" this should be).