Feature/add pixi by mfroelund · Pull Request #75 · ACCORD-NWP/tactus

mfroelund · 2026-05-27T15:43:43Z

Describe your changes

This PR introduced the possibility to manage the tactus project and its dependencies using pixi (https://pixi.prefix.dev/latest/).

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist before requesting a review

Testing

I have tested this on ATOS using the atos_bologna.toml configuration in the latest version of Tactus-test-runner
I have tested this on LUMI using the lumi.toml configuration in the latest version of Tactus-test-runner

For further information see the development guide

Code quality

My change follows the best practices for this project.
My local environment is correctly initialised as described in the README file.
My branch is up-to-date with the target branch - if not update your fork with the changes from the target branch (use pull with --rebase option if possible).
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation and docstrings to reflect the changes
I have added tests that prove my fix is effective or that my feature works
I have ensured that the code is still installable with poetry after the changes and runs
I have requested one or more reviewer(s) and an assignee (assignee is responsible for merging). At least one reviewer has accepted to review.

Checklist for reviewers

Each PR comes with its own improvements and flaws. The reviewer should check the following:

the code readable
the code well tested (checked coverage report)
the code documented
the code easy to maintain

Author checklist after completed review

I have added a line to the CHANGELOG describing this change (in section
reflecting type of change, for example "bug fixes", add section where
missing)

Checklist for assignees

PR is up to date with the base branch
the tests passing
author has added an entry to the changelog (and designated the change as added, changed or fixed)
the PR has been approved by all the reviewers, that accepted to review.
Once the PR ready to be merged, squash commits and merge the PR.

Tag possible reviewers

You can @-tag people to review this PR in addition to formal review requests.

trygveasp · 2026-06-04T11:48:48Z

I have run this now on Fahrenheit until the MARS stuff which needs fixing. Pgd + input was fine. Modifcations needed:

[troika]
  troika = "/nobackup/forsk/sm_tryas/pixi/tactus-4397999228747316762/envs/default/bin/troika"

[submission.types.parallel]
  tasks = ["TactusBundleBuild", "Forecast", "TactusBuild", "C903"]

[submission.task_exceptions.TactusBundleBuild.ENV]
  OMP_NUM_THREADS = 144

[submission.task_exceptions.PgdUpdate]
  binary = "gl_DP"

And I needed to update the Fahrenheit toolchains, by adding:

set(ECBUILD_IFS_Fortran_FLAGS
    "-align" "array64byte"
    "-assume" "byterecl,realloc_lhs"
    "-traceback"
    "-fpic"
    "-qopenmp"
    "-qopenmp-threadprivate" "compat"
    "-fp-model" "precise"
    "-fp-speculation=safe"
    "-qopt-report=2"
    "-qopt-report-phase=all"
    "-fast-transcendentals"
    "-ftz"
    "-finline-functions"
    "-finline-limit=1500"
    "-Winline"
    "-diag-disable=5462"   # Global name too long, shortened symbol
    "-diag-disable=7713"   # disable unused statement functions warning
    "-diag-disable=11021"  # disable unresolved libraries in ipo warning
    "-diag-disable=10397"  # disable message reporting location of opt report files
    "-diag-disable=8291"   # disable remark: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7')
    "-diag-disable=10441") # disable ifort deprecation warning

Will set the boundaries complete and test with existing boundaries

trygveasp · 2026-06-04T12:36:54Z

I was able to run this PR on Fahrenheit, but C903 still fails so needed to use pre-interpolated boundaries. Traceback:


forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
libc.so.6          000014B15D63FC30  Unknown               Unknown  Unknown
libectrans_dp.so   000014B1756E449B  gpnorm_trans_ctl_         179  gpnorm_trans_ctl_mod.F90
libectrans_lam_dp  000014B175A82A48  egpnorm_trans_dp_          97  egpnorm_trans.F90
MASTERODB          00000000071C052C  suemapf_                   92  suemapf.F90
MASTERODB          000000000058B890  sufpmapf_                  88  sufpmapf.F90
MASTERODB          0000000000585427  sufpgeometry_             271  sufpgeometry.F90
MASTERODB          0000000000545CCB  subfpos_                  312  subfpos.F90
MASTERODB          000000000044ED72  cprep3_                   197  cprep3.F90
MASTERODB          0000000000435E53  cnt0_                     198  cnt0.F90
MASTERODB          0000000000430104  MAIN__                    263  master.F90
MASTERODB          000000000042FA7D  Unknown               Unknown  Unknown
libc.so.6          000014B15D62A610  Unknown               Unknown  Unknown
libc.so.6          000014B15D62A6C0  __libc_start_main     Unknown  Unknown
MASTERODB          000000000042F995  Unknown               Unknown  Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred

mfroelund · 2026-06-04T12:59:22Z

Before I use to much time on it, I want to know if we still want to be able to install the project with poetry after merging this PR in? It should be possible, but doesn't work out-of-the-box, so it requires some more time to understand how to fix it.

jacobsnoeijer · 2026-06-04T14:43:18Z

Status on LUMI
When moving both cache and venv to $SCRATCH I managed to get tactus installed on LUMI. Getting a recent Python version seems working, getting the version number of a tool listed in the dependencies of tactus does also work but getting the version number from tactus results in a Segmentation Fault:

snoeijer@uan03:~/repos/ACCORD-NWP/tactus (pr74+75)$ pixi run python --version
Python 3.14.5
snoeijer@uan03:~/repos/ACCORD-NWP/tactus (pr74+75)$ pixi run tactus --version
tactus v1.1.0

Note: in an earlier version of this comment a segmentation fault was mentioned; re-creating the environment results in a working Python 3.14 environment;

Performance wise the setup on LUMI seems to be slow; it feels even slower as the previous Python3.10+poetry setup.

Status on LEONARDO
Setup a micromamba envirement with Python 3.12, etc. similar to the current way of installing on LEONARDO. With the python from this micromamba we can install tactus and its dependencies using pixi; running tactus with Python 3.14 works also on LEONARDO:

jsnoeije@login01:...ACCORD-NWP/tactus (pr74+75)$ pixi run python --version
Python 3.14.5
jsnoeije@login01:...ACCORD-NWP/tactus (pr74+75)$ pixi run tactus --version
tactus v1.1.0

observingClouds · 2026-06-04T16:37:10Z

@jacobsnoeijer how is the performance if you use the frozen environment, i.e. run pixi install --frozen so that pixi does not have to do the dependency resolution again. As pixi includes now also non-python dependencies and ships its own python this might take slightly longer, but not much longer.

observingClouds · 2026-06-08T09:31:35Z

Before I use to much time on it, I want to know if we still want to be able to install the project with poetry after merging this PR in? It should be possible, but doesn't work out-of-the-box, so it requires some more time to understand how to fix it.

I argue that the whole point of implementing pixi is to reduce the assumptions of the presence of non-pythonic dependencies ( e.g. module load gdal) and resolve them with pixi as well. This should allow us to remove quite a few module loads. The MPI library might be a different case for now, but certainly gdal and eccodes can come from conda-forge. I would therefore no longer support poetry and remove those module loads, particularly because the user should not even notice it much, just switching poetry for pixi in most cases.

jacobsnoeijer · 2026-06-08T13:43:27Z

pixi install --frozen

pixi install --frozen does seem to improve performance although timings are quite slow:

For example time pixi run -e py312 python --version returns

Python 3.12.13

real    0m1,931s
user    0m0,388s
sys     0m1,423s

Next, time pixi run -e py312 tactus --version yields

tactus v1.1.0

real    3m46,986s
user    0m1,506s
sys     0m2,219s

Note: this ~4 minutes loading of the environment is sometimes also observable in task executions (i.e. consumed WALLTIME per task might increase with ~4 minutes on LUMI). This causes also the crash reported in #75 (comment) when WALLTIME are set quite tight.

jacobsnoeijer · 2026-06-08T13:44:46Z

For reference: to use the gdal loaded with pixi on LUMI one needs to remove

[submission.task_exceptions.Soil.MODULES]
  GDAL = ["load", "scl-gdal_23"]

[submission.task_exceptions.Topography.MODULES]
  GDAL = ["load", "scl-gdal_23"]

from tactus/data/config_files/include/submission/lumi.toml

Crash in Pgd due to WALLTIME: increased time from 0:10:00 to 0:20:00

observingClouds · 2026-06-08T15:25:28Z

Thanks for all the performance analysis @jacobsnoeijer

The following comment strikes me

pixi install --frozen does seem to improve performance although timings are quite slow

Do I interpret it correctly that once you have resolved the environment, e.g. with pixi install --frozen and you run some tactus tasks that the performance is degraded compared to when it was install with poetry? This would not make much sense, right? I mean these are both virtual environments that are (mostly) independent from the package manager once they are resolved. So I can only imagine that the python packages and python itself are installed on a partition that is not as performant compared to when installed with poetry.

jacobsnoeijer · 2026-06-08T16:26:04Z

Thanks for all the performance analysis @jacobsnoeijer

The following comment strikes me

pixi install --frozen does seem to improve performance although timings are quite slow

Do I interpret it correctly that once you have resolved the environment, e.g. with pixi install --frozen and you run some tactus tasks that the performance is degraded compared to when it was install with poetry? This would not make much sense, right? I mean these are both virtual environments that are (mostly) independent from the package manager once they are resolved. So I can only imagine that the python packages and python itself are installed on a partition that is not as performant compared to when installed with poetry.

My assumption is that reading from disk (or at least reading a lot of files from disk) is really slow om LUMI. Also with poerty loading the python environment before the first tactus-call was not always that fast but never in the order of 4 min. Before with poetry we used the system python / gdal / eccodes so I don't know if that can make a difference. At least the virtual environment with poerty was much smaller with <30K files while the pixi generated virtual environment doesn't fit into the 100K file limit.

uandrae · 2026-06-08T19:07:32Z

On belenos I had to switch of the SSL certificat check by downlading the install.sh script and do some editing. Not very satisfactory. Would you be able to assist here @ludovicauger, @nicolasCtd, @tlestang ?

When running pixi install it stops on eccodes after a while

Error:   × Cannot install environment 'default'
  ╰─▶ Wheel: eccodeslib-2.45.4.11-cp314-cp314-manylinux_2_28_x86_64.whl doesn't match this systems virtual capabilities for tags:
      cp314-cp314-manylinux_2_17_x86_64
      cp314-cp314-manylinux2014_x86_64
...

Does anyone have any knowledge about python versions and eccodes/eccodeslib on belenos?

observingClouds · 2026-06-09T06:52:03Z

doesn't match this systems virtual capabilities for tags: cp314-cp314-manylinux_2_17_x86_64

So this means Belenos only advertises compatibility with manylinux_2_17 wheels, while this package requires manylinux_2_28. Since manylinux_2_17 corresponds to glibc 2.17, it suggests the system userspace is based on a fairly old glibc baseline from 2012 (!). 2_28 was released in 2018 so it is not an outrages requirement :D

uandrae · 2026-06-09T07:05:10Z

doesn't match this systems virtual capabilities for tags: cp314-cp314-manylinux_2_17_x86_64

So this means Belenos only advertises compatibility with manylinux_2_17 wheels, while this package requires manylinux_2_28. Since manylinux_2_17 corresponds to glibc 2.17, it suggests the system userspace is based on a fairly old glibc baseline from 2012 (!). 2_28 was released in 2018 so it is not an outrages requirement :D

Indeed

[andraeu@belenoslogin1 ~]$ ldd --version
ldd (GNU libc) 2.17

Regardless of what we think targeted platforms should look like we need to handle these cases. Especially for a platform like belenos which is considered as a platform where tactus should work. What's the level of configurability with pixi given that we have been able to install with micromamba/conda?

nicolasCtd · 2026-06-09T08:12:56Z

On belenos I had to switch of the SSL certificat check by downlading the install.sh script and do some editing. Not very satisfactory. Would you be able to assist here @ludovicauger, @nicolasCtd, @tlestang ?

You have to use the certificate: export CURL_CA_BUNDLE="/opt/softs/certificats/proxy1_1.pem"

observingClouds · 2026-06-09T08:44:39Z

@uandrae or others with access to Belenos, could you try removing the eccodes and eccodeslib dependencies from [dependencies] in pyproject.toml and add python-eccodes = "*" to [tool.pixi.dependencies]? This way we omit the python wheels and might not have the glib restriction.

uandrae · 2026-06-12T06:32:45Z

@uandrae or others with access to Belenos, could you try removing the eccodes and eccodeslib dependencies from [dependencies] in pyproject.toml and add python-eccodes = "*" to [tool.pixi.dependencies]? This way we omit the python wheels and might not have the glib restriction.

Thanks for the suggestion. Tried it but I'm afraid eccodes is not the only library depending on newer versions. This is the next problem:

Error:   × Cannot install environment 'default'
  ╰─▶ Wheel: pyproj-3.7.2-cp314-cp314-manylinux_2_28_x86_64.whl doesn't match this systems virtual capabilities for tags: cp314-cp314-manylinux_2_17_x86_64

observingClouds · 2026-06-12T07:31:51Z

Well, as container support will probably still take some time, we could check if all those wheels that Belenos has issues with are available on conda-forge. For pyproj this is for example also the case, so you could move pyproj also from [dependencies] to [tool.pixi.dependencies]. If this is of interest I would ask you to do this game until the environment resolves and then report the packages here.

uandrae · 2026-06-12T08:57:45Z

Well, as container support will probably still take some time, we could check if all those wheels that Belenos has issues with are available on conda-forge. For pyproj this is for example also the case, so you could move pyproj also from [dependencies] to [tool.pixi.dependencies]. If this is of interest I would ask you to do this game until the environment resolves and then report the packages here.

Thanks for the suggestion. With the changes below we arrive to

(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ tactus --version
2026-06-12 08:53:55 | INFO     | Start tactus v1.1.0 --> "tactus --version"
tactus v1.1.0
(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ python3 --version
Python 3.14.5

Of course the proof of functionality is in the full execution, but at least this gives a hint about possibilities and limitations.

(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ git diff pyproject.toml
diff --git a/pyproject.toml b/pyproject.toml
index 65fb5f73..ad69ae01 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,8 +21,8 @@
     "dateutils>=0.6.12",
     "dicttoxml>=1.0.0",
     "ecbundle==2.4.0",
-    "eccodes==2.45",
-    "eccodeslib==2.45.4.11",
+    #"eccodes==2.45",
+    #"eccodeslib==2.45.4.11",
     "f90nml>=1.4.3",
     "fastjsonschema==2.18.1",
     "frozendict>=2.4.7",
@@ -33,16 +33,16 @@
     "humanize>=4.8.0",
     "isodate>=0.6.1",
     "json-schema-for-humans>=1.5.1",
-    "json2tab @ git+https://github.com/mfroelund/json2tab.git@feature/adjust-python-version",
+    #"json2tab @ git+https://github.com/mfroelund/json2tab.git@feature/adjust-python-version",
     "jsonref>=1.1.0",
     "loguru>=0.7.1",
-    "numexpr>=2.8.4",
+    #"numexpr>=2.8.4",
     "numpy>=1.22.4",
     "omegaconf>=2.3",
-    "pandas>=1.4.0,<3.0.0",
+    #"pandas>=1.4.0,<3.0.0",
     "pydantic>=2.9.2",
     "pyfdb==0.1.3",
-    "pyproj>=3.3.0,!=3.5.0",
+    #"pyproj>=3.3.0,!=3.5.0",
     "pysurfex>=0.1.2",
     "pyyaml>=6.0",
     "thenamelisttool @ git+https://github.com/UMR-CNRM/thenamelisttool.git",
@@ -102,7 +102,11 @@
 
 [tool.pixi.dependencies]
   # conda-forge provides C libraries that pip wheels don't bundle
+  numexpr = "*"
+  pandas = "*"
+  pyproj = "*"
   pip = "*"
+  python-eccodes = "*"
   eccodes = "==2.45"
   ecflow = ">=5.15.0"
   gdal = "*"

mfroelund · 2026-06-17T09:17:33Z

Well, as container support will probably still take some time, we could check if all those wheels that Belenos has issues with are available on conda-forge. For pyproj this is for example also the case, so you could move pyproj also from [dependencies] to [tool.pixi.dependencies]. If this is of interest I would ask you to do this game until the environment resolves and then report the packages here.

Thanks for the suggestion. With the changes below we arrive to

(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ tactus --version
2026-06-12 08:53:55 | INFO     | Start tactus v1.1.0 --> "tactus --version"
tactus v1.1.0
(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ python3 --version
Python 3.14.5

Of course the proof of functionality is in the full execution, but at least this gives a hint about possibilities and limitations.

(tactus) (tactus) [andraeu@belenoslogin0 tactus]$ git diff pyproject.toml
diff --git a/pyproject.toml b/pyproject.toml
index 65fb5f73..ad69ae01 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,8 +21,8 @@
     "dateutils>=0.6.12",
     "dicttoxml>=1.0.0",
     "ecbundle==2.4.0",
-    "eccodes==2.45",
-    "eccodeslib==2.45.4.11",
+    #"eccodes==2.45",
+    #"eccodeslib==2.45.4.11",
     "f90nml>=1.4.3",
     "fastjsonschema==2.18.1",
     "frozendict>=2.4.7",
@@ -33,16 +33,16 @@
     "humanize>=4.8.0",
     "isodate>=0.6.1",
     "json-schema-for-humans>=1.5.1",
-    "json2tab @ git+https://github.com/mfroelund/json2tab.git@feature/adjust-python-version",
+    #"json2tab @ git+https://github.com/mfroelund/json2tab.git@feature/adjust-python-version",
     "jsonref>=1.1.0",
     "loguru>=0.7.1",
-    "numexpr>=2.8.4",
+    #"numexpr>=2.8.4",
     "numpy>=1.22.4",
     "omegaconf>=2.3",
-    "pandas>=1.4.0,<3.0.0",
+    #"pandas>=1.4.0,<3.0.0",
     "pydantic>=2.9.2",
     "pyfdb==0.1.3",
-    "pyproj>=3.3.0,!=3.5.0",
+    #"pyproj>=3.3.0,!=3.5.0",
     "pysurfex>=0.1.2",
     "pyyaml>=6.0",
     "thenamelisttool @ git+https://github.com/UMR-CNRM/thenamelisttool.git",
@@ -102,7 +102,11 @@
 
 [tool.pixi.dependencies]
   # conda-forge provides C libraries that pip wheels don't bundle
+  numexpr = "*"
+  pandas = "*"
+  pyproj = "*"
   pip = "*"
+  python-eccodes = "*"
   eccodes = "==2.45"
   ecflow = ">=5.15.0"
   gdal = "*"

These changes will most likely also work on other machines than Belenos, so should I implement them on the branch, and then we can do a full test on Belenos to see if it works?

observingClouds added 30 commits March 16, 2026 19:00

WIP

b102b1b

WIP

4ab6d41

WIP

89cbf2f

WIP

96199be

Merge branch 'develop' into feat/pixi

677d5a5

move additional deps under pixi control

87057c8

WIP

444bd54

WIP

1308d15

WIP

9866e51

WIP

816cd25

WIP

2e68327

WIP

3124199

WIP

4c6d9bb

WIP

0c37f72

WIP

e6ba800

WIP

f4ed800

WIP

524b6d8

WIP

bdc4125

WIP

1f9e8e3

WIP

f32a56c

WIP

2d3884f

gdal

8b2d05c

WIP

3560cba

WIP

28ebc3c

gdal

28add9c

WIP

9b443bf

WIP

45d3bd2

pixi.lock

dcd0c74

Merge branch 'develop' into feat/pixi

e23f73d

improve pixi setup

f870fab