Automate building Linux jenkins-node-alma9 docker image

[idigs]

Create a github workflow that handles building the Linux jenkins alma9 docker image. There are a couple motivating factors for this effort:

• enhance our agility in addressing security vulnerabilities
• formalize and enforce semantic versioning for jenkins alma9 images

Use case:

1. Practitioner creates a new branch (let's call it branchA) and PR containing changes to Alma9.Dockerfile
2. For every change made to the dockerfile in branchA, the new dockerfile is built on a github runner (ubuntu-latest) and the resulting image is tested for sanity.
3. After approval, the PR is merged, a new release of the docker image is tagged and uploaded to the project's container package registry: https://github.com/mantidproject/dockerfiles/pkgs/container/jenkins-node-alma9

[sf1919]

My primary concern is using a github runner for 2 is whether there is capacity for that. We use github runners for some elements of different repos hosted within the Mantid Project as a whole. We have in the past had warnings about reaching our limit of free github runner use. The limitations on free runner use is one of the major factors as to why we have our own agents. How much we use the github runner might depend on the answer to my next point.

I have to ask what impact step 2 will have on developers debugging the dockerfile image when it fails to build. My own experience is that when it fails to build locally you have to start picking it apart to work out which part of the block where it seems to fail building is the problem. Sometimes it's not that block but something you set wrong earlier. This involves making small changes, blocking out lines and re-running. Not a problem doing this a number of times locally but with a github runner.... I'd want reassurance that you can get the stack trace easily for debugging and that it wouldn't take us over our free amount.

[peterfpeterson]

How long does it take to build the docker images? I don't think we'll have a ton of PRs and the benefit of automation is pretty big.

In principle, people make sure it works locally before committing changes or creating a PR.

[idigs]

Capacity is definitely a valid concern with the free tier azure runners. If it's any consolation, we have an abundance of compute resources for self-hosted runners to augment the free tier Github runners.

Regarding item 2, what do you think about running the docker build workflow only when merging into main and tagging it as a release?

As far as github job log retention, is it possible to configure this for mantidproject?
https://docs.github.com/en/organizations/managing-organization-settings/configuring-the-retention-period-for-github-actions-artifacts-and-logs-in-your-organization

[idigs]

How long does it take to build the docker images? I don't think we'll have a ton of PRs and the benefit of automation is pretty big.

35sec on my test machine:

$ time -- ./build.sh 
[+] Building 35.3s (11/11) FINISHED                                                                                                                                                                                    docker:default
 => [internal] load build definition from Alma9.Dockerfile                                                                                                                                                                       0.0s
 => => transferring dockerfile: 1.32kB                                                                                                                                                                                           0.0s
 => WARN: LegacyKeyValueFormat: "LABEL key=value" should be used instead of legacy "LABEL key value" format (line 7)                                                                                                             0.0s
 => [internal] load metadata for ghcr.io/mantidproject/mantid-development-alma9:0.1                                                                                                                                              0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                                  0.0s
 => CACHED [1/6] FROM ghcr.io/mantidproject/mantid-development-alma9:0.1@sha256:1ba38b41f4f7505bb42cff3a62f3dad1d9c29f0bdbb2aadfbcf380919d6b05b8                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                0.0s
 => => transferring context: 126B                                                                                                                                                                                                0.0s
 => [2/6] RUN yum install -y   java-17-openjdk-devel &&   rm -rf /tmp/* /var/tmp/*                                                                                                                                              27.0s
 => [3/6] RUN export DOCKER_BUILDKIT=1 &&   mkdir -p /jenkins_workdir &&   chmod o+rw /jenkins_workdir &&   curl   --location   --create-dirs   --output /usr/share/jenkins/agent.jar   "https://repo.jenkins-ci.org/public/org  7.1s 
 => [4/6] RUN echo 'export DOCKER_BUILDKIT=1' >> $HOME/.profile                                                                                                                                                                  0.3s 
 => [5/6] COPY --chmod=755 jenkins_agent /usr/share/jenkins/agent.sh                                                                                                                                                             0.0s 
 => [6/6] ADD entrypoint.d/* /etc/entrypoint.d/                                                                                                                                                                                  0.0s 
 => exporting to image                                                                                                                                                                                                           0.7s 
 => => exporting layers                                                                                                                                                                                                          0.7s 
 => => writing image sha256:f0446d3d8b1f39e053d3f7ecb062c1b30c6087c643c63fc734bac3dd3132e1d3                                                                                                                                     0.0s
 => => naming to ghcr.io/mantidproject/jenkins-node-alma9:0.2                                                                                                                                                                    0.0s

 1 warning found (use docker --debug to expand):
 - LegacyKeyValueFormat: "LABEL key=value" should be used instead of legacy "LABEL key value" format (line 7)
[+] Building 0.2s (11/11) FINISHED                                                                                                                                                                                     docker:default
 => [internal] load build definition from Alma9.Dockerfile                                                                                                                                                                       0.0s
 => => transferring dockerfile: 1.32kB                                                                                                                                                                                           0.0s
 => [internal] load metadata for ghcr.io/mantidproject/mantid-development-alma9:0.1                                                                                                                                              0.2s
 => [internal] load .dockerignore                                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                                  0.0s
 => [1/6] FROM ghcr.io/mantidproject/mantid-development-alma9:0.1@sha256:1ba38b41f4f7505bb42cff3a62f3dad1d9c29f0bdbb2aadfbcf380919d6b05b8                                                                                        0.0s
 => [internal] load build context                                                                                                                                                                                                0.0s
 => => transferring context: 126B                                                                                                                                                                                                0.0s
 => CACHED [2/6] RUN yum install -y   java-17-openjdk-devel &&   rm -rf /tmp/* /var/tmp/*                                                                                                                                        0.0s
 => CACHED [3/6] RUN export DOCKER_BUILDKIT=1 &&   mkdir -p /jenkins_workdir &&   chmod o+rw /jenkins_workdir &&   curl   --location   --create-dirs   --output /usr/share/jenkins/agent.jar   "https://repo.jenkins-ci.org/pub  0.0s
 => CACHED [4/6] RUN echo 'export DOCKER_BUILDKIT=1' >> $HOME/.profile                                                                                                                                                           0.0s
 => CACHED [5/6] COPY --chmod=755 jenkins_agent /usr/share/jenkins/agent.sh                                                                                                                                                      0.0s
 => CACHED [6/6] ADD entrypoint.d/* /etc/entrypoint.d/                                                                                                                                                                           0.0s
 => exporting to image                                                                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                                                          0.0s
 => => writing image sha256:f0446d3d8b1f39e053d3f7ecb062c1b30c6087c643c63fc734bac3dd3132e1d3                                                                                                                                     0.0s
 => => naming to ghcr.io/mantidproject/jenkins-node-alma9:latest                                                                                                                                                                 0.0s

 1 warning found (use docker --debug to expand):
 - LegacyKeyValueFormat: "LABEL key=value" should be used instead of legacy "LABEL key value" format (line 7)
real 35.98
user 0.46
sys 0.42

$ docker images -a
REPOSITORY                                               TAG                   IMAGE ID       CREATED          SIZE
ghcr.io/mantidproject/jenkins-node-alma9                 0.2                   f0446d3d8b1f   15 seconds ago   3.71GB
ghcr.io/mantidproject/jenkins-node-alma9                 latest                f0446d3d8b1f   15 seconds ago   3.71GB