feat(docker): Consolidate per-arch manylinux/musllinux dependency images into multi-arch images.#2357
Open
jackluo923 wants to merge 3 commits into
Open
feat(docker): Consolidate per-arch manylinux/musllinux dependency images into multi-arch images.#2357jackluo923 wants to merge 3 commits into
jackluo923 wants to merge 3 commits into
Conversation
Contributor
WalkthroughThe PR updates container build and consumption paths for multi-architecture dependency images. It adds arch-aware naming to build and run actions, switches the dependency base images and build scripts to multi-arch targets, rewires the workflow to build and merge per-arch images, and updates related docs. ChangesMulti-arch dependency image pipeline
Sequence Diagram(s)sequenceDiagram
participant clp_artifact_build
participant manylinux_2_28_deps_image
participant musllinux_1_2_deps_image
participant deps_image_merge
participant manylinux_2_28_x86_64_binaries
participant musllinux_1_2_x86_64_binaries
clp_artifact_build->>manylinux_2_28_deps_image: build amd64 and arm64 images
clp_artifact_build->>musllinux_1_2_deps_image: build amd64 and arm64 images
manylinux_2_28_deps_image->>deps_image_merge: publish main-amd64 and main-arm64 tags
musllinux_1_2_deps_image->>deps_image_merge: publish main-amd64 and main-arm64 tags
deps_image_merge->>manylinux_2_28_x86_64_binaries: expose multi-arch :main manifest
deps_image_merge->>musllinux_1_2_x86_64_binaries: expose multi-arch :main manifest
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jackluo923
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/clp-artifact-build.yaml:
- Around line 178-220: The deps-image merge logic is incorrectly shared across
OS matrix values, but jobs.<job_id>.if cannot reliably use matrix so this job
will not run as intended. Split deps-image-merge into two separate jobs named
manylinux_2_28-deps-image-merge and musllinux_1_2-deps-image-merge, each with
its own fixed needs on the matching OS build job, and keep the manifest creation
steps scoped to that OS’s image base. Then update any downstream dependencies
that currently reference deps-image-merge to point to the corresponding per-OS
merge job.
In `@components/core/tools/docker-images/clp-env-base-manylinux_2_28/Dockerfile`:
- Around line 3-5: The Dockerfile base image reference is unpinned, so builds
can drift when the upstream tag changes. Update the FROM instruction in the
manylinux base image Dockerfile to use an explicit version tag or a sha256
digest for quay.io/pypa/manylinux_2_28, and keep the multi-arch build flow
intact so build.sh and DOCKER_PULL still control when the base image is
refreshed.
In `@components/core/tools/docker-images/clp-env-base-musllinux_1_2/Dockerfile`:
- Around line 3-5: Pin the base image used by the Dockerfile so builds are
reproducible; the current FROM quay.io/pypa/musllinux_1_2 in
clp-env-base-musllinux_1_2 resolves to latest and can drift. Update the image
reference in the Dockerfile to an explicit version tag or a sha256 digest,
matching the pinning approach used by the corresponding manylinux base image.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: f4ef5897-65a1-46fe-a1c6-a6c80566ee8a
📒 Files selected for processing (17)
.github/actions/clp-core-build-containers/action.yaml.github/actions/run-on-image/action.yaml.github/workflows/clp-artifact-build.yamlcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64/build.shcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/Dockerfilecomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/build.shcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28/Dockerfilecomponents/core/tools/docker-images/clp-env-base-manylinux_2_28/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-aarch64/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/Dockerfilecomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2/Dockerfilecomponents/core/tools/docker-images/clp-env-base-musllinux_1_2/build.shcomponents/core/tools/packaging/build.shcomponents/core/tools/packaging/universal-deb/Dockerfiledocs/src/dev-docs/tooling-containers.mddocs/src/dev-docs/tooling-gh-workflows.md
💤 Files with no reviewable changes (6)
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/Dockerfile
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-aarch64/build.sh
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/build.sh
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/Dockerfile
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64/build.sh
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/build.sh
jackluo923
force-pushed
the
feat/multiarch-linux
branch
from
e2e7f61 to
1f60651
Compare
…ges into multi-arch images
Replace the four architecture-specific base-image directories
(clp-env-base-{manylinux_2_28,musllinux_1_2}-{aarch64,x86_64}) with two
multi-arch directories that select the target architecture via Docker
buildx `--platform` (locally through the `PLATFORM` env var in build.sh).
Both architectures now share a single Dockerfile/build.sh per image family.
Publish the dependency images as multi-arch manifests on `main`: each
architecture is built natively on its matching runner and published under an
arch-suffixed tag (e.g. `:main-amd64`), then per-OS `*-deps-image-merge` jobs
combine the per-arch tags into a single multi-arch `:main` manifest. Consumers
pull `:main` and Docker auto-selects the host architecture. On PRs the images
are still consumed as local tar artifacts (no publish/merge).
Update the `clp-core-build-containers` and `run-on-image` composite actions
with an `arch` input to carry the architecture through arch-suffixed
tags/artifacts, update `packaging/build.sh` to build the consolidated base
image via `PLATFORM`, and refresh the dev docs (pull commands, paths, and the
workflow diagram).
Also fix a Python 3.6 regression in the shared deb/rpm builder image:
`dnf install dpkg` pulls in AlmaLinux's python3-3.6, which wins the
`/usr/bin/python3` alternative and is too old for the auto-parallelism script
invoked by the packaging task. Re-register the manylinux python3.12 at a
higher alternatives priority to restore it as the default `python3`.
jackluo923
force-pushed
the
feat/multiarch-linux
branch
from
1f60651 to
bb00596
Compare
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/clp-artifact-build.yaml (1)
387-413: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winAllow image-only PR changes through the consumer jobs.
manylinux_2_28-deps-image-mergeandmusllinux_1_2-deps-image-mergeare skipped on PRs, but these downstream jobs still need them and only reopen the skipped path forclp_changed/yscope_clp_core_changed. A PR that changes only the manylinux or musllinux dependency image will skip the binaries/wheels validation instead of running against the rebuilt local tarball. Add the image-changed PR case to theseifs, or split the PR/main dependency chains.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/clp-artifact-build.yaml around lines 387 - 413, The downstream consumer jobs are still gated out when only the dependency image changes, so image-only PRs skip validation instead of running against the rebuilt local tarball. Update the job conditions in the workflow entries that consume the manylinux/musllinux images (including the manylinux_2_28-x86_64 linked-bins job and the corresponding image-dependent jobs) to also reopen the path when the relevant image-changed PR outputs are true, or otherwise split the PR/main dependency gating so image rebuilds flow into the consumer jobs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/clp-artifact-build.yaml:
- Around line 387-413: The downstream consumer jobs are still gated out when
only the dependency image changes, so image-only PRs skip validation instead of
running against the rebuilt local tarball. Update the job conditions in the
workflow entries that consume the manylinux/musllinux images (including the
manylinux_2_28-x86_64 linked-bins job and the corresponding image-dependent
jobs) to also reopen the path when the relevant image-changed PR outputs are
true, or otherwise split the PR/main dependency gating so image rebuilds flow
into the consumer jobs.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: af76fa27-b879-46be-b632-03ed3b1bb6ce
📒 Files selected for processing (17)
.github/actions/clp-core-build-containers/action.yaml.github/actions/run-on-image/action.yaml.github/workflows/clp-artifact-build.yamlcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64/build.shcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/Dockerfilecomponents/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/build.shcomponents/core/tools/docker-images/clp-env-base-manylinux_2_28/Dockerfilecomponents/core/tools/docker-images/clp-env-base-manylinux_2_28/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-aarch64/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/Dockerfilecomponents/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/build.shcomponents/core/tools/docker-images/clp-env-base-musllinux_1_2/Dockerfilecomponents/core/tools/docker-images/clp-env-base-musllinux_1_2/build.shcomponents/core/tools/packaging/build.shcomponents/core/tools/packaging/universal-deb/Dockerfiledocs/src/dev-docs/tooling-containers.mddocs/src/dev-docs/tooling-gh-workflows.md
💤 Files with no reviewable changes (6)
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64/build.sh
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/build.sh
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/Dockerfile
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-aarch64/build.sh
- components/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/Dockerfile
- components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/build.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Consolidates the four architecture-specific base-image directories (
clp-env-base-{manylinux_2_28,musllinux_1_2}-{aarch64,x86_64}) into two multi-arch directories that select the target architecture via Docker buildx--platform(locally through aPLATFORMenv var inbuild.sh). Both architectures now share a single Dockerfile/build.sh per image family.The dependency images are published as multi-arch manifests on
main: each architecture is built natively on its matching runner and published under an arch-suffixed tag (e.g.:main-amd64), then a singledeps-image-mergematrix job combines the per-arch tags into one multi-arch:mainmanifest. Consumers pull:mainand Docker auto-selects the host architecture. On PRs the images are still consumed as local tar artifacts (no publish/merge), so the merge job is gated to push-to-mainonly; each leg is gated on its own deps-image matrix result so a failure or skip of one OS doesn't block the other.Other changes:
clp-core-build-containersandrun-on-imagecomposite actions gain anarchinput to carry the architecture through arch-suffixed tags/artifacts.packaging/build.shbuilds the consolidated base image viaPLATFORMand the three downstream binary/wheel jobs nowneedthe mergeddeps-image-mergematrix job.dnf install dpkgpulls in AlmaLinux's python3-3.6, which wins the/usr/bin/python3alternative and is too old for the auto-parallelism script (build-and-run-unit-tests.py --compute-max-parallel-jobs) invoked by the packaging task. Re-registers the manylinuxpython3.12at a higher alternatives priority to restore it as the defaultpython3.Migration note: the published ghcr repository name changes from separate per-arch repos (
clp-core-dependencies-{aarch64,x86}-{manylinux_2_28,musllinux_1_2}) to a single multi-arch repo per family (clp-core-dependencies-{manylinux_2_28,musllinux_1_2}). The old per-arch ghcr repos will no longer receive:mainupdates; any external consumers pulling the old per-arch repo names should switch to the new:mainmulti-arch tag. No in-repo consumers reference the old names.Scope note: ubuntu-jammy and centos-stream-9 dependency images are intentionally left on the legacy per-arch model (out of scope for this PR).
Checklist
Validation performed
clp-env-base-manylinux_2_28andclp-env-base-musllinux_1_2— for amd64 and arm64 (viaPLATFORM).clp-send-to-end inside the new manylinux image.packaging/build.sh --format all --arch aarch64; verified each package installs andclp-s --helpruns in emulated arm64 containers (debian:bookworm, almalinux:9, alpine:3.20).python3 --versionreports 3.12.13 (was 3.6.8) and the taskfile auto-parallelism command (build-and-run-unit-tests.py --compute-max-parallel-jobs) runs successfully (previously failed with aSyntaxError).needs/ifskip-propagation across event scenarios (PR image-only, PR clp-only, push-to-main, schedule).Summary by CodeRabbit
New Features
Documentation