Switch the nightly build to producing a multiarch docker image that supports both x86_64 and arm64 by eranrund · Pull Request #133 · clux/muslrust

eranrund · 2024-02-26T06:17:30Z

Hi!

This PR reworks how the Docker image is built in a way that allows building a multiarch image that supports both x86_64 and arm64. I based it on #104 but the changes are pretty significant due to the need to use the docker buildx flow, QEMU, etc. I based it off the example in https://docs.docker.com/build/ci/github-actions/multi-platform/.

I hope you'll be open to this approach. ~~If so, I'll be happy to convert the stable CI job as well.~~ this now includes the stable build as well since it was needed for my use-case.

Happy to answer any questions about why I did things a certain way in case it is not obvious.

Thank you :)

…upports both x86_64 and arm64

clux

thanks a lot for doing this! i am definitely open to this approach! the relatively small amount of changes herein are definitely small enough that it is maintainable, and the extensions look clean.

CI is red atm, but happy to press buttons whenever. Have only suggested minor things so far from a quick scan.

clux · 2024-02-26T19:38:52Z

+RUN mkdir $PREFIX && \
+    echo "$PREFIX/lib" >> /etc/ld-musl-aarch64.path && \
+    ln -s /usr/include/aarch64-linux-gnu/asm /usr/include/aarch64-linux-musl/asm && \
+    ln -s /usr/include/asm-generic /usr/include/aarch64-linux-musl/asm-generic && \
+    ln -s /usr/include/linux /usr/include/aarch64-linux-musl/linux


this seems to be the most changed bit! apart from a few aarch references this is remarkably similar!

Yeah, it might be possible to merge the two files in the future. I opted not to try just to make things easier to review.
Similarly, it should be possible to merge the nightly and stable actions by using a reusable action and just passing different parameters.

Co-authored-by: Eirik A <sszynrae@gmail.com>

eranrund · 2024-02-26T20:00:19Z

thanks a lot for doing this! i am definitely open to this approach! the relatively small amount of changes herein are definitely small enough that it is maintainable, and the extensions look clean.

CI is red atm, but happy to press buttons whenever. Have only suggested minor things so far from a quick scan.

You're welcome! I'm happy you're open to accepting it :)
Just committed your suggestions. We do need to get the docker login to work for the buildx stuff to work, and I imagine the dockerhub secret isn't being passed to CI checks coming from a fork (which is reasonable since you don't want random people having access to it).
You might have to push a copy of this branch into this repo yourself, and then create a PR for it to get the dockerhub token secret thing working...

clux · 2024-02-26T20:04:34Z

Ah, I didn't realise the CI stuff was already tested elsewhere! Figured there had to be a few cycles.

I'll give it a go!

eranrund · 2024-02-26T20:15:43Z

Ah, I didn't realise the CI stuff was already tested elsewhere! Figured there had to be a few cycles.

I'll give it a go!

Oh, yes, this took many iterations to nail down :)

eranrund · 2024-02-26T20:24:38Z

I just realized that we'll need some hack to force the stable build to run again since otherwise CI will just skip it since there hasn't been a new Rust release since the last time it ran. So probably need to comment out that check, let it run that way, and then undo that.

clux · 2024-02-26T20:46:06Z

Oh, yes, this took many iterations to nail down :)

🙈 i feel your pain. This must have been quite a slog with how slow the arm build is.

clux · 2024-02-26T20:48:35Z

I just realized that we'll need some hack to force the stable build to run again since otherwise CI will just skip it since there hasn't been a new Rust release since the last time it ran.

yeah, that's an easy one to fix by the end. am cleaning up a few things in the justfile and test.sh for quick local testing first. also want to do some sanity on mac tomorrow.

..wondering also if this is a good time to do some other breaking changes i've been wanting to do like fixing the workspace dir and maybe ripping out pq. could do a lot of this under a new tagging scheme.

clux · 2024-02-26T21:10:47Z

under a new tagging scheme

ah, i suppose the tags are already overwritten. this worked immediately!

have pushed some small test stuff and will do some sanity tests tomorrow on a mac. once that is done, feel free to cherry pick my changes back into your branch and we can probably merge.

eranrund · 2024-02-26T21:12:12Z

maybe ripping out pq

Please don't! I am relying on it (and may not be the only one)...

eranrund · 2024-02-26T21:13:14Z

have pushed some small test stuff and will do some sanity tests tomorrow on a mac. once that is done, feel free to cherry pick my changes back into your branch and we can probably merge.

Amazing, thank you. Please let me know once you're done with your testing and I'll cherry pick

clux · 2024-02-26T21:16:47Z

Please don't! I am relying on [pq] (and may not be the only one)...

ok, i'll leave it, but be aware that it's not in the greatest state atm.

eranrund · 2024-02-26T21:46:13Z

Please don't! I am relying on [pq] (and may not be the only one)...

ok, i'll leave it, but be aware that it's not in the greatest state atm.

Interesting, good to know. Just saw you mentioning that blackdex has a newer version so would be nice to see if that can work here as well

clux

quick questions

clux · 2024-02-27T18:33:16Z

+ENV PATH=/root/.cargo/bin:$PREFIX/bin:$PATH \
+    RUSTUP_HOME=/root/.rustup \
+    CARGO_BUILD_TARGET=aarch64-unknown-linux-musl \
+    CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-Clink-self-contained=yes -Clinker=rust-lld -Ctarget-feature=+crt-static" \


If we were to smash these dockerfiles together, having both CARGO_TARGET_X evars here for different arches should be fine, right? this is one of the few big changes

I think so, but we will have to verify.

clux · 2024-02-27T18:34:36Z

+# TODO: fix so that it works
+RUN curl -sSL https://www.openssl.org/source/openssl-$SSL_VER.tar.gz | tar xz && \
+    cd openssl-$SSL_VER && \
+    CFLAGS="-mno-outline-atomics" ./Configure no-zlib no-shared -fPIC --prefix=$PREFIX --openssldir=$PREFIX/ssl linux-aarch64 && \


comparing amd64 and arm64:

- CFLAGS="-mno-outline-atomics" ./Configure no-zlib no-shared -fPIC --prefix=$PREFIX --openssldir=$PREFIX/ssl linux-aarch64 && \ + ./Configure no-zlib no-shared -fPIC --prefix=$PREFIX --openssldir=$PREFIX/ssl linux-x86_64 && \

this is the other big change, do you remember what was the deal with this?

If I recall correctly, without CFLAGS="-mno-outline-atomics", I got a bunch of linker errors for some missing symbols.

hehe ok. sounds like a standard wtf style google session then.

clux · 2024-02-27T19:06:30Z

Sanity done in #134, feel free to cherry-pick my changes and i'll merge it in :-)

Signed-off-by: clux <sszynrae@gmail.com>

eranrund · 2024-02-27T19:25:00Z

Hi @clux , just pushed the cherry-picked changes. Thanks again for moving fast on this!

clux · 2024-02-27T19:29:45Z

Awesome. Thank you so much.

This was one of those big projects that I always struggled to understand how to do it, and you've done it with very elegantly.
Will look towards maybe combining the two dockerfiles later, and moving on some kube stuff to produce multi-arch images as a result!

Super helpful. Thank you again.

eranrund · 2024-02-27T19:34:40Z

Awesome. Thank you so much.

This was one of those big projects that I always struggled to understand how to do it, and you've done it with very elegantly. Will look towards maybe combining the two dockerfiles later, and moving on some kube stuff to produce multi-arch images as a result!

Super helpful. Thank you again.

My pleasure! Thanks for maintaining this!

Could you please force a release of a new stable image so we can get the multiarch published to dockerhub?

clux · 2024-02-27T19:41:43Z

Could you please force a release of a new stable image so we can get the multiarch published to dockerhub?

sure thing, it's building from this as we speak

My pleasure! Thanks for maintaining this!

work is easy when people do all the work for me :-)
on that note, i did also send you collab access if you should ever need it, but don't feel pressured to take it / use it

eranrund · 2024-02-27T21:19:33Z

sure thing, it's building from this as we speak

It built but the merge step was skipped :(

eranrund · 2024-02-27T22:52:29Z

Oops, we forgot to remove

muslrust/.github/workflows/stable.yml

Line 178 in 1149c56

if: false

clux · 2024-02-27T22:54:04Z

yea 😞

running a faster build now without tests just to get something out the door

clux · 2024-02-27T23:22:09Z

ok, we have a stable build 🎉

eranrund · 2024-02-27T23:44:37Z

beautiful, thank you! just used it to build a project on my Apple Silicon Mac :)

Switch the nightly build to producing a multiarch docker image that s…

1af7be7

…upports both x86_64 and arm64

eranrund commented Feb 26, 2024

View reviewed changes

Comment thread Dockerfile.x86_64

eranrund commented Feb 26, 2024

View reviewed changes

Comment thread test.sh

Multi-arch stable build.

8781ff0

clux reviewed Feb 26, 2024

View reviewed changes

clux linked an issue Feb 26, 2024 that may be closed by this pull request

explore building an arm64 variant #97

Closed

Apply suggestions from code review

0f5cd44

Co-authored-by: Eirik A <sszynrae@gmail.com>

clux mentioned this pull request Feb 26, 2024

arm draft verify #134

Closed

clux reviewed Feb 27, 2024

View reviewed changes

Various small fixes.

bce06bd

Signed-off-by: clux <sszynrae@gmail.com>

clux approved these changes Feb 27, 2024

View reviewed changes

clux merged commit 24af91b into clux:main Feb 27, 2024

clux mentioned this pull request Feb 27, 2024

Aarch64 Draft #104

Closed

clux mentioned this pull request Feb 27, 2024

explore building an arm64 variant #97

Closed

This was referenced May 23, 2025

ci: Refactor stable.yml #163

Merged

chore: Refactor test.sh #165

Merged

Uh oh!

Conversation

eranrund commented Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

clux commented Feb 26, 2024

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

clux commented Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clux commented Feb 26, 2024

Uh oh!

clux commented Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

clux commented Feb 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eranrund commented Feb 26, 2024

Uh oh!

clux left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clux commented Feb 27, 2024

Uh oh!

eranrund commented Feb 27, 2024

Uh oh!

clux commented Feb 27, 2024

Uh oh!

eranrund commented Feb 27, 2024

Uh oh!

clux commented Feb 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eranrund commented Feb 27, 2024

Uh oh!

eranrund commented Feb 27, 2024

Uh oh!

clux commented Feb 27, 2024

Uh oh!

clux commented Feb 27, 2024

Uh oh!

eranrund commented Feb 27, 2024

Uh oh!

Reviewers

Assignees

eranrund commented Feb 26, 2024 •

edited

Loading

clux commented Feb 26, 2024 •

edited

Loading

clux commented Feb 26, 2024 •

edited

Loading

clux commented Feb 26, 2024 •

edited

Loading

clux commented Feb 27, 2024 •

edited

Loading