Add ARM64 host support#152
Conversation
Most of the Dockerfile already supports ARM64 through the IMAGE_ARCH build argument. This commit adds a comment at the top explaining how to use IMAGE_ARCH for ARM64 builds, and adapts the Go installation in the skopeo-builder stage to download the correct binary for the target architecture Related-to: TCB-882 Signed-off-by: Gabriel Almeida <gabriel.almeida@toradex.com>
|
|
||
| # Toolchain versions used when the host is aarch64 and the target is also aarch64. | ||
| # These are built with crosstool-ng and have a different naming convention. | ||
| IMAGE_MAJOR_TO_GCC_MAP_AARCH64_NATIVE = { |
There was a problem hiding this comment.
At the time of writing Torizon 6 is still supported. Should we provide a toolchain in this case as well? Or was it already decided that we only support Torizon 7?
There was a problem hiding this comment.
Or was it already decided that we only support Torizon 7?
As far as I know nothing was decided yet. I think we should have the arm64 host toolchains available for Torizon OS 6 as well given that it is currently supported.
| # Toolchain versions used when the host is aarch64 and the target is also aarch64. | ||
| # These are built with crosstool-ng and have a different naming convention. | ||
| IMAGE_MAJOR_TO_GCC_MAP_AARCH64_NATIVE = { | ||
| 7: "crosstool-ng-13.3" |
There was a problem hiding this comment.
Where is this toolchain coming from exactly? Is it part of the TCB container somehow? I don't see in the download sources with the other toolchains: https://sources.toradex.com/tcb/toolchains/
There was a problem hiding this comment.
This is a point that I forgot to explain in my PR description. I haven't put the toolchain on https://sources.toradex.com/tcb/toolchains/ yet, because I didn't know if you guys would accept this method. So, just for my local test, I put the toolchain locally in the Docker's /storage. Do you want me to upload this toolchain before continuing this review?
There was a problem hiding this comment.
As long as the toolchain reliably works I don't have any objections.
But I'll let @lucas-akira and @rborn-tx chime in here as well with their thoughts.
There was a problem hiding this comment.
IMO it's a bit concerning to publish and use a toolchain that was built on someone's machine with an unknown setup. Since the toolchain was built by Toradex, it means we'd be responsible for it.
Say a user complains about some setting used for building the toolchain. We would have trouble to generate a new version only changing that setting since we wouldn't know the original settings.
Given the above, if the only option is to build the toolchain ourselves, then I think we should set up a new CI project for building and possibly publishing a toolchain with a well-tracked setup.
This is of course not a small topic we should discuss.
Cc @griloHBG
There was a problem hiding this comment.
Just an extra piece of information regarding your concern: I used Claude Code to build a crosstool-ng setup using Docker to standardize this toolchain build. When I started this project, I wasn't expecting to improve or maintain it, but after some improvements, it could be a good starting point for the CI project mentioned, at least to see all the versions I used.
To support building kernel modules on an ARM64 host, a new native toolchain built with crosstool-ng was introduced. The existing arm-gnu-toolchain-13.3 is not compatible with ARM64 native builds, so a dedicated toolchain was built matching the following package versions: gcc 13.3, linux kernel 4.20.17, binutils 2.42, glibc 2.38, gdb 14.2, gmp 6.2.1, mpfr 4.2.1, mpc 1.3.1 and isl 0.26. When the host is ARM64 and the target is ARM32, the existing arm-gnu-toolchain cross-compiler is used (e.g. arm-gnu-toolchain-13.3.rel1-aarch64-arm-none-linux-gnueabihf), following the same naming convention already in place for x86_64 hosts. No additional toolchain is needed for this case. Related-to: TCB-882 Signed-off-by: Gabriel Almeida <gabriel.almeida@toradex.com>
Extend the pipeline to natively build, test, and publish the ARM64 variant of torizoncore-builder, following the ARM64 host support added to the Dockerfile and kernel backend. Changes cover the pre-build lint, build-multiarch, test, deploy, and deploy-multiarch stages. All multiarch manifest jobs now include both amd64 and arm64. Previously commented-out ARM64 deploy jobs for the early-access and internal release tracks are now active. Not implemented: - test-common-torizon-* ARM64 variants - test-torizoncore ARM64 variant: this job is externally triggered by Jenkins via BUILD_MACHINE and requires separate Jenkins-side changes to select the ARM64 image - WSL test ARM64 variants: WSL is Windows-only with no ARM64 runner equivalent Related-to: TCB-882 Signed-off-by: Gabriel Almeida <gabriel.almeida@toradex.com>
Add ARM64 env vars (ARM64_RUNNER, ARM64_ARCH, ARM64_IMAGE_NAME) with ARM64_RUNNER centralizing the runner label for easy future updates. Also add lint-python-arm64 job running pylint inside a native ARM64 container and add arm64-build-test-torizoncore6 job to validate the ARM64 TCB image build and integration tests on ubuntu-24.04-arm runners Related-to: TCB-882 Signed-off-by: Gabriel Almeida <gabriel.almeida@toradex.com>
| @@ -239,10 +264,13 @@ def download_toolchain(toolchain, toolchain_path, version_gcc): | |||
| """Download toolchain from online if it doesn't already exist""" | |||
|
|
|||
| url_prefix = "https://sources.toradex.com/tcb/toolchains/" | |||
There was a problem hiding this comment.
It isn't the main purpose of this PR, but I think it would be nice to have URL fallback options when downloading the toolchains.
I imagined something like this: try to download the toolchain from our servers, and if that's not possible for whatever reason e.g. outage, then try to download the toolchain from the official ARM GNU toolchain URL. If that fails, then raise an exception.
This can be implemented for currently supported images (Torizon OS 6 and 7).
| lint-python-arm64: | ||
| runs-on: ${ARM64_RUNNER} | ||
| steps: | ||
| - name: Checkout PR merge branch | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Build dev container tcbuilder-dev-arm64 | ||
| run: docker build . -f torizoncore-builder.Dockerfile | ||
| --build-arg IMAGE_ARCH=linux/arm64/v8 | ||
| --target tcbuilder-dev | ||
| --tag tcbuilder-dev-arm64 | ||
|
|
||
| # Run the code linter disabling only docstring related warnings, use this run as exit code | ||
| # IMPORTANT: when running locally, DO NOT pass these --disable arguments to the program | ||
| - name: Run pylint | ||
| run: docker run --rm -v ${GITHUB_WORKSPACE}:/work -w /work tcbuilder-dev-arm64 | ||
| pylint --output-format=colorized | ||
| --disable=missing-module-docstring | ||
| --disable=missing-class-docstring | ||
| --disable=missing-function-docstring | ||
| --disable=fixme | ||
| $(find tcbuilder/ -type f -name "*.py") *.py || echo "exit_code=$?" >> ${GITHUB_ENV} | ||
| - name: Pylint result | ||
| run: exit ${exit_code} | ||
|
|
There was a problem hiding this comment.
Do we need to run the Python linter on an arm64 host? Its results should be independent from the host architecture, so I don't see the need for this job.
| lint-python-arm64: | ||
| extends: .lint-python-base | ||
| tags: | ||
| - arm64 | ||
| variables: | ||
| ARCH_SUFFIX: arm64 | ||
| IMAGE_ARCH: linux/arm64/v8 | ||
| DEV_CACHE_TAG: tcbuilder-dev-arm64-latest |
There was a problem hiding this comment.
Same point I made on the corresponding GitHub Actions entry: I don't think we need to run the linters on a different host architecture.
| # Toolchain versions used when the host is aarch64 and the target is also aarch64. | ||
| # These are built with crosstool-ng and have a different naming convention. | ||
| IMAGE_MAJOR_TO_GCC_MAP_AARCH64_NATIVE = { | ||
| 7: "crosstool-ng-13.3" |
There was a problem hiding this comment.
IMO it's a bit concerning to publish and use a toolchain that was built on someone's machine with an unknown setup. Since the toolchain was built by Toradex, it means we'd be responsible for it.
Say a user complains about some setting used for building the toolchain. We would have trouble to generate a new version only changing that setting since we wouldn't know the original settings.
Given the above, if the only option is to build the toolchain ourselves, then I think we should set up a new CI project for building and possibly publishing a toolchain with a well-tracked setup.
This is of course not a small topic we should discuss.
Cc @griloHBG
| toolchain_path = os.path.join(os.path.dirname(linux_src), "toolchain") | ||
| host = _host_arch() | ||
| if arch == "arm": | ||
| c_c = "arm-none-linux-gnueabihf-" | ||
| toolchain = os.path.join( | ||
| toolchain_path, f"{version_gcc}-x86_64-arm-none-linux-gnueabihf/bin") | ||
| toolchain_path, f"{version_gcc}-{host}-arm-none-linux-gnueabihf/bin") | ||
| elif arch == "arm64": | ||
| c_c = "aarch64-none-linux-gnu-" | ||
| toolchain = os.path.join( | ||
| toolchain_path, f"{version_gcc}-x86_64-aarch64-none-linux-gnu/bin") | ||
| if host == "aarch64": | ||
| # On aarch64 hosts, use a crosstool-ng-built native toolchain. | ||
| version_gcc = IMAGE_MAJOR_TO_GCC_MAP_AARCH64_NATIVE.get(image_major_version) | ||
| assert version_gcc, "Unable to determine native aarch64 GCC toolchain version" | ||
| c_c = "aarch64-unknown-linux-gnu-" | ||
| toolchain = os.path.join( | ||
| toolchain_path, f"{version_gcc}-{host}-aarch64-unknown-linux-gnu/bin") | ||
| else: | ||
| c_c = "aarch64-none-linux-gnu-" | ||
| toolchain = os.path.join( | ||
| toolchain_path, f"{version_gcc}-{host}-aarch64-none-linux-gnu/bin") | ||
| else: | ||
| assert False, "build_module: Unhandled architecture" |
There was a problem hiding this comment.
Idea for consideration:
Now that we have two dimensions being used to select the toolchain, maybe we could refactor this code to be more declarative. Instead of having logic to define the cross-compiler prefix and toolchain names we could have a 2-dimensional variable in this form:
CROSS_COMPILERS = {
("<host-arch>", "<target-arch>"): ("<c_c>", "<toolchain>")
}where <host-arch> comes from host and <target-arch> comes from arch (I'd actually rename that variable to target.
I think this would be helpful for readability, but before settling on that let me ask @jsrc27 and @lucas-akira opinions.
There was a problem hiding this comment.
I agree that if the custom-built toolchain is needed, we would need to somehow maintain it.
But what could we do to no have this burden? We can discuss in a meeting.
But one idea: whatif we use a newer toolchain version, which IIRC has both ARM64->ARM32 and ARM64->ARM64 (the missing one) available as pre-built?
Which issues could happen?
| # --- | ||
| # ARM64 host test variants | ||
| # Each job mirrors its amd64 counterpart but pulls torizoncore-builder-arm64. | ||
| # --- |
There was a problem hiding this comment.
Given that the majority of TCB's code is independent of the host architecture, I'd guess we are not adding much value by mirroring these various tests from amd64.
Maybe we could consider something more focused such as running the "host tests" for a single image and then only the kernel build_module command (and other commands that we know could be impacted by the host arch) with various image combinations? This should save energy, time and money 😉...
There was a problem hiding this comment.
Make sense. I'll update it.
5 participants
What changed
Toolchain decision
ARM's official releases do not provide a native ARM64→ARM64 toolchain for GCC 13.3, which is the version used in TorizonOS Scarthgap 7.x.y. Three approaches were considered to solve this:
gnu-devtools-for-arm was the first attempt. However, this tool does not support Canadian Cross builds (x86→ARM64→ARM64), which is the required setup to cross-compile the toolchain from an x86 host. Due to this limitation, this approach was discarded.
Linaro ABE was the second attempt, as it was the tool used when GCC 13.3 was originally released. However, it has been fully discontinued and is actively discouraged. At this point, it is not even possible to clone the repository from official sources, so this approach was also dropped.
crosstool-ng was the final choice. Being a highly versatile tool, it fits the build scenario perfectly. To avoid any compatibility issues, the toolchain was built as close as possible to the versions described in the ARM GNU Toolchain 13.3.Rel1 release notes. This approach proved efficient and functional, with validation tests described in the next section.
Testing
The TCB integration tests were run locally and almost all passed on the first run.
The tests that did not pass were not due to bugs in the implementation, but rather limitations of the current test setup: either the new crosstool-ng toolchain is not yet available for download at https://sources.toradex.com/tcb/toolchains/ so the tests could not fetch it automatically, or the test environment used Docker-in-Docker (DinD), which conflicted with tests that also rely on DinD internally.
To validate the kernel module build functionality manually, the toolchain was placed directly in /storage/toolchain and the relevant test was run manually. The full process can be seen here:
Target machine:
Limitations / Future work
Secure Boot was not addressed in this ticket and remains unchanged.
AVAL tests were not configured for ARM64. Tests requiring physical devices depend on internal runners that are not available for ARM64, and no changes were made on that side. The following test jobs remain unimplemented:
test-common-torizon-*ARM64 variantstest-torizoncoreARM64 variant — externally triggered by Jenkins viaBUILD_MACHINE, requiring separate Jenkins-side changes to select the ARM64 image