bazel-aarch64-linux-gnu

This repo provides a Bazel cross toolchain for x86_64/Linux host and aarch64/Linux target, plus native toolchains for x86_64/Linux and aarch64/Linux.

The structure was inspired by: https://asnaghi.me/post/embedded-bazel/, then re-worked to support bazel CC toolchain auto-resolution / platforms.

Toolchain

The toolchain implemented is: gcc / 11.4.0, from Ubuntu 22.04, in three configurations:

• cross build for aarch64 target on x86_64 host
• native build for aarch64
• native build for x86_64

The toolchain is split into two packages, one with almost everything needed to compile, the other with linux-libc headers, which need to match the target platform's kernel.

The main package contains (tarred from / in their original install paths):

• gcc
• g++
• cpp
• libgcc-s1
• libgcc-dev
• libstdc++
• libstdc++-dev
• libc
• libc-dev
• binutils

Usage

Requires Bazel 7.7.1 or newer, and is tested on 7.7.1 and 9.x in CI. This is a bzlmod-only module: Bazel 9 removed WORKSPACE support, so there is no deps.bzl entry point to load from a WORKSPACE any more.

Include the following in your MODULE.bazel with appropriate commit and integrity hash

bazel_dep(name = "aarch64_linux_gnu", version = "0.0.0")

AARCH64_LINUX_GNU_COMMIT = "INSERT COMMIT HASH HERE"

archive_override(
    module_name = "aarch64_linux_gnu",
    # archive_override takes an SRI integrity hash, not a hex sha256sum:
    #   curl -sL <url> | openssl dgst -sha256 -binary | openssl base64 -A
    integrity = "sha256-INSERT BASE64 DIGEST HERE",
    strip_prefix = "bazel-aarch64-linux-gnu-" + AARCH64_LINUX_GNU_COMMIT,
    urls = ["https://github.com/agtonomy/bazel-aarch64-linux-gnu/archive/" + AARCH64_LINUX_GNU_COMMIT + ".tar.gz"],
)

The module registers its own toolchains, but a dependency's registrations rank below every registration the root module makes -- including the auto-detected host cc toolchain that rules_cc registers, which otherwise wins the native builds and quietly compiles with the host's own gcc. So register them from your root MODULE.bazel too, Jetpack first, so bazel prefers those over the generic toolchains:

register_toolchains(
    "@aarch64_linux_gnu//toolchain:jp512_linux_x86_64",
    "@aarch64_linux_gnu//toolchain:jp62_linux_x86_64",
    "@aarch64_linux_gnu//toolchain:jp512_linux_aarch64",
    "@aarch64_linux_gnu//toolchain:aarch64_linux_x86_64",
    "@aarch64_linux_gnu//toolchain:native_linux_aarch64",
    "@aarch64_linux_gnu//toolchain:native_linux_x86_64",
)

Then include the following in your .bazelrc

build:jetpack_512 --platforms=@aarch64_linux_gnu//platforms:tegra_jetpack_512
build:jetpack_62 --platforms=@aarch64_linux_gnu//platforms:tegra_jetpack_62

A side effect of toolchain resolution is that the bazel output paths don't include the target cpu, to achieve something similar add:

build --experimental_platform_in_output_dir

Finally, build a cc_binary target like so:

bazel build --config jetpack62 //path/to/target

Remote execution

The toolchains declare every file gcc, ld and the wrapper scripts touch as an action input, so they work under remote execution as well as locally. Two things are required of the RE setup:

1. The exec platform must be linux/x86_64 (or linux/aarch64 for the native aarch64 toolchain) and must name a container image. The compilers in these archives are Ubuntu 22.04 host binaries, so the executor has to be able to run them:

   platform(
       name = "re_platform",
       constraint_values = [
           "@platforms//os:linux",
           "@platforms//cpu:x86_64",
       ],
       exec_properties = {
           "container-image": "docker://<your-ubuntu-22.04-based-image>",
       },
   )

   bazel build //path/to/target --config=aarch64 \
       --remote_executor=grpc://<host>:<port> \
       --spawn_strategy=remote \
       --extra_execution_platforms=//:re_platform

2. The image must provide bash and coreutils. The toolchain tools are shell wrappers (see README.hacks) that run bash, realpath, dirname and pwd. Those are host tools, not Bazel inputs, so the executor image has to supply them. Any standard Ubuntu 22.04 base image does.

Checking hermeticity without an RE backend

tests/.bazelrc defines a hermetic config that builds under Bazel's hermetic linux sandbox. Unlike the default sandbox, which symlinks inputs back into the output base and therefore lets an action reach files that were never declared, the hermetic sandbox bind-mounts only the declared inputs — the same contract a remote executor gives an action. An under-declared toolchain input fails there exactly as it would remotely:

cd tests
bazel build //:all --config=aarch64 --config=hermetic_x86_64   # x86_64 host
bazel build //:all --config=hermetic                           # aarch64 host

CI runs this for every configuration, so under-declaration cannot regress silently.

How does this all work?

Beginning from the project repository, the toolchain resolution process goes like:

1. This repo's MODULE.bazel pulls in the gcc and linux-libc archives through the toolchains module extension in extensions.bzl, which declares them with http_archive.
2. MODULE.bazel calls register_toolchains with labels of toolchains, which are defined in toolchain/BUILD. Bazel prefers toolchains registered first if multiple toolchains support a target platform.
3. toolchain/BUILD calls toolchain on outputs from cc_toolchain to associate them with platforms via host and target constraints. Note that constraints which are set on build target platform which are not assigned to a given toolchain are not considered, thus the order mentioned above is used to give jetpack toolchains (with more specific platform constraints) higher selection priority over the generic toolchains.
4. Each toolchain() target's cc_toolchain doesn't live in this repo: a cc_toolchain's tool_path strings (the wrapper scripts) resolve relative to the package of the cc_toolchain target itself, with no way to reach into a different repository, so the cc_toolchain, its config, and its wrapper scripts all have to live in the same repository as the compiler archive they wrap. toolchain/archive_repo.bzl's cc_archive_repo repository rule handles this by fetching the archive itself (in place of a separate http_archive) and writing the wrapper scripts and cc_toolchain/cc_linux_gnu_config declarations into that same generated repo (eg ubuntu-22.04-arm64-cross), called from extensions.bzl. Each compiler may support multiple similar platforms by including slightly different headers and calling cc_toolchain with platform-specific filegroups and config within that same generated repo.
5. The config is created by calling cc_linux_gnu_config, loaded from toolchain/config.bzl, which is a wrapper around cc_common.create_cc_toolchain_config_info. This is where the various toolchain executables (really shell scripts which wrap them,) and build flags are defined. Keep in mind that relative paths are relative to the execroot which bazel sets up for each step of the build, with files from packages in a subdirectory of external, eg <execroot>/external/<repo>/usr/bin/x86_64-linux-gnu-ld, which breaks down to external, then the repo directory, then the path from within the tar archive (after the strip prefix is applied.) Under bzlmod that repo directory is the canonical name Bazel derives from the module and extension, eg aarch64_linux_gnu+toolchains+linux-libc-5.15.0-x86_64, not the plain name passed to http_archive. cc_linux_gnu_config's sysroot attr now points within the same generated archive repo, so it needs no such trick, but libc_headers still crosses into one of the separately fetched linux-libc repos, and takes a Label to read Label.workspace_root off it for exactly this reason. The wrapper scripts have no repository name to resolve at all: see README.hacks.