Add gleam_host convenience repository; complete Phase 4#95
Merged
Conversation
Resolves the TODO in gleam_register_toolchains: creates a "<name>_host"
repository (gleam/private/host_repo.bzl) that detects the host OS/arch and
re-exposes the matching per-platform toolchain repo, so `gleam` can be run
directly without knowing the host's platform string:
bazel run @gleam_host//:gleam -- format
Also fixes a visibility gap in _gleam_repo_impl's generated BUILD.bazel (no
default_visibility, unlike hermetic_erlang_repository.bzl) that would have
made the per-platform repos' raw gleam/gleam.exe file targets unreachable
from gleam_host's alias.
repository_ctx.os.arch on real Apple Silicon macOS reports "arm64" (confirmed by CI on this PR), but _ARCH mapped it to itself instead of normalizing to PLATFORMS' "aarch64" spelling, so host_platform returned the non-existent "arm64-apple-darwin" instead of "aarch64-apple-darwin".
…ITECTURE.md - Pin real, CI-observed sha256 checksums for the hermetic Erlang/OTP toolchain: the built-in default (OTP-28.1, linux_amd64 + macos_arm64) and examples/hermetic_erlang's explicit pin (OTP-29.0.2, same two platforms -- CI's macos-latest runner is Apple Silicon). Platforms CI doesn't exercise (linux_arm64, macos_amd64) remain unpinned until observed on a real run. - Add a CI step building examples/smoke's //:my_app twice (with a `bazel clean` in between to force real re-execution, not a cache hit) and diffing the two escripts byte-for-byte, to catch nondeterminism in gleam_binary's output. - Add ARCHITECTURE.md documenting the repo's layout and its four-layer test taxonomy (Starlark unit tests, gleam_gazelle's Go tests/golden files, example projects, CI-only verification steps), linked from README.md and CONTRIBUTING.md.
The new determinism CI check (bazel build //:my_app twice with a `bazel clean` in between) caught a real bug: escript:create's zip archive stamps each entry with the current wall-clock time when no FileInfo is given, not anything about the input files, so two builds run even a few seconds apart produced byte-different escripts. Reproduced and verified standalone with erl: the old code differs across a 3s gap (char 164, line 4 -- matching the exact CI failure), the fixed code doesn't. Fixed by passing an explicit, fixed file_info() (atime/mtime/ctime pinned to a constant) for every archive entry, and sorting file:list_dir/1's output (also unordered, another potential source of nondeterminism) before building the archive.
The escript_builder.erl fix (previous commit) addressed one real source of nondeterminism (zip entry timestamps) but the determinism check is still failing in CI, now at a different byte offset -- something else still differs. Add a temporary diagnostic that extracts both escripts' archives and reports which named entry's bytes actually differ, so the next fix is based on real evidence instead of another guess.
The diagnostic added in the previous commit showed every single .beam file in the escript archive differed between two builds -- including trivially small ones like my_app.beam. Root-caused locally: erlc embeds the absolute source path (CInf chunk) and compiler cwd (Dbgi chunk) into every .beam file it produces. Gleam's own compiler invokes erlc internally as part of `gleam build`, and Bazel runs each action in a fresh sandbox directory, so that path differs between two otherwise-identical builds of the same target -- reproduced directly with `erlc` compiling identical source from two different directories and diffing the resulting .beam files. Fixed by running beam_lib:strip/1 (which removes these chunks, along with other debug-only metadata not needed at runtime) on every .beam file's bytes before archiving it. Verified end-to-end: compiling the same source in two different directories and building an escript from each now produces byte-identical output; without the fix, they differ. Also removes the temporary archive-diffing diagnostic from ci.yaml added while investigating.
The beam_lib:strip fix (previous commit) closed the gap significantly -- the failure now happens much further into the escript (byte 36436 vs the original byte 167) -- but something still differs. Refine the diagnostic to report the first differing byte offset and a snippet of both entries' bytes for whichever named archive entry still differs, since the previous version only reported sizes.
The refined diagnostic (previous commit) pinpointed the exact remaining culprit:
gleam_stdlib.beam's "Line" chunk, which beam_lib:strip does not remove. Root-caused locally:
Gleam's own codegen emits `-file("<absolute path>/src/foo.gleam", N).` attributes, and erlc
encodes that path into the Line chunk's filename table regardless of stripping -- reproduced
directly by compiling identical source containing such an attribute from two different
directories and diffing the .beam output.
The clean fix is to prevent this at the compiler invocation instead of trying to scrub it
afterwards: erlc's `+deterministic` compile option (or, for `gleam compile-package`'s internal
erlc calls, the equivalent ERL_COMPILER_OPTIONS=[deterministic] environment variable, which
`compile:file` reads regardless of caller) stops all absolute-path embedding at the source.
Applied to every erlc/gleam-compile action: gleam_library.bzl and gleam_test.bzl's
`gleam compile-package` invocations (env var), and gleam_binary.bzl and gleam_test.bzl's direct
erlc calls for the escript shim and eunit runner shim (+deterministic flag).
Verified end-to-end locally: compiling a package and a shim from two different directories,
each containing a `-file()` attribute with a different absolute path, now produces byte-
identical escript output; without either fix it doesn't.
This makes the beam_lib:strip/1 post-processing added two commits ago (and the diagnostic
added after it) unnecessary -- reverted both, keeping only the still-needed zip entry
timestamp fix in escript_builder.erl.
_ARCH maps both "arm64"->"aarch64" and "amd64"->"x86_64", but only the arm64 side had test coverage (which is exactly what caught the real arm64->aarch64 bug earlier). Add the missing symmetric cases for amd64 on both Linux and macOS, so a typo there wouldn't go unnoticed the same way.
4 tasks
aaylward
added a commit
that referenced
this pull request
Jul 3, 2026
## Summary - Follow-up from #95: `erlang/private/hermetic_erlang_repository.bzl`'s `_ARCH` mapping table has the exact same shape as `gleam_host`'s (`arm64`/`amd64` vs. `aarch64`/`x86_64` spellings), and it's the file where the original macOS hermetic-Erlang arch bug was found earlier this session — purely via a real CI failure on a real macOS runner, since no test existed for this file at all. - Extracts the OS/arch/platform-constraint resolution out of `_hermetic_erlang_repository_impl` into a pure `resolve_host(repository_ctx)` function, using the same technique already used for `gleam_host`'s `host_platform` (a plain fake struct standing in for a real `repository_ctx`, no Bazel machinery needed). - Adds `test/unit/hermetic_erlang_repository/` covering every OS x arch combination (including both the raw and erlef-normalized arch spellings on both Linux and macOS) plus the two failure cases (`fail()` on unsupported OS/arch), so a repeat of that bug class fails fast locally instead of needing a real CI run on the affected platform to notice. - No behavior change: `resolve_host` reproduces the exact same URL/checksum-key/constraint values the inlined per-branch logic did before. ## Test plan - [x] `buildifier -mode=diff`/`-lint=warn` clean on all changed/added files (buildifier built locally via `go install` since Bazel itself is unavailable in this environment) - [x] Relevant pre-commit hooks pass (`end-of-file-fixer`, `typos`) - [ ] CI: `bazel test //test/...` (new `hermetic_erlang_repository_test_suite`) - [ ] CI: existing `examples/hermetic_erlang` build/test still passes on both Linux and macOS (proves the refactor didn't change runtime behavior) --- _Generated by [Claude Code](https://claude.ai/code/session_011YhQ33xAXjUGpy7BxwcEhg)_ Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gleam_hostTODO ingleam_register_toolchains: a new<name>_hostrepository (gleam/private/host_repo.bzl) detects the host OS/arch and re-exposes the matching per-platform toolchain repo, so thegleamCLI can be run directly without knowing the host's platform string:bazel run @gleam_host//:gleam -- format._gleam_repo_impl's generatedBUILD.bazel(missingdefault_visibility, unlikehermetic_erlang_repository.bzl) that would otherwise make the per-platform repos' rawgleam/gleam.exefile targets unreachable fromgleam_host's alias.test/unit/host_repo/unit tests (bazel-skylibunittest/analysistest) covering host platform resolution (Linux/macOS/Windows, thearm64/amd64alternate arch spelling, and failure on unsupported OS/arch). CI on a real macOS (Apple Silicon) runner caught a genuine bug this test was built to catch:arm64was being mapped to itself instead of normalized toPLATFORMS'aarch64spelling — fixed.gleam_hostinto the rootMODULE.bazel'suse_repo(...)and adds a CI step that runsbazel run @gleam_host//:gleam -- --versionand checks its output.gleam/extensions.bzl's module docstring andREADME.md.Phase 4 (stretch items), also included in this PR:
linux_amd64+macos_arm64) andexamples/hermetic_erlang's explicit pin (OTP-29.0.2, same two platforms). Platforms CI doesn't exercise (linux_arm64,macos_amd64) remain unpinned until observed on a real run, consistent with the existing convention.examples/smoke's//:my_apptwice (with abazel cleanin between to force real re-execution rather than a cache hit) and diffs the two escripts byte-for-byte, to catch nondeterminism ingleam_binary's output. This check found two real, previously-undetected reproducibility bugs, both fixed here:escript_builder.erl's zip archive stamped each entry with the current wall-clock time (not anything about the input files) when no explicitfile_info()was given, so two builds even a few seconds apart produced different bytes. Fixed by pinning a fixed timestamp for every entry, and sortingfile:list_dir/1's output (also unordered) before archiving..beamfile embedded the compiling machine's absolute sandbox path (via Gleam's own codegen-file(...)attributes, encoded into the "Line" chunk thatbeam_lib:strip/1does not remove) — since Bazel runs each action in a fresh sandbox directory, this path differs between two builds of the same target. Fixed at the source instead of scrubbing it after the fact:erlc's+deterministiccompile option (and, forgleam compile-package's internalerlccalls, the equivalentERL_COMPILER_OPTIONS=[deterministic]env var) stops the embedding entirely. Applied to everyerlc/gleam compile-packageinvocation ingleam_library.bzl,gleam_binary.bzl, andgleam_test.bzl.ARCHITECTURE.mddocumenting the repo's layout and its four-layer test taxonomy (Starlark unit tests,gleam_gazelle's Go tests/golden files, example projects, CI-only verification steps), linked fromREADME.mdandCONTRIBUTING.md.Test plan
buildifier -mode=diff/-lint=warnclean on all changed.bzl/BUILD.bazel/MODULE.bazelfiles (buildifier built locally viago installsince Bazel itself is unavailable in this environment)check-yaml,yamlfmt,end-of-file-fixer,trailing-whitespace,prettier,typos)erl/erlcreproduction (outside Bazel) before pushing each fix, confirming the old code differs and the fixed code doesn'tbazel test //test/...(newhost_repo_test_suite), the new@gleam_host//:gleamrunnability check, and the new determinism checkNOTE:log lines, not guessed