Skip to content

Add gleam_host convenience repository; complete Phase 4#95

Merged
aaylward merged 9 commits into
mainfrom
claude/gleam-bazel-rules-review-0pfz4h
Jul 3, 2026
Merged

Add gleam_host convenience repository; complete Phase 4#95
aaylward merged 9 commits into
mainfrom
claude/gleam-bazel-rules-review-0pfz4h

Conversation

@aaylward

@aaylward aaylward commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Resolves the gleam_host TODO in gleam_register_toolchains: a new <name>_host repository (gleam/private/host_repo.bzl) detects the host OS/arch and re-exposes the matching per-platform toolchain repo, so the gleam CLI can be run directly without knowing the host's platform string: bazel run @gleam_host//:gleam -- format.
  • Fixes a visibility gap in _gleam_repo_impl's generated BUILD.bazel (missing default_visibility, unlike hermetic_erlang_repository.bzl) that would otherwise make the per-platform repos' raw gleam/gleam.exe file targets unreachable from gleam_host's alias.
  • Adds test/unit/host_repo/ unit tests (bazel-skylib unittest/analysistest) covering host platform resolution (Linux/macOS/Windows, the arm64/amd64 alternate arch spelling, and failure on unsupported OS/arch). CI on a real macOS (Apple Silicon) runner caught a genuine bug this test was built to catch: arm64 was being mapped to itself instead of normalized to PLATFORMS' aarch64 spelling — fixed.
  • Wires gleam_host into the root MODULE.bazel's use_repo(...) and adds a CI step that runs bazel run @gleam_host//:gleam -- --version and checks its output.
  • Documents the new repository in gleam/extensions.bzl's module docstring and README.md.

Phase 4 (stretch items), also included in this PR:

  • Pins real, CI-observed sha256 checksums for the hermetic Erlang/OTP toolchain: the built-in default (OTP-28.1, linux_amd64 + macos_arm64) and examples/hermetic_erlang's explicit pin (OTP-29.0.2, same two platforms). Platforms CI doesn't exercise (linux_arm64, macos_amd64) remain unpinned until observed on a real run, consistent with the existing convention.
  • Adds a CI step that builds examples/smoke's //:my_app twice (with a bazel clean in between to force real re-execution rather than a cache hit) and diffs the two escripts byte-for-byte, to catch nondeterminism in gleam_binary's output. This check found two real, previously-undetected reproducibility bugs, both fixed here:
    • escript_builder.erl's zip archive stamped each entry with the current wall-clock time (not anything about the input files) when no explicit file_info() was given, so two builds even a few seconds apart produced different bytes. Fixed by pinning a fixed timestamp for every entry, and sorting file:list_dir/1's output (also unordered) before archiving.
    • Every compiled .beam file embedded the compiling machine's absolute sandbox path (via Gleam's own codegen -file(...) attributes, encoded into the "Line" chunk that beam_lib:strip/1 does not remove) — since Bazel runs each action in a fresh sandbox directory, this path differs between two builds of the same target. Fixed at the source instead of scrubbing it after the fact: erlc's +deterministic compile option (and, for gleam compile-package's internal erlc calls, the equivalent ERL_COMPILER_OPTIONS=[deterministic] env var) stops the embedding entirely. Applied to every erlc/gleam compile-package invocation in gleam_library.bzl, gleam_binary.bzl, and gleam_test.bzl.
  • Adds ARCHITECTURE.md documenting the repo's layout and its four-layer test taxonomy (Starlark unit tests, gleam_gazelle's Go tests/golden files, example projects, CI-only verification steps), linked from README.md and CONTRIBUTING.md.

Test plan

  • buildifier -mode=diff/-lint=warn clean on all changed .bzl/BUILD.bazel/MODULE.bazel files (buildifier built locally via go install since Bazel itself is unavailable in this environment)
  • Relevant pre-commit hooks pass (check-yaml, yamlfmt, end-of-file-fixer, trailing-whitespace, prettier, typos)
  • Both reproducibility bugs verified end-to-end locally with a standalone erl/erlc reproduction (outside Bazel) before pushing each fix, confirming the old code differs and the fixed code doesn't
  • CI green on both Linux and macOS: bazel test //test/... (new host_repo_test_suite), the new @gleam_host//:gleam runnability check, and the new determinism check
  • Real checksums for the hermetic Erlang/OTP pins were taken directly from CI's own printed NOTE: log lines, not guessed

claude added 3 commits July 3, 2026 21:57
Resolves the TODO in gleam_register_toolchains: creates a "<name>_host"
repository (gleam/private/host_repo.bzl) that detects the host OS/arch and
re-exposes the matching per-platform toolchain repo, so `gleam` can be run
directly without knowing the host's platform string:

    bazel run @gleam_host//:gleam -- format

Also fixes a visibility gap in _gleam_repo_impl's generated BUILD.bazel (no
default_visibility, unlike hermetic_erlang_repository.bzl) that would have
made the per-platform repos' raw gleam/gleam.exe file targets unreachable
from gleam_host's alias.
repository_ctx.os.arch on real Apple Silicon macOS reports "arm64" (confirmed by CI on this PR),
but _ARCH mapped it to itself instead of normalizing to PLATFORMS' "aarch64" spelling, so
host_platform returned the non-existent "arm64-apple-darwin" instead of "aarch64-apple-darwin".
…ITECTURE.md

- Pin real, CI-observed sha256 checksums for the hermetic Erlang/OTP toolchain: the built-in
  default (OTP-28.1, linux_amd64 + macos_arm64) and examples/hermetic_erlang's explicit pin
  (OTP-29.0.2, same two platforms -- CI's macos-latest runner is Apple Silicon). Platforms CI
  doesn't exercise (linux_arm64, macos_amd64) remain unpinned until observed on a real run.
- Add a CI step building examples/smoke's //:my_app twice (with a `bazel clean` in between to
  force real re-execution, not a cache hit) and diffing the two escripts byte-for-byte, to catch
  nondeterminism in gleam_binary's output.
- Add ARCHITECTURE.md documenting the repo's layout and its four-layer test taxonomy (Starlark
  unit tests, gleam_gazelle's Go tests/golden files, example projects, CI-only verification
  steps), linked from README.md and CONTRIBUTING.md.
@aaylward aaylward changed the title Add gleam_host convenience repository Add gleam_host convenience repository; complete Phase 4 Jul 3, 2026
claude added 6 commits July 3, 2026 22:16
The new determinism CI check (bazel build //:my_app twice with a `bazel clean` in between)
caught a real bug: escript:create's zip archive stamps each entry with the current wall-clock
time when no FileInfo is given, not anything about the input files, so two builds run even a
few seconds apart produced byte-different escripts. Reproduced and verified standalone with erl:
the old code differs across a 3s gap (char 164, line 4 -- matching the exact CI failure), the
fixed code doesn't.

Fixed by passing an explicit, fixed file_info() (atime/mtime/ctime pinned to a constant) for
every archive entry, and sorting file:list_dir/1's output (also unordered, another potential
source of nondeterminism) before building the archive.
The escript_builder.erl fix (previous commit) addressed one real source of nondeterminism
(zip entry timestamps) but the determinism check is still failing in CI, now at a different
byte offset -- something else still differs. Add a temporary diagnostic that extracts both
escripts' archives and reports which named entry's bytes actually differ, so the next fix is
based on real evidence instead of another guess.
The diagnostic added in the previous commit showed every single .beam file in the escript
archive differed between two builds -- including trivially small ones like my_app.beam.
Root-caused locally: erlc embeds the absolute source path (CInf chunk) and compiler cwd (Dbgi
chunk) into every .beam file it produces. Gleam's own compiler invokes erlc internally as part
of `gleam build`, and Bazel runs each action in a fresh sandbox directory, so that path differs
between two otherwise-identical builds of the same target -- reproduced directly with `erlc`
compiling identical source from two different directories and diffing the resulting .beam files.

Fixed by running beam_lib:strip/1 (which removes these chunks, along with other debug-only
metadata not needed at runtime) on every .beam file's bytes before archiving it. Verified
end-to-end: compiling the same source in two different directories and building an escript from
each now produces byte-identical output; without the fix, they differ.

Also removes the temporary archive-diffing diagnostic from ci.yaml added while investigating.
The beam_lib:strip fix (previous commit) closed the gap significantly -- the failure now
happens much further into the escript (byte 36436 vs the original byte 167) -- but something
still differs. Refine the diagnostic to report the first differing byte offset and a snippet
of both entries' bytes for whichever named archive entry still differs, since the previous
version only reported sizes.
The refined diagnostic (previous commit) pinpointed the exact remaining culprit:
gleam_stdlib.beam's "Line" chunk, which beam_lib:strip does not remove. Root-caused locally:
Gleam's own codegen emits `-file("<absolute path>/src/foo.gleam", N).` attributes, and erlc
encodes that path into the Line chunk's filename table regardless of stripping -- reproduced
directly by compiling identical source containing such an attribute from two different
directories and diffing the .beam output.

The clean fix is to prevent this at the compiler invocation instead of trying to scrub it
afterwards: erlc's `+deterministic` compile option (or, for `gleam compile-package`'s internal
erlc calls, the equivalent ERL_COMPILER_OPTIONS=[deterministic] environment variable, which
`compile:file` reads regardless of caller) stops all absolute-path embedding at the source.
Applied to every erlc/gleam-compile action: gleam_library.bzl and gleam_test.bzl's
`gleam compile-package` invocations (env var), and gleam_binary.bzl and gleam_test.bzl's direct
erlc calls for the escript shim and eunit runner shim (+deterministic flag).

Verified end-to-end locally: compiling a package and a shim from two different directories,
each containing a `-file()` attribute with a different absolute path, now produces byte-
identical escript output; without either fix it doesn't.

This makes the beam_lib:strip/1 post-processing added two commits ago (and the diagnostic
added after it) unnecessary -- reverted both, keeping only the still-needed zip entry
timestamp fix in escript_builder.erl.
_ARCH maps both "arm64"->"aarch64" and "amd64"->"x86_64", but only the arm64 side had test
coverage (which is exactly what caught the real arm64->aarch64 bug earlier). Add the missing
symmetric cases for amd64 on both Linux and macOS, so a typo there wouldn't go unnoticed the
same way.
@aaylward aaylward merged commit 17bdad1 into main Jul 3, 2026
5 checks passed
@aaylward aaylward deleted the claude/gleam-bazel-rules-review-0pfz4h branch July 3, 2026 22:53
aaylward added a commit that referenced this pull request Jul 3, 2026
## Summary
- Follow-up from #95: `erlang/private/hermetic_erlang_repository.bzl`'s
`_ARCH` mapping table has the exact same shape as `gleam_host`'s
(`arm64`/`amd64` vs. `aarch64`/`x86_64` spellings), and it's the file
where the original macOS hermetic-Erlang arch bug was found earlier this
session — purely via a real CI failure on a real macOS runner, since no
test existed for this file at all.
- Extracts the OS/arch/platform-constraint resolution out of
`_hermetic_erlang_repository_impl` into a pure
`resolve_host(repository_ctx)` function, using the same technique
already used for `gleam_host`'s `host_platform` (a plain fake struct
standing in for a real `repository_ctx`, no Bazel machinery needed).
- Adds `test/unit/hermetic_erlang_repository/` covering every OS x arch
combination (including both the raw and erlef-normalized arch spellings
on both Linux and macOS) plus the two failure cases (`fail()` on
unsupported OS/arch), so a repeat of that bug class fails fast locally
instead of needing a real CI run on the affected platform to notice.
- No behavior change: `resolve_host` reproduces the exact same
URL/checksum-key/constraint values the inlined per-branch logic did
before.

## Test plan
- [x] `buildifier -mode=diff`/`-lint=warn` clean on all changed/added
files (buildifier built locally via `go install` since Bazel itself is
unavailable in this environment)
- [x] Relevant pre-commit hooks pass (`end-of-file-fixer`, `typos`)
- [ ] CI: `bazel test //test/...` (new
`hermetic_erlang_repository_test_suite`)
- [ ] CI: existing `examples/hermetic_erlang` build/test still passes on
both Linux and macOS (proves the refactor didn't change runtime
behavior)


---
_Generated by [Claude
Code](https://claude.ai/code/session_011YhQ33xAXjUGpy7BxwcEhg)_

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants