feat(self-packaging #66): fat / universal macOS binary support#70
Merged
Conversation
Before this change, `LocateFlapiSectionInBuffer` returned `nullopt` for any input whose magic wasn't `MH_MAGIC_64`. Universal binaries (`FAT_MAGIC` / `FAT_MAGIC_64`) -- the format produced by `lipo -create` and consumed by Homebrew-style formulas that ship both arches -- therefore couldn't be packed or self-inspected. A universal binary is a small big-endian fat header followed by N thin Mach-O slices at distinct file offsets. The `section_64.offset` read inside a slice is relative to the slice's base, not the fat file -- both `OverwriteFlapiSection` (write side) and `LocateBundleInRange` (read side) treat the returned offset as an absolute file offset, so a naive accept-fat patch would have written to the wrong byte address. This PR: - Adds `ReadU32BE` / `ReadU64BE` -- fat headers are big-endian on disk regardless of host endianness. - Adds `ParseFatHeader` (namespace-private). Walks `fat_arch[]` (20-byte records for `FAT_MAGIC/CIGAM`, 32-byte for the `_64` variants). Selection rule: first slice whose cputype matches the host arch (compile-time, via `__aarch64__` / `__x86_64__`), else the first slice. Rejects malformed `nfat_arch` (0 or > 64) and slice extents past EOF. - Splits `LocateFlapiSectionInBuffer` into an outer dispatch + an inner `LocateFlapiSectionAt(buffer, base)` overload. On fat input the outer call parses the fat header, picks a slice, and recurses via the inner overload with the slice's absolute offset as base. The inner overload adds base to whatever the load-cmd walker returns, so callers see an absolute file offset. - `IsMachOMagic` extended to recognise `FAT_MAGIC_64` / `FAT_CIGAM_64` in addition to the 32-bit fat variants it already accepted. - `OverwriteFlapiSection` is unchanged -- the new absolute-offset invariant makes its existing `seekp(file_offset)` land in the correct slice automatically. - Header doc updated to drop the "fat / universal not supported" caveat and document the slice-selection rule. Test fixture: new `BuildFatMachO(slices)` helper wraps any number of `BuildMachO64` outputs with a big-endian fat header + 4 KiB- aligned slice placement. Four new test cases (issue #66 acceptance): - single-slice fat: section located at the absolute offset (slice_offset + intra-slice section offset). - two-slice fat (arm64 + x86_64): parser picks the host-matching slice; deterministic per-host expected offset via `#ifdef`. - two PPC slices (host arch doesn't match either): parser falls back to the first slice and stays deterministic across exotic hosts. - OverwriteFlapiSection round-trip inside a slice: write a 7-byte payload through the located section, verify the bytes land at the returned absolute offset, and confirm a re-locate finds the same section. Test plan: - ctest: 642 / 642 pass (637 previous + 5 new -- 4 fat cases + the round-trip). - pytest test_self_packaging.py + test_self_packaging_http.py: 11 / 11 pass (no regression on the Linux EOCD tail-scan path, which is unaffected by this change). Closes #66.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LocateFlapiSectionInBufferrejected any input whose magic wasn'tMH_MAGIC_64, so universal binaries (FAT_MAGIC/FAT_MAGIC_64)-- the format produced by
lipo -createand consumed byHomebrew-style formulas that ship both arches -- couldn't be packed
or self-inspected.
flapi inforeported "no bundle";flapi packeither failed outright or appended the bundle past EOF, producing an
ad-hoc-signed binary that fails notarisation.
A universal binary is a small big-endian fat header followed by N
thin Mach-O slices at distinct file offsets. The
section_64.offsetfield inside a slice is relative to the slice's base, not the fat
file -- both
OverwriteFlapiSection(write side) andLocateBundleInRange(read side) treat the returned offset as anabsolute file offset, so a naive accept-fat patch would have written
to the wrong byte address.
This PR fixes both halves.
Changes (all in
src/macho_bundle.cpp+src/include/macho_bundle.hpp)ReadU32BE,ReadU64BE). Fat headers arebig-endian on disk regardless of host endianness, so Linux/macOS
little-endian hosts must byte-swap every multi-byte field.
ParseFatHeader(namespace-private). Walksfat_arch[]records(20 bytes for
FAT_MAGIC/CIGAM, 32 bytes for the_64variants).Selection rule: first slice whose
cputypematches the host arch(compile-time, via
__aarch64__/__x86_64__), else the firstslice. Rejects malformed
nfat_arch(0 or > 64) and slice extentspast EOF.
LocateFlapiSectionInBuffersplit into an outer dispatch + aninner
LocateFlapiSectionAt(buffer, base)overload. On fat inputthe outer call parses the fat header, picks a slice, and recurses
via the inner overload with the slice's absolute offset as base.
The inner overload adds
baseto whatever the load-cmd walkerreturns, so all callers see an absolute file offset.
IsMachOMagicextended to recogniseFAT_MAGIC_64/FAT_CIGAM_64in addition to the 32-bit fat variants it already accepted.
OverwriteFlapiSectionis unchanged -- the newabsolute-offset invariant makes its existing
seekp(file_offset)land in the correct slice automatically.This is the whole reason for the split + base-addition design.
documents the slice-selection rule.
32-bit Mach-O (
MH_MAGIC/MH_CIGAM) is still rejected with aninline comment that we don't ship 32-bit artifacts.
Test fixture & cases (
test/cpp/macho_bundle_test.cpp)New
BuildFatMachO(slices)helper wraps any number ofBuildMachO64outputs with a big-endian fat header + 4 KiB-aligned slice placement.
The existing helpers (
BuildMachO64,WriteU32LE,SectionSpec)are reused unchanged.
Four new test cases:
(slice_offset + intra-slice section offset).
slice; deterministic per-host expected offset via
#ifdef.to the first slice and stays deterministic across exotic hosts.
7-byte payload through the located section, verify the bytes land
at the returned absolute offset, and confirm a re-locate finds the
same section -- proves the read+write absolute-offset invariant.
Test plan
cmake --build build/release --config Releasecleanctest-- 642 / 642 pass (637 previous + 5 new)pytest test_self_packaging.py test_self_packaging_http.py -v-- 11 / 11 pass (no regression on the Linux EOCD tail-scan path,
which is unaffected)
lipo -createan arm64 + x86_64 flapi, run./flapi-universal pack --in examples --out fat-bundled, then./fat-bundled infoto confirm the bundle is found in the hostslice. The macOS CI builder will exercise the existing macOS
pack-smoke against thin binaries; this PR doesn't add an automated
Mac path for fat binaries.
Notes
clang-tidyflagged unrelated pre-existing warnings in the touchedfiles (
std::endl, widening literal multiplications); leftuntouched per the no-incidental-cleanup convention.
Credential env-var audit + 12-factor inventory #64, ZIP64 read-side defensiveness in bundle_locator + archive_io #65 landed in PRs feat: 12-factor env vars FLAPI_PORT + FLAPI_HOST (#63) #67/feat(self-packaging #65): ZIP64-sentinel WARN log in bundle_locator #68/fix(self-packaging #62): bundled-mode HTTP serving end-to-end #69 plus a manual close earlier.
Closes #66.