fix(release,#339): bundle deps before erasing the RUNPATH that finds them

.github/workflows/ci.yml

@@ -75,8 +75,12 @@ jobs:
         with:
           submodules: recursive
 
+      # patchelf is for test-bundle-linux-runtime, which exercises the release
+      # packaging scripts. Without it that test SKIPs, and a skip is
+      # indistinguishable from a pass in the summary — which is how the release
+      # packaging went unobserved in the first place.
       - name: Install dependencies
-        run: sudo apt-get update && sudo apt-get install -y cmake ninja-build g++ libopenblas-dev
+        run: sudo apt-get update && sudo apt-get install -y cmake ninja-build g++ libopenblas-dev patchelf
 
       # Examples/server stay at their defaults: some tests link `common`, which
       # lives under examples/, so switching them off would break configure.

.github/workflows/release.yml

@@ -41,7 +41,12 @@ jobs:
         with:
           submodules: recursive
           ref: ${{ inputs.tag || github.ref }}
+      # A dry run (empty tag, dispatched from a branch) has no tag to match, so
+      # this compared VERSION against a branch name and failed every time —
+      # which made the run red and the one job you dispatched it for unreadable
+      # among the noise. There is nothing to verify when nothing is published.
       - name: Verify VERSION matches tag
+        if: inputs.tag != '' || startsWith(github.ref, 'refs/tags/')
         run: |
           tag_version="${{ inputs.tag || github.ref_name }}"
           tag_version="${tag_version#v}"
@@ -126,6 +131,12 @@ jobs:
           # base-system means the loader dies before main() and the process
           # exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64/
           cd release && tar czf crispasr-linux-x86_64.tar.gz crispasr-linux-x86_64/
 
       - uses: actions/upload-artifact@v4
@@ -225,6 +236,12 @@ jobs:
           # Then assert it: any DT_NEEDED that is neither bundled nor base-system
           # means the loader dies before main() and the process exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64-avx512
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64-avx512/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64-avx512/
           cd release && tar czf crispasr-linux-x86_64-avx512.tar.gz \
               crispasr-linux-x86_64-avx512/
 
@@ -302,6 +319,12 @@ jobs:
           # means the loader dies before main() and the process exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64-cuda \
             --allow 'libcuda.so.*' --allow 'libcudart.so.*' --allow 'libcublas*.so.*' --allow 'libnv*.so.*'
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64-cuda/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64-cuda/
           cd release && tar czf crispasr-linux-x86_64-cuda.tar.gz crispasr-linux-x86_64-cuda/
           ls -lh crispasr-linux-x86_64-cuda.tar.gz
 
@@ -378,6 +401,12 @@ jobs:
           # means the loader dies before main() and the process exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64-cuda13 \
             --allow 'libcuda.so.*' --allow 'libcudart.so.*' --allow 'libcublas*.so.*' --allow 'libnv*.so.*'
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64-cuda13/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64-cuda13/
           cd release && tar czf crispasr-linux-x86_64-cuda13.tar.gz crispasr-linux-x86_64-cuda13/
           ls -lh crispasr-linux-x86_64-cuda13.tar.gz
 
@@ -452,6 +481,12 @@ jobs:
           # means the loader dies before main() and the process exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64-hip \
             --allow 'libamdhip64.so.*' --allow 'librocblas.so.*' --allow 'libhsa*.so.*'
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64-hip/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64-hip/
           cd release && tar czf crispasr-linux-x86_64-hip.tar.gz crispasr-linux-x86_64-hip/
           ls -lh crispasr-linux-x86_64-hip.tar.gz
 
@@ -534,6 +569,13 @@ jobs:
           # because the gate only checked dependencies were PRESENT, never that
           # RUNPATH could find them. Fails the release rather than the consumer.
           bash tools/package-lib-bundle.sh "$OUT"
+          # And assert the result. verify-lib-bundle.sh (inside the script above)
+          # gates INTRA-bundle resolvability only — a dependency absent from the
+          # bundle entirely is outside what it asks, which is how the HIP bundle
+          # shipped needing an unbundled libomp.so (#341). This is the same gate
+          # the CLI tarballs have had since #296.
+          python3 scripts/check-bundled-deps.py "$OUT/lib" \
+            --allow 'libamdhip64.so.*' --allow 'librocblas.so.*' --allow 'libhsa*.so.*'
           cp include/crispasr.h "$OUT/include/"
           cp ggml/include/*.h   "$OUT/ggml/include/" 2>/dev/null || true
           cp LICENSE "$OUT/"
@@ -610,6 +652,12 @@ jobs:
           # means the loader dies before main() and the process exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-x86_64-vulkan \
             --allow 'libvulkan.so.*'
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-x86_64-vulkan/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-x86_64-vulkan/
           cd release && tar czf crispasr-linux-x86_64-vulkan.tar.gz crispasr-linux-x86_64-vulkan/
           ls -lh crispasr-linux-x86_64-vulkan.tar.gz
 
@@ -673,6 +721,12 @@ jobs:
           # base-system means the loader dies before main() and the process
           # exits 127 silently.
           python3 scripts/check-bundled-deps.py release/crispasr-linux-arm64
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these bundle
+          # third-party shared libraries (libgomp, and libomp on HIP) whose terms
+          # require the notice to travel with them.
+          cp LICENSE release/crispasr-linux-arm64/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-linux-arm64/
           cd release && tar czf crispasr-linux-arm64.tar.gz crispasr-linux-arm64/
 
       - uses: actions/upload-artifact@v4
@@ -744,10 +798,14 @@ jobs:
           cp build/ggml/src/libggml-cpu.so  "$PKG/" 2>/dev/null || true
           cp build/libcrispasr_helpers.so   "$PKG/"
 
-          # Patch RPATH to $ORIGIN so libs resolve siblings without LD_LIBRARY_PATH
-          for f in "$PKG"/*.so; do
-            patchelf --set-rpath '$ORIGIN' "$f" 2>/dev/null || true
-          done
+          # Bundle the external dependency closure and patch RPATH to $ORIGIN,
+          # in that order (CrispASR #339). The inline loop this replaces did only
+          # the RUNPATH half, so the shipped tarball needed libgomp.so.1 and
+          # libblas.so.3 from the user's machine and `import crispasr` died in
+          # the loader on any host without them.
+          bash scripts/bundle-linux-runtime.sh "$PKG"
+          # Then assert it — the gate the other Linux legs have had since #296.
+          python3 scripts/check-bundled-deps.py "$PKG"
 
           cd release && tar czf crispasr-python-linux-x86_64.tar.gz crispasr-python-linux-x86_64/
 
@@ -817,10 +875,14 @@ jobs:
           cp build/ggml/src/libggml-cpu.so  "$PKG/" 2>/dev/null || true
           cp build/libcrispasr_helpers.so   "$PKG/"
 
-          # RPATH = $ORIGIN so siblings resolve without LD_LIBRARY_PATH
-          for f in "$PKG"/*.so; do
-            patchelf --set-rpath '$ORIGIN' "$f" 2>/dev/null || true
-          done
+          # Bundle the external dependency closure and patch RPATH to $ORIGIN,
+          # in that order (CrispASR #339). The inline loop this replaces did only
+          # the RUNPATH half, so the shipped tarball needed libgomp.so.1 and
+          # libblas.so.3 from the user's machine and `import crispasr` died in
+          # the loader on any host without them.
+          bash scripts/bundle-linux-runtime.sh "$PKG"
+          # Then assert it — the gate the other Linux legs have had since #296.
+          python3 scripts/check-bundled-deps.py "$PKG"
 
           cd release && tar czf crispasr-python-linux-arm64.tar.gz crispasr-python-linux-arm64/
 
@@ -858,6 +920,12 @@ jobs:
           cp build/bin/crispasr release/crispasr-macos/
           cp build/bin/crispasr-quantize release/crispasr-macos/
           bash scripts/bundle-c2pa.sh release/crispasr-macos || true
+          # Ship the licence text with the binaries. Every Windows, Android and
+          # libcrispasr artifact already did; no CLI tarball did, and these carry
+          # bundled third-party libraries whose terms require the notice to
+          # travel with them.
+          cp LICENSE release/crispasr-macos/
+          cp THIRD_PARTY_NOTICES.txt release/crispasr-macos/
           cd release && tar czf crispasr-macos.tar.gz crispasr-macos/
 
       - uses: actions/upload-artifact@v4
@@ -1097,6 +1165,12 @@ jobs:
           # because the gate only checked dependencies were PRESENT, never that
           # RUNPATH could find them. Fails the release rather than the consumer.
           bash tools/package-lib-bundle.sh "$OUT"
+          # And assert the result. verify-lib-bundle.sh (inside the script above)
+          # gates INTRA-bundle resolvability only — a dependency absent from the
+          # bundle entirely is outside what it asks, which is how the HIP bundle
+          # shipped needing an unbundled libomp.so (#341). This is the same gate
+          # the CLI tarballs have had since #296.
+          python3 scripts/check-bundled-deps.py "$OUT/lib"
           # Headers so binding authors can extend the FFI without
           # vendoring the whole CrispASR repo.
           cp include/crispasr.h "$OUT/include/"
@@ -1169,6 +1243,12 @@ jobs:
           # because the gate only checked dependencies were PRESENT, never that
           # RUNPATH could find them. Fails the release rather than the consumer.
           bash tools/package-lib-bundle.sh "$OUT"
+          # And assert the result. verify-lib-bundle.sh (inside the script above)
+          # gates INTRA-bundle resolvability only — a dependency absent from the
+          # bundle entirely is outside what it asks, which is how the HIP bundle
+          # shipped needing an unbundled libomp.so (#341). This is the same gate
+          # the CLI tarballs have had since #296.
+          python3 scripts/check-bundled-deps.py "$OUT/lib"
           cp include/crispasr.h "$OUT/include/"
           cp ggml/include/*.h   "$OUT/ggml/include/" 2>/dev/null || true
           cp LICENSE "$OUT/"
@@ -1243,6 +1323,12 @@ jobs:
           # because the gate only checked dependencies were PRESENT, never that
           # RUNPATH could find them. Fails the release rather than the consumer.
           bash tools/package-lib-bundle.sh "$OUT"
+          # And assert the result. verify-lib-bundle.sh (inside the script above)
+          # gates INTRA-bundle resolvability only — a dependency absent from the
+          # bundle entirely is outside what it asks, which is how the HIP bundle
+          # shipped needing an unbundled libomp.so (#341). This is the same gate
+          # the CLI tarballs have had since #296.
+          python3 scripts/check-bundled-deps.py "$OUT/lib"
           # Headers so binding authors can extend the FFI without
           # vendoring the whole CrispASR repo.
           cp include/crispasr.h "$OUT/include/"
@@ -1502,6 +1588,13 @@ jobs:
           # because the gate only checked dependencies were PRESENT, never that
           # RUNPATH could find them. Fails the release rather than the consumer.
           bash tools/package-lib-bundle.sh "$OUT"
+          # And assert the result. verify-lib-bundle.sh (inside the script above)
+          # gates INTRA-bundle resolvability only — a dependency absent from the
+          # bundle entirely is outside what it asks, which is how the HIP bundle
+          # shipped needing an unbundled libomp.so (#341). This is the same gate
+          # the CLI tarballs have had since #296.
+          python3 scripts/check-bundled-deps.py "$OUT/lib" \
+            --allow 'libcuda.so.*' --allow 'libcudart.so.*' --allow 'libcublas*.so.*' --allow 'libnv*.so.*'
           cp include/crispasr.h "$OUT/include/"
           cp ggml/include/*.h   "$OUT/ggml/include/" 2>/dev/null || true
           cp LICENSE "$OUT/"

LEARNINGS.md

@@ -11,12 +11,69 @@ If a lesson is still "live" (affects current work), it's linked from
 ---
 
 <!-- BEGIN INDEX POINTER — generated by tools/gen-learnings-index.py -->
-**Do not read this file top to bottom** — it is ~800 KB across 275 sections.
+**Do not read this file top to bottom** — it is ~800 KB across 276 sections.
 The generated topic + per-model index is [`docs/LEARNINGS-INDEX.md`](docs/LEARNINGS-INDEX.md).
 Find the lesson there, then jump with `grep -n "<heading text>" LEARNINGS.md`.
 Regenerate both with `python tools/gen-learnings-index.py` after adding a section.
 <!-- END INDEX POINTER -->
 
+## A packaging step that destroys the state a later step reads — and scripts that only run in a release job (#339 HIP, 2026-08-10)
+
+`bundle-linux-runtime.sh` makes a staged Linux directory self-contained in two
+moves: rewrite `RUNPATH` to `$ORIGIN`, and copy in the non-system dependency
+closure. It did them in that order for two releases, and the order is the bug.
+
+**1. `ldd` answers through the RUNPATH you just erased.** Resolution is not a
+property of the binary alone. Rewriting `RUNPATH` first turns every dependency
+that was reachable *only* that way into `=> not found`, so the copy loop —
+`ldd … | awk '/=>/ {print $3}' | grep '^/'` — dropped them along with the blank
+lines. The libraries most in need of bundling are precisely the ones in
+non-default directories, i.e. precisely the ones this loses. **If step A mutates
+the state step B queries, B goes first**, and this is worth stating in the
+script's header rather than leaving as an ordering accident.
+
+The concrete casualty: ROCm's clang links OpenMP against LLVM's `libomp.so` in
+`/opt/rocm/lib/llvm/lib`. gcc's `libgomp.so.1` is in the default loader path, so
+six Linux legs worked and only HIP failed — a one-leg failure that looks
+GPU-specific and is actually generic.
+
+Corollary for the copy itself: **scan a copied library at its ORIGINAL path**,
+not at the staged copy. A `$ORIGIN`-relative RUNPATH resolves somewhere else
+once the file has moved, so `ldd` on the copy answers a different question.
+
+**2. `grep '^/'` is a filter that eats its own error message.** `ldd` reports a
+failure as `libfoo.so => not found` — a line with no path, indistinguishable
+from noise to a path filter. The script then printed `rpaths normalised, 0
+librar(ies) bundled`: a success line, with a count, over a dropped dependency.
+A count of what was done is not a statement about what was missed. Anything the
+resolver could not resolve is now fatal **and named** — with the same exclusion
+list the copy loop uses, since `libcuda.so.1` is legitimately absent from a
+driverless CI runner and would otherwise fail every CUDA leg.
+
+**3. The log printed the answer and it read as progress.** The failing run
+contains `rpath crispasr: '$ORIGIN:…:/opt/rocm-6.3.0/lib/llvm/lib:…' ->
+'$ORIGIN'`. The directory holding the missing library is inside the string being
+discarded, one line above the count that should have been non-zero. Both lines
+describe work completed. **When a step reports what it did rather than what it
+found, its output cannot be scanned for a problem** — you have to know the
+expected value to notice a wrong one.
+
+**4. A script that only ever runs inside a release job cannot be observed
+without publishing a release.** That is why two defects in these packaging
+scripts shipped: v0.8.18's unloadable lib bundles, and this. The fix is not more
+care at release time — it is a test in the ordinary tier. `cc`, a private
+directory and `-Wl,-rpath` reproduce the exact condition with no ROCm, no GPU
+and no release; the acceptance check is the staged binary *running* with its
+build-time libdir deleted, not the file being present (presence is not
+resolvability — v0.8.18 shipped bundles with every dependency beside them and
+one unreachable). Red-verify against the shipped script before trusting the
+green: the control printed `0 librar(ies) bundled` and failed, which is the
+whole defect in one line.
+
+⚠ And install the tool the test needs on the CI job — `patchelf` was absent, and
+a ctest SKIP reads exactly like a PASS in the summary. A gate that cannot run is
+a gate that cannot go red.
+
 ## A watermarked TTS output cannot be A/B'd with `cmp` — and whisper LID is not an accent metric (omnivoice, 2026-08-07)
 
 Four transferable things from SubtitleEdit-13273, where OmniVoice's target-language