Redesign the build summary and rewrite the README - #10
Merged
Conversation
Records the approved design before implementation: - Cache v5: shared bucket per (toolchain, LTO, kernel source) with weekly rotation, no moving commit SHAs in the primary key, single elected cache writer per matrix, 2 GiB ccache cap and a bounded ThinLTO cache policy. - Pipeline: single SUSFS clone, no duplicate preset resolve, background disk cleanup, trimmed apt plus a toolchain completeness check, metadata-only artifact for the aggregate job, reproducible build timestamps. - Simplification of the two summary generators and package-anykernel. - Docs rewrite on a six-emoji vocabulary enforced by a test. Includes the evidence for each claim and the WildKernels settings that are deliberately not copied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cache was structurally broken rather than merely suboptimal. GitHub allows 10 GB of Actions cache per repository (LRU, 7-day idle expiry) and four things combined to blow it on every run: - the v4 ccache key embedded source/manager/susfs commits, all moving branches, so the primary key missed every run and uploaded a fresh 4-6 GiB entry; - the ThinLTO cache had no --thinlto-cache-policy, so LLVM's default of 75% of free disk applied and a single save could reach tens of GB; - five matrix managers each saved their own copy against the same budget; - compression_level=6 slowed compilation for no benefit, and with no sloppiness ccache refused direct mode because a fresh clone gives every include a new mtime. Cache - New scripts/generate-cache-keys.sh: one bucket per (toolchain, LTO, kernel source), rotated weekly, no run-varying value in the key. Restore drops the week, then the code hash, and stops - a bucket-wide fallback would pull another source's cache for near-zero hits. - code_hash narrowed to build-kernel.sh + marble.env, so adding a preset no longer invalidates every cache. - ccache capped at 2 GiB, compression level 1, inode_cache on, and sloppiness set so freshly cloned trees still get direct hits. - ThinLTO bounded at cache_size_bytes=1g:prune_after=168h. - generate-build-matrix.sh elects exactly one cache_writer; build-core gates both save steps on it. - Hit rate and direct rate are computed and reported, so the next run is the first measurable one. Deliberately not copied from WildKernels: CCACHE_FILE_CLONE (disables compression), CCACHE_BASEDIR (documented as brittle, breaks dep files), CCACHE_MAXSIZE=12G, and CCACHE_DEPEND (documented lower hit rate). Pipeline - susfs4ksu was cloned twice, the second time without --filter; now once. - resolve-toolchain.sh no longer re-runs resolve-kernel-source.sh. - Disk cleanup relocates then purges in the background instead of blocking. - apt no longer installs clang/llvm/lld; the pinned toolchain provides them and a completeness check now asserts every llvm-* binary LLVM=1 needs. - New marble-meta-* artifact so the aggregate job stops downloading every ZIP. - eval "$(python3 ...)" no longer swallows failures. - Preflight runs on every branch. - The manager setup.sh digest is recorded before it is executed. - SOURCE_DATE_EPOCH/KBUILD_BUILD_TIMESTAMP derive from the source commit, so identical inputs produce a byte-identical Image. uname -a now shows the source commit date; the real build time stays in build-info and the banner. - build-info.txt writer extracted from inline YAML into a script. Simplification - Both summary generators onto shared summary-common.sh emitters, reading each build-info.txt once instead of six greps per manager (-238 lines). - package-anykernel.sh caller-env dance, read-manager-version.sh sed branches. Docs - README 428 -> 294 lines; six-emoji vocabulary (device, build, manager, SUSFS, artifacts, warning) enforced by tests/test-emoji-vocabulary.sh. - ARCHITECTURE section 11 rewritten around the 10 GB constraint; versions.md gains a cache budget table; the matrix summary preview is now generated from the real generator so it cannot drift. Tests - New test-cache-policy.sh and test-emoji-vocabulary.sh. - test-build-info-json.sh now drives the real writers end to end and asserts resolved-refs.env stays source-safe. - 20/20 tests, shellcheck, bash -n, and actionlint pass. Speedup is not quantified: today's warm runs are effectively cold, so there is no honest baseline to measure against until this lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`bash f1 f2` executes f1 and passes the remaining paths as positional arguments. Preflight used `bash tests/test-*.sh`, so it has only ever run tests/test-banner-text.sh, and `bash -n scripts/*.sh scripts/lib/*.sh tests/*.sh` only ever syntax-checked scripts/apply-kernel-source-patches.sh. Both reported success regardless of the other files. Visible in run 30202513721, where the policy step printed a single line. Both now loop. The policy loop unsets the related env vars, matching what build-core.yml and build-matrix.yml already do, so ambient CI env cannot fake a pass, and it reports every failing test rather than stopping at the first. test-workflow-policy.sh asserted the broken form, which is how it survived; it now asserts the looping form and rejects the single-invocation one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The combined summary opened with configuration tables, so the two things a reader actually wants — did it pass, and which ZIP do I flash — were buried below the fold. Restructure the generator to answer those first: - New "Result" section: pass count plus a per-manager version/code/SUSFS/size table, stated as a superset of the existing Managers table rather than a replacement, so every downstream assertion still holds. - New "Which ZIP do I flash?" section. Three ZIPs in one artifact set is the most confusing moment for a user and nothing said they are alternatives, not components of a set. Maps each manager to its ZIP and its manager app. - New "Reproduce this build" section: every resolved commit in one copyable YAML block plus the `gh attestation verify` invocation. These values were already in build-info.txt but scattered across the document. - Surface build-info fields the generator was discarding: defconfig and defconfig_mode, quality_label, runner image and free disk. All emitted conditionally so older artifacts still render. - Cache table gains a hit-rate column via the existing summary_format_ccache_hits helper, and per-manager ZIP name is added to the manager detail blocks. - Promote the flash warnings to GitHub alert callouts and lead the pre-flash checklist with ROM-family mismatch, the most common bootloop cause here. Section headings, table shapes and the CI-cache markers are unchanged, so tests/test-matrix-summary.sh passes untouched and the cache section still strips cleanly out of release notes. Regenerate docs/previews/matrix-summary-preview.md from the real generator with fixture data instead of hand-editing it; sizes and checksums are placeholders since fixtures carry dummy ZIPs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Restructure the README so a newcomer reaches a working build sooner and the reference material sits behind disclosure instead of in front of it. - Lead with a quick start and a recommended first build (melt + build_none) rather than the full input reference. - Add Mermaid diagrams for the matrix job graph and the build-core stage flow; GitHub renders these natively. - Move toolchain, LTO, free-runner limits and the full input table into <details> blocks so the linear read stays short. - Use GitHub alert callouts for the safety-critical points, and give ROM-family mismatch its own warning next to the preset table since it is the main bootloop risk. - Annotate the ZIP naming scheme inline and explain why LTO and toolchain are deliberately absent from the filename. - Explain the scripts/-over-composite-actions choice, which was previously only documented in ARCHITECTURE.md. Correct the workflow input table: `toolchain` defaults to `auto`, not `android-r416183b`. The old table contradicted build-matrix.yml and the seven other places in the same file that already recommended `auto`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mohdakil2426
added a commit
that referenced
this pull request
Jul 26, 2026
actionlint runs shellcheck over embedded run: blocks, and flagged the unquoted $(find patches/kernel-sources ...) inside the ccache key hash. The word splitting is deliberate — each patch file has to arrive at sha256sum as a separate argument — so annotate it rather than change the expansion. This is pre-existing on main, not introduced by this branch: 927dac9 restored build-core.yml to the 600d978 form, which carries this line, and that revert was pushed with [skip ci] so no preflight ran against it. The first CI run since is PR #10, which is where it surfaced. Comment-only; the computed cache key is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
actionlint runs shellcheck over embedded run: blocks, and flagged the unquoted $(find patches/kernel-sources ...) inside the ccache key hash. The word splitting is deliberate — each patch file has to arrive at sha256sum as a separate argument — so annotate it rather than change the expansion. This is pre-existing on main, not introduced by this branch: 927dac9 restored build-core.yml to the 600d978 form, which carries this line, and that revert commit asked CI to be skipped, so no preflight ever ran against it. The first CI run since is PR #10, which is where it surfaced. Comment-only; the computed cache key is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mohdakil2426
force-pushed
the
feature/pipeline-cache-docs-overhaul
branch
from
July 26, 2026 21:22
09ded58 to
cccde25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Two presentation changes, plus one small CI unblock. No pipeline, cache, or build behaviour is altered.
1. The combined matrix build summary (
scripts/generate-matrix-summary.sh)The summary opened with configuration tables, so the two things a reader actually wants — did it pass and which ZIP do I flash — were below the fold. It now leads with those.
gh attestation verify. These values were already inbuild-info.txt, just scattered.defconfig/defconfig_mode,quality_label, runner image, free disk before build. All conditional, so older artifacts still render.summary_format_ccache_hitshelper.Built as a superset: every existing heading, table shape and CI-cache marker is unchanged, so
tests/test-matrix-summary.shpasses without modification and the cache section still strips cleanly from release notes.docs/previews/matrix-summary-preview.mdis regenerated from the real generator with fixture data rather than hand-edited. Sizes and checksums are placeholders because the fixtures carry dummy ZIPs.2. README
Restructured around the build path: quick start first, reference material behind
<details>. Adds Mermaid diagrams for the matrix job graph and the build-core stages.One factual fix: the input table listed
toolchainas defaulting toandroid-r416183b.build-matrix.ymlsetsdefault: auto, and seven other lines in the same README already recommendedauto.3. CI unblock (
build-core.yml) — comment onlyPreflight was failing on a pre-existing
SC2046in the ccache-key step, where$(find patches/kernel-sources ...)is left unquoted on purpose so each patch file reachessha256sumas its own argument. Annotated with a# shellcheck disable=SC2046directive rather than changing the expansion.This is not introduced by this branch.
origin/maincarries the identical line (build-core.yml:283).927dac9restoredbuild-core.ymlto the600d978form and was pushed with CI skipped, so no preflight ever ran against it — this PR is simply the first run since. The computed cache key is unchanged.If you would rather fix it by restoring the v5
scripts/generate-cache-keys.shapproach, drop that commit and this PR still stands on its own.Verification
shellcheck -e SC1090,SC1091,SC2016,SC2153,SC2154clean on the modified generator.git diff --checkclean.summary_strip_cache_sectionstill removes the entire Cache block from release notes while the new sections survive.Note on the commit list
This branch already carried
023bc44 → 927dac9(the v5 cache overhaul and its revert) before these commits. Since927dac9restored the tree to600d978, the net file diff againstmainis only the four files below — the earlier commits cancel out.🤖 Generated with Claude Code