ci: accelerate Subtensor CI with R2 sccache and tiered Fireactions pools#2857
ci: accelerate Subtensor CI with R2 sccache and tiered Fireactions pools#2857UnArbosFive wants to merge 28 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🛡️ AI Review — Skeptic (security review)VERDICT: SAFE VERY HIGH account-age scrutiny, mitigated by repository admin permission and matching authorship; no Gittensor association found; stacked PR targets bittensor-core-exploration. Static review covered R2 credentials, writer-event restrictions, untrusted PR execution, production artifact publication, runner routing, and review-instruction integrity. No review-instruction files were modified. FindingsNo findings. Prior-comment reconciliation
ConclusionWriter credentials remain restricted to trusted 🔍 AI Review — Auditor (domain review)VERDICT: 👍 Gittensor association UNKNOWN; the author’s account is new but has repository admin permission, so workflow routing received direct scrutiny. The implementation otherwise matches the substantive description, and overlapping PRs are upstream/dependent work rather than competing implementations. No runtime change or spec-version bump applies. Description discrepancy: the body says the PR remains draft, while current metadata reports it as ready for review. No build or test was run because the remaining issue is statically evident. Findings
Prior-comment reconciliation
ConclusionApprove after applying the runner-label correction so both release production-binary builds retain the documented 16-core lane. 📜 Previous run (superseded)
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build uses the general 8-core pool
The PR promises to reserve fireactions-turbo for production-binary builds, and the equivalent PR validation build does so, but the release workflow routes this CPU-heavy binary build to fireactions-turbo-8. That contradicts the stated allocation and can lengthen the release critical path. Keep only the image validation and publishing jobs on the 8-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR says production-binary builds retain the reserved 16-core lane, and the equivalent PR build correctly uses fireactions-turbo. This release job performs the same cross-compiled production build but routes both architectures to the general 8-core pool. Keep the expensive compilation on the documented reserved lane; the image and publish jobs can remain on fireactions-turbo-8.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR says production-binary builds retain the 16-core fireactions-turbo lane, but this production binary matrix is routed to the general 8-core pool. This is the cross-compilation-heavy build the capacity policy explicitly reserves for 16 cores.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that production-binary builds retain the reserved 16-core lane, but this release job routes both architecture builds to the general 8-core pool. This contradicts the documented capacity policy and can lengthen release builds or contend with general workloads. Route this compilation job to fireactions-turbo-16; the image-validation and publishing jobs can remain on the 8-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
UnArbosFive
changed the title
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
c157fb5 to
db0d556
Compare
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that both amd64 and arm64 production-binary builds retain 16 cores, but this release matrix still runs them on fireactions-turbo-8. This also diverges from the equivalent PR validation job in check-docker.yml. Route this compile-heavy job to the 16-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that both amd64 and arm64 production binary builds retain 16 cores, and the equivalent PR validation job uses fireactions-turbo-16. This release matrix still selects the 8-core pool, contradicting that allocation and potentially doubling latency on the release-critical builds.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description says the amd64 and arm64 production-binary builds retain the 16-core lane, but this release workflow routes both builds to the 8-core pool. This also differs from the equivalent PR validation build in check-docker.yml. Route this compile-heavy matrix to fireactions-turbo-16.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
This production-binary matrix still runs on the 8-core general lane, contrary to the PR’s stated allocation of the amd64 and arm64 production builds to 16 cores. It also differs from the equivalent matrix in check-docker.yml. Route this compile-heavy release job to the 16-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
163e63a to
c5dd58b
Compare
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description explicitly assigns both amd64 and arm64 production binary builds to the 16-core lane, but this release matrix still uses fireactions-turbo-8. Route it to the dedicated compile pool to preserve the documented capacity plan and avoid making release builds the exception.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| local output_file="$2" | ||
|
|
||
| case "${GITHUB_EVENT_NAME:-}:${GITHUB_REF:-}" in | ||
| push:refs/heads/main|schedule:refs/heads/main|workflow_dispatch:refs/heads/main|workflow_dispatch:refs/heads/codex/fireactions-runner-density-experiment) |
There was a problem hiding this comment.
[HIGH] Branch exception exposes the shared-cache writer credential
This permits a workflow dispatched from the PR branch to receive the persistent R2 writer keys. The checked-out branch controls the composite action and every subsequent build step, so approved execution can exfiltrate those keys or poison shared compiler-cache objects later consumed by trusted builds. Environment approval does not constrain what the approved branch code does, and this escape hatch remains reachable after merge. Restrict writer mode to main; benchmark this branch with reader credentials or an isolated disposable bucket.
| push:refs/heads/main|schedule:refs/heads/main|workflow_dispatch:refs/heads/main|workflow_dispatch:refs/heads/codex/fireactions-runner-density-experiment) | |
| push:refs/heads/main|schedule:refs/heads/main|workflow_dispatch:refs/heads/main) |
|
🔄 AI review updated — Skeptic: VULNERABLE |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description explicitly reserves 16 cores for both amd64 and arm64 production binary builds, but this release matrix still uses the 8-core pool. This contradicts the documented runner-density design and can double the compile load assigned to an 8-core runner.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
a3fa449 to
c5dd58b
Compare
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description says the amd64/arm64 production-binary builds retain 16 cores, but this release matrix routes both expensive builds to the 8-core pool. This also differs from the PR validation workflow’s production build. Route this job to fireactions-turbo-16 so the release path matches the documented capacity plan.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
1 participant
Summary
This is the unified Subtensor CI acceleration change. It supersedes #2855 and combines:
sccachefor Rust compilationfireactions-turbo-16andfireactions-turbo-8runner lanesShared compiler cache
sccachecentrally through the shared Rust setup pathsccache --show-statsfor cache observabilityRunner-density experiment
The ten-machine 9950X experiment is intentionally fixed at:
fireactions-turbo-16fireactions-turbo-8The four PR-critical compile jobs retain 16 cores: cargo test, runtime artifact build, and the amd64/arm64 production binary builds. General Rust checks, Docker packaging, SDK/docs, and e2e build work use the 8-core pool.
Try-runtime remains on
fireactions-tryruntime: historical AX102 runs were roughly twice as fast as BHS despite fewer cores, and direct endpoint probes showed materially lower archive-RPC latency. The former Finney-specific label is folded into the same interchangeable AX102 pool.Observed capacity
On the representative PR run:
fireactions-turbo-16: peak 4 concurrent jobs against 8 planned runnersfireactions-turbo-8: peak 15 concurrent jobs against 24 planned runnersThis is sized for one full PR without runner queuing. Two perfectly aligned PRs can briefly queue up to six 8-core jobs; that contention is part of the density experiment.
Infrastructure dependency
The R2 bucket, MMDS reader configuration, writer secrets, and matching Fireactions pools must remain deployed from the infrastructure repository. Excess compatibility capacity is intentionally excluded from the designed 8/24 runner totals so it can be retired without reducing the experiment pool.
Rollout status
This PR remains draft until the final live host swaps drain and the combined cache + runner result receives a final review.