ci: the shared workflows themselves must never use a GitHub-hosted runner

[ywatanabe1989]

Operator mandate (2026-07-14): zero GitHub-hosted runners across the scitex ecosystem, no exceptions. 「PR用のテストとgithub側のランナーというのは本当にもう一切使わないでください」

The SSOT was the bug

This repo is about to become the single source of truth that ~68 ecosystem repos consume as thin callers. Before this PR, two of its six reusable workflows still declared runs-on: ubuntu-latest:

workflow	job	before	after
auto-merge-to-develop.yml	automerge	ubuntu-latest	[self-hosted, Linux, X64, spartan-cpu]
cla.yml	owner-bypass	ubuntu-latest	[self-hosted, Linux, X64, spartan-cpu]
cla.yml	CLAssistant	ubuntu-latest	[self-hosted, Linux, X64, spartan-cpu]

The other four (import-smoke, pytest-matrix, quality-audit, rtd-sphinx-build) were already correct.

Migrating the ecosystem onto these workflows unfixed would have propagated the violation ~68 times and called it "unified." A violation that looks compliant is worse than 68 honest ones, because nothing downstream would ever flag it. Hence this lands first, before any consumer repo is touched.

Why the org pool, not per-repo labels

runs-on targets the scitex-ai org runner pool (spartan-cpu-org-01, spartan-cpu-org-02, spartan-pooled-cpu-01 — all online, label set [self-hosted, Linux, X64, spartan-cpu]). A per-repo runner label cannot work for 68 consumers, and pooled org-level runners are the standing preference (fewer machines, better utilisation, fewer surfaces to drift).

Verification

• All 6 workflows parse (yaml.safe_load).
• Zero hosted-runner references remain in the repo (rg ubuntu-latest|ubuntu-2x|macos-|windows- → no matches).
• Diff is runs-on:-only; no on:/permissions:/step logic touched.

Known follow-up (raised with scitex-hpc, not a blocker for this PR)

auto-merge-to-develop and cla shell out to the gh CLI, which is preinstalled on ubuntu-latest but is not guaranteed on the Spartan runner image. Tracked on the scitex-hpc card along with org-pool capacity (3 runners for 68 repos) and a live shared-venv contamination bug (scitex-stats --version prints scitex-dev 0.29.0 on the runner). Per the mandate, if Spartan cannot run something we fix Spartan — we never fall back to a hosted runner.

[ywatanabe1989]

DO NOT MERGE YET — blocked on the org runner pool actually dispatching

This diff is correct, but merging it today would cause a silent org-wide outage. I probed the runner before trusting it, and the result changes the sequencing.

What I measured

I pushed a temporary probe workflow requesting exactly what this PR targets — runs-on: [self-hosted, Linux, X64, spartan-cpu] — on this branch, and watched it:

run 29312674047  created 2026-07-14T06:52:09Z
status = queued        (10+ minutes)
runner = NONE ASSIGNED

Meanwhile the org pool reports itself perfectly healthy:

spartan-cpu-org-01     status=online  busy=false  labels=self-hosted,Linux,X64,spartan-cpu
spartan-cpu-org-02     status=online  busy=false  labels=self-hosted,Linux,X64,spartan-cpu
spartan-pooled-cpu-01  status=online  busy=false  labels=self-hosted,Linux,X64,spartan-cpu,scitex-ci

Group membership: spartan-pooled-cpu-01 is in Default (id=1); the other two are in the custom spartan-cpu group (id=3). Both groups are visibility=all, Actions are enabled on this repo, and the labels are a superset of the request. A matching, idle, online runner exists in the Default group and the job was still never dispatched.

For contrast, the runner that demonstrably does execute jobs is the repo-level one: spartan-cpu-scitex-stats-01 (group=Default) ran scitex-stats’ pytest matrix. The org-level pool has not been observed to execute anything.

Org plan is free (1 seat) — custom runner groups are a Team/Enterprise feature, which is my leading hypothesis for the non-dispatch, but the mechanism is scitex-hpc’s call. The observation is not in doubt.

Why this blocks the merge

auto-merge-to-develop and cla are the only two shared workflows that shell out to gh, and they are load-bearing: auto-merge lands PRs across the fleet, CLA gates external contributions. If this merges while the org pool does not dispatch, both hang in queued forever rather than failing loudly — no red X, just PR automation quietly stopping org-wide. That is worse than the violation it fixes.

(Separately: none of the four workflows already on Spartan invoke gh, so gh-on-the-runner is also still unproven. The probe existed to answer that too, and never got to run.)

Sequencing

1. scitex-hpc: make the org pool actually dispatch (card ci-spartan-capacity-and-runner-env), and confirm gh is on the runner image.
2. Re-run the probe; see it get a runner and print GH_PRESENT=yes.
3. Then merge this, then pilot, then batch the rollout.

Per the mandate — 「スパルタンがうまく動かない場合は我々の問題です。github runner に決して逃げないでください」 — the answer here is to fix Spartan, not to revert these three jobs to ubuntu-latest. This PR stays open and correct until the runners can take it.

[ywatanabe1989]

Dispatch is PROVEN — the gate on this PR is cleared

This PR was held until the org runner pool was shown to actually dispatch. It does. Verified live today, not inferred from the runners API.

Method. scitex-config has zero repo-level runners, so any runner that picks up its jobs must come from the org pool. Pushed one commit with two jobs differing only in the final label — a controlled A/B, not a single observation:

job	label	result
A	spartan-cpu (group 3, allows_public_repositories: true)	assigned in ~3s to spartan-cpu-org-02, group spartan-cpu — completed success
B	scitex-ci (group 1 / Default, still false)	queued 11m54s, runner: null, never assigned — cancelled

Same repo, same push, same instant. A ran; B never got a runner. That confirms both the exclusion mechanism and the fix.

Toolchain on the runner (this PR migrates auto-merge-to-develop.yml + cla.yml, which shell out to gh):

• gh 2.40.1 — present. The gh-dependent workflows are safe to migrate.
• git 2.52.0, nproc 1, host spartan-bm155.hpc.unimelb.edu.au.
• System python3 is 3.9.25, but that does not matter: setup-python resolves from a pre-seeded tool cache, and the reusable pytest-matrix installs into a uv venv.

End-to-end pilot. scitex-config migrated onto the reusable pytest-matrix (scitex-ai/scitex-config#23) and went genuinely green on the org pool — 185 passed, 1 skipped, 90% coverage, across py3.11/3.12/3.13 on spartan-cpu-org-01/-02. Not just a green check: the pytest summary was read from the logs.

---

One blocker remains, and it is in this PRs blast radius

spartan-pooled-cpu-01 is the only org runner carrying the scitex-ci label, and it sits in the Default group, which is still allows_public_repositories: false. Job B above is the proof that this is live.

This is currently masked: ~65 of 71 repos have their own repo-level runner (which carries scitex-ci and is not subject to the org group gate). But the moment a repo has no repo-level runner — which is the stated direction ("prefer pooled, org-level runners over one-per-repo") — anything still targeting scitex-ci queues forever, silently. That default is baked into the pypi-publish template as the runs-on fallback:

runs-on: ${{ fromJSON(vars.CI_RUNS_ON || [self-hosted,Linux,X64,scitex-ci]) }}

Fix is one call — put the pooled runner in the group that allows public repos:

gh api --method PUT /orgs/scitex-ai/actions/runner-groups/3/runners/348

The six reusable workflows here all correctly target spartan-cpu, so this PR is not itself exposed — but any repo migrated onto the pooled model while still defaulting to scitex-ci will hang with no error.

[ywatanabe1989]

scitex-ci now dispatches too — the last known blocker is cleared

Re-ran the exact negative control that failed before, on the same repo (scitex-config, zero repo-level runners, so only the org pool can serve it), after runner 348 (spartan-pooled-cpu-01 — the sole holder of the scitex-ci label) was moved out of the Default group into group 3.

	before the group move	after
scitex-ci	queued 11m54s, runner: null, never assigned	assigned in 3s → spartan-pooled-cpu-01, group spartan-cpu → success
spartan-cpu (positive control)	assigned ~3s	assigned 3s → spartan-cpu-org-02 → success

The positive control is there on purpose: had scitex-ci hung and spartan-cpu hung, the pool would simply have been down and the test would prove nothing. Both dispatched, so the result is attributable to the group move.

Consequence: the pypi-publish templates baked-in runs-on fallback —

runs-on: ${{ fromJSON(vars.CI_RUNS_ON || [self-hosted,Linux,X64,scitex-ci]) }}

— is now safe on a public repo with no repo-level runner. That was the last thing standing between this PR and a merge. Both org runner groups now allow public repos, and Default is empty.

This PR is unblocked. Its six reusable workflows all target spartan-cpu; gh 2.40.1 is present on the runner, so the two workflows this PR migrates (auto-merge-to-develop.yml, cla.yml) will work.

Carry this rule into the rollout

Migrating a repo onto these reusable workflows renames its status checks (a <caller-job-id> / prefix is added). Branch protection must be flipped to the new names in the same change, or the repo goes green-but-unmergeable — silently. Verified on scitex-config: PR #23 sat mergeStateStatus=BLOCKED with all 7 checks green until protection was patched. Read the names off a real PR; never guess them.

[ywatanabe1989]

Re-validated against today's main (2026-07-22) — merging as-is, hardcoded label confirmed correct

This PR sat 8 days and today's 74-repo sweep raised a fair challenge against it: the sweep counted three runner regimes (ubuntu-latest 818, vars.CI_RUNS_ON 441, literal [self-hosted, Linux, X64, spartan-cpu] 64) and concluded this PR hardcodes the rarest of the three, and should instead emit the canonical ci-template form:

runs-on: ${{ fromJSON(vars.CI_RUNS_ON || '["self-hosted","Linux","X64","scitex-ci"]') }}

I resolved the variable instead of counting its occurrences, and that inverts the conclusion. vars.CI_RUNS_ON is a reference, not a value — counting 441 references says nothing about which pool they select. Resolved across all 74 org repos:

CI_RUNS_ON value	repos
["self-hosted","Linux","X64","spartan-cpu"]	65
["self-hosted","Linux","X64","scitex-ci"]	1
unset (404)	8

So the 441 vars.CI_RUNS_ON instances and the 64 literal instances select the same pool. spartan-cpu is not the rarest regime — it is the fleet standard (65 of the 66 repos that set the variable). The literal scitex-ci is the rare one.

Runner capacity agrees:

spartan-cpu-org-01     online  self-hosted,Linux,X64,spartan-cpu
spartan-cpu-org-02     online  self-hosted,Linux,X64,spartan-cpu
spartan-pooled-cpu-01  online  self-hosted,Linux,X64,spartan-cpu,scitex-ci

Three runners serve spartan-cpu; one serves scitex-ci. Adopting the suggested form would have pointed these jobs at a single-runner pool.

Why not the vars form anyway, for overridability

Three reasons, in order of weight:

1. The default would fire where it hurts. There is no org-level CI_RUNS_ON (orgs/scitex-ai/actions/variables is empty) and .github itself sets none. In a reusable workflow it is not settled whether vars resolves to the caller's repo or the callee's; if the callee's, the || '[...scitex-ci]' default fires for every caller and lands all six workflows on the one-runner pool. The literal has no such failure mode.
2. It would split the regime inside this repo. The four other reusables (import-smoke, pytest-matrix, quality-audit, rtd-sphinx-build) already hardcode this exact label set. Making 2 of 6 vars-based creates precisely the per-file divergence that ci: standardize the reusable-workflow caller job id (it is the branch-protection prefix) #5 exists to prevent. If the fleet wants vars-based selection in the org reusables, that is one coherent change across all six — not a side effect of a PR about hosted runners.
3. These two are governance workflows. cla.yml gates the CLA and auto-merge-to-develop.yml merges code. A per-repo variable that redirects those jobs onto a repo-chosen runner is arguably a misfeature rather than a feature.

PS-169 (_check_hosted_runners.py) is satisfied either way — it accepts any statically-resolvable self-hosted label set, and explicitly does not require the label be scitex-ci.

Verified against today's three merges

uv venv --seed (#11), the .venv/bin PATH export (#12), and the post-merge gate dispatch + permissions: actions: write (#10) are all untouched by this PR and confirmed present in the merged tree — this PR changes runs-on: lines only, in two files neither #11 nor #12 touched.

Follow-up this PR does NOT cover

self-test.yml was added to main today (after this branch was cut) and declares runs-on: ubuntu-latest. That is a live PS-169 violation this PR cannot see. Tracking separately — it needs the same treatment.