fix(ci): pin xdist auto to the cgroup allocation, not the host's 128 cores

[ywatanabe1989]

The defect

pytest -n auto in the org-shared pytest-matrix.yml starts ~128 workers into a 48-CPU Spartan lease — 2.7x continuous oversubscription on every CI run across the ~74 repos that call this workflow.

Measured directly on spartan-bm155, inside a live 48-CPU lease cgroup (job_27144058/step_extern):

probe	value
os.sched_getaffinity(0)	48 (the actual allocation)
os.cpu_count()	128
psutil.cpu_count(logical=False)	128  ← what -n auto resolves to
SLURM_CPUS_PER_TASK	empty
SLURM_JOB_CPUS_PER_NODE	empty

Independently corroborated: scitex-hub's CI log shows worker ids reaching gw121.

Mechanism

xdist's pytest_xdist_auto_num_workers consults psutil first and returns immediately. Only if psutil is ABSENT does it fall back to sched_getaffinity, which would have been correct. So auto is right without psutil and wrong with it.

PYTEST_XDIST_AUTO_NUM_WORKERS is read at the top of that hook and returns before psutil is imported — verified against pytest-xdist v3.8.0 source:

env_var = os.environ.get("PYTEST_XDIST_AUTO_NUM_WORKERS")
if env_var:
    return int(env_var)
...
import psutil
count = psutil.cpu_count(logical=use_logical) or psutil.cpu_count()

So exporting it corrects auto rather than replacing it. -n auto is kept; no worker count is hardcoded.

Why not the SLURM vars

Both are EMPTY on these runners. The runner is ssh-adopted into the lease cgroup, so it inherits the cpuset but not the step environment. Any fix keyed on $SLURM_CPUS_PER_TASK silently produces nothing. ADR-0004 P1(b) prescribed exactly that and was wrong; already corrected in scitex-dev.

The change

One line in .github/workflows/pytest-matrix.yml, immediately before the pytest invocation (verified at line 89 on main), plus explanatory comments:

export PYTEST_XDIST_AUTO_NUM_WORKERS="$(.venv/bin/python -c 'import os; print(len(os.sched_getaffinity(0)))')"

Uses .venv/bin/python explicitly, matching the surrounding steps — the venv is created by uv venv --seed earlier in the same step, so the interpreter provably exists.

Platform safety

sched_getaffinity is Linux-only. Checked rather than assumed: runs-on is the fixed label [self-hosted, Linux, X64, spartan-cpu] and the matrix varies only python-version, so no non-Linux leg exists. No guard needed.

Scope

git grep across all tracked files on main for -n auto / xdist / numprocesses / PYTEST_XDIST: pytest-matrix.yml is the only file that matches. No other workflow shares the pattern; nothing else to fix.

Verification

• Positive control — the one-liner reads affinity, not core count:

  $ python3 -c 'import os; print(len(os.sched_getaffinity(0)))'         -> 16
  $ taskset -c 0-3 python3 -c 'import os; print(len(os.sched_getaffinity(0)))'  -> 4
  

• YAML parses (yaml.safe_load), job/matrix/steps structure intact.
• Shell syntax of the run: block passes bash -n.
• The Spartan environment is not reproducible locally — this container has sched_getaffinity == cpu_count == 16, so the 48-vs-128 distinction is invisible here. Not claimed as verified.

Pending confirmation: after merge, the proof is a CI run whose worker ids top out near the allocation (~gw47) instead of gw121.