From aa53581e74a9e1123316b2bb44aa47b1b81d7208 Mon Sep 17 00:00:00 2001 From: TGPSKI Date: Tue, 4 Aug 2026 00:50:58 -0700 Subject: [PATCH] Add make setup: discover the machine instead of assuming it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every prerequisite in v0.1.0 was a hardcoded default that happened to be true on one workstation. check-env validated those assumptions rather than discovering the machine, and there was nowhere to record a per-machine answer — so each new host rediscovered its own paths by failing a build, and each fix so far has been a slightly better guess hardcoded in the same place. scripts/setup.sh does the discovery. It finds every CUDA toolkit and gcc/g++ pair on the machine, reads the gcc cap out of the chosen toolkit's own crt/host_config.h, and compiles a real CUDA translation unit with the pair you pick. That last step is the point: nvcc -ccbin on a three-line kernel takes three seconds and is the only evidence that a toolkit and a host compiler actually agree. A version table is inference, and it goes stale one CUDA release later. When they disagree, the compile wins. The scan covers PATH, the usual distro and /opt prefixes, asdf installs, and CONCEIT_CC_SEARCH_PATH. That last one is not an escape hatch: a toolchain you built from source is precisely the one PATH does not know about, which is the case that motivated this. Answers persist to conceit.env, gitignored because it describes one machine. cuda-env.sh sources it ahead of every default, and each line is itself a ${VAR:-default}, so the precedence chain stays readable: exported by hand > this machine's intake > the guesses in cuda-env.sh. check-env now points at `make setup` instead of naming a package. .agents/skills/env-intake/SKILL.md wraps the script for the parts a scan cannot decide: which toolkit a machine standardizes on when several are installed, where an unpackaged toolchain lives, and whether a failed smoke test means "pick another compiler" or "this machine needs a package." It also says not to reach for -allow-unsupported-compiler, which trades a five-second failure for a corrupt build hours later. Verified on this machine (CUDA 13.3, gcc 16 default with gcc 15 alongside): auto mode picks gcc-15 and compiles clean; interactively picking gcc 16 fails the smoke test with nvcc's own "gcc versions later than 15 are not supported" and re-prompts; a clean shell sourcing cuda-env.sh inherits the file; a hand-exported CC still overrides it; check-env passes on the result. Caught one bug in the process — nvidia-smi prints a driver/library version mismatch on stdout in the same shape as a result, which sailed straight into the arch list until the caps were filtered to well-formed values. --- .agents/skills/env-intake/SKILL.md | 179 ++++++++++++++ .cursor/skills/env-intake | 1 + .gitignore | 3 + AGENTS.md | 17 +- CHANGELOG.md | 24 ++ Makefile | 20 +- README.md | 21 +- scripts/cuda-env.sh | 11 + scripts/setup.sh | 361 +++++++++++++++++++++++++++++ 9 files changed, 628 insertions(+), 9 deletions(-) create mode 100644 .agents/skills/env-intake/SKILL.md create mode 120000 .cursor/skills/env-intake create mode 100755 scripts/setup.sh diff --git a/.agents/skills/env-intake/SKILL.md b/.agents/skills/env-intake/SKILL.md new file mode 100644 index 0000000..6b026e4 --- /dev/null +++ b/.agents/skills/env-intake/SKILL.md @@ -0,0 +1,179 @@ +--- +name: env-intake +description: "Walk a user through conceit's environment intake on a new machine: discover every CUDA toolkit and host compiler present, resolve which pair nvcc will actually accept, prove it with a compile, and persist the answers to conceit.env. Use before the first build on any machine, when check-env fails on CUDA_HOME/CC/CXX, or when a toolchain moved and the build stopped finding it." +metadata: + author: conceit + version: "1.0" +compatibility: "bash, nvcc, gcc/g++, nvidia-smi, uv, make" +--- + +# conceit Environment Intake + +Establish a working build environment on this machine and write it down, so the +next shell and the next agent inherit it instead of rediscovering it. + +`scripts/setup.sh` does the mechanical work: scanning, smoke testing, writing +`conceit.env`. This workflow covers what the script cannot decide alone — which +toolkit a machine should standardize on, where an unpackaged toolchain lives, +whether a failed smoke test means "pick another compiler" or "this machine needs +a package." + +## Design Principles + +- **Discover, never assume.** A hardcoded path passes a prerequisite check on a + machine that cannot compile. Every value here comes from the filesystem or + from the user, never from what worked on another host. +- **The compile is the evidence.** Version tables go stale one CUDA release + later. `nvcc -ccbin ` on a three-line kernel is DEFINITIVE; a version + comparison is CONFIG. When they disagree, the compile wins. +- **Persist the answer.** An answer that lives only in this session is one the + next machine, shell, or agent pays for again. +- **Ask only what the disk cannot say.** Run the scan before the questions. + +## Prerequisites + +- A CUDA toolkit installed somewhere on this machine (`scripts/install-cuda-toolkit.sh` + installs one from an NVIDIA runfile, with a gcc shim on PATH). +- A gcc/g++ pair no newer than that toolkit accepts. +- Repo cloned; `make help` runs. + +--- + +## Step 1: Establish the starting state + +**Inspect**: + +```bash +ls -l conceit.env 2>/dev/null && make setup-show +``` + +| Status | Action | +|--------|--------| +| `conceit.env` missing | Continue to Step 2 | +| `conceit.env` exists, `make check-env` passes | Report the resolved values, stop. Nothing to do | +| `conceit.env` exists, `check-env` fails | Note which value is wrong, continue to Step 2 — the scan will re-resolve it | +| `conceit.env` exists, toolchain since moved | Continue to Step 2, regenerate | + +**Decide**: nothing yet. Do not ask the user anything before the scan runs. + +--- + +## Step 2: Scan the machine + +**Inspect**: + +```bash +bash scripts/setup.sh --auto # non-interactive: best candidate for each, still smoke tested +``` + +Read what it reports before interpreting anything: toolkits found, the gcc cap +parsed from that toolkit's own `crt/host_config.h`, compilers found with +versions, and the smoke-test result. + +| Status | Action | +|--------|--------| +| One toolkit, one supported compiler, smoke test clean | Accept it. Skip to Step 5 | +| Multiple toolkits found | Step 3 — the user picks | +| Compilers found but all over the cap | Step 4 — this machine needs a package or a from-source toolchain | +| No compiler found at all | Step 4 | +| Smoke test failed on the auto-picked pair | Step 4 — do not paper over it by raising the cap | + +**Never** reach for `nvcc -allow-unsupported-compiler` to make a failing smoke +test pass. It converts a clear five-second failure into a compile error hours +into a build, or into a binary that miscompiles at run time. + +--- + +## Step 3: Choose among what was found + +**Inspect**: the candidate list from Step 2. Versions and paths are already +resolved — do not re-derive them. + +**Decide**: + +1. "Which CUDA toolkit should this machine build against?" — default to the + newest present. A machine with 12.x and 13.x installed is usually mid-upgrade; + ask rather than assume the newest is the intended one. +2. "Which host compiler?" — default to the newest at or under the cap. + +Then run the interactive intake and make those selections: + +```bash +make setup +``` + +--- + +## Step 4: Resolve a missing or rejected compiler + +**Inspect**: + +| Status | Action | +|--------|--------| +| Distro packages a supported gcc | Offer the install command; do not run sudo without confirmation | +| A from-source toolchain exists but is off PATH | Re-scan with `CONCEIT_CC_SEARCH_PATH` | +| A from-source toolchain is too old for the rest of the stack | Say so plainly — rebuilding it is its own project, not a step here | + +**Decide**: + +1. "Where does your toolchain live?" — only if the scan found nothing usable. + A compiler built from source is precisely the one PATH does not know about. + +```bash +CONCEIT_CC_SEARCH_PATH=/prefix/bin:/other/prefix/bin make setup +``` + +Install commands, when the machine simply lacks one: + +| Distro | Command | +|--------|---------| +| Arch / Manjaro | `sudo pacman -S gcc15` | +| Debian / Ubuntu | `sudo apt install gcc-15 g++-15` | +| Fedora | `sudo dnf install gcc-15 gcc-c++-15` | + +Re-run Step 2 after any install. The smoke test, not the package manager's exit +code, is what says the machine is ready. + +--- + +## Step 5: Confirm and hand off + +**Generate**: `scripts/setup.sh` writes the file. Do not hand-write it. + +Every line keeps the `${VAR:-default}` form the rest of the repo uses, so +precedence stays readable: what the user exported by hand beats +`conceit.env`, which beats the detection defaults in `scripts/cuda-env.sh`. + +Write to: `conceit.env` (gitignored — it describes one machine) + +```bash +export CUDA_HOME="${CUDA_HOME:-/usr/local/cuda-13.3}" +export CC="${CC:-/usr/bin/gcc-15}" +export CXX="${CXX:-/usr/bin/g++-15}" +export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-12.0+PTX}" +export UV="${UV:-/home/you/.local/bin/uv}" +``` + +## Validate + +```bash +source scripts/cuda-env.sh +make check-env # every value above, re-verified from a clean shell +``` + +| Status | Action | +|--------|--------| +| `check-env` passes | Hand off to `make build-all`, or to the build-triage skill if a build then fails | +| `check-env` disagrees with `conceit.env` | The file is stale or something exported in this shell overrides it — `env \| grep -E 'CC\|CXX\|CUDA_HOME'` | + +## PR Checkpoint + +Intake produces no committed files — `conceit.env` is gitignored and describes +one machine. Open a PR only when the intake exposed a gap in the tooling itself: + +**Title**: `[env] ` + +**Files to include**: +- `scripts/setup.sh` — a search path or toolkit layout the scan should have found +- `scripts/cuda-env.sh` — a default that was wrong for this machine +- `Makefile` — a `check-env` gate that passed on an environment that could not build diff --git a/.cursor/skills/env-intake b/.cursor/skills/env-intake new file mode 120000 index 0000000..2632af4 --- /dev/null +++ b/.cursor/skills/env-intake @@ -0,0 +1 @@ +../../.agents/skills/env-intake \ No newline at end of file diff --git a/.gitignore b/.gitignore index 3f70e1d..d20cd91 100644 --- a/.gitignore +++ b/.gitignore @@ -15,6 +15,9 @@ venv/ **/*.egg-info/ **/.eggs/ +# ─── Machine-specific environment (written by `make setup`) ────────────────── +conceit.env + # ─── Runtime state files ────────────────────────────────────────────────────── **/.build-status.json **/.build-status.json.tmp diff --git a/AGENTS.md b/AGENTS.md index e9d5e17..087c3c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,11 @@ Repository: `github.com/TGPSKI/conceit` ## Before you build anything +On a machine that has never built here, run `make setup` first — it discovers +the CUDA toolkit and host compiler and proves they compile together, which is +the failure this repo has paid for most often. See +[`.agents/skills/env-intake/SKILL.md`](.agents/skills/env-intake/SKILL.md). + Read [`.agents/skills/build-triage/SKILL.md`](.agents/skills/build-triage/SKILL.md) first. It is not background reading — it is the procedure. The single highest-value rule in it: **coordinates before compilation.** Read the target repo's own @@ -51,12 +56,15 @@ wrong is measured in hours of nvcc, not seconds. ``` scripts/ + setup.sh # environment intake — scan, smoke test, write conceit.env cuda-env.sh # every env default; sourced by everything else build-upstream.sh # the orchestrator — presets, preflight, monitor, status gen-patches.sh # export live src/ diffs into patches/ install-cuda-toolkit.sh # CUDA runfile install behind a gcc-15 shim patches// # generated; committed; replayed by make patch-* .agents/skills/build-triage/SKILL.md +.agents/skills/env-intake/SKILL.md +conceit.env # gitignored — this machine's answers, written by make setup src/ # gitignored — cloned upstream trees live here ``` @@ -73,7 +81,14 @@ src/ # gitignored — cloned upstream trees live here and `clone` only show up on a re-run. Emitting it is best-effort by design: a status write must never take down a build that has been running for hours. - **Every env var takes the form `${VAR:-default}`**, so any single value can be - overridden without editing a file. Keep it that way. + overridden without editing a file. Keep it that way. `conceit.env` (written by + `make setup`, gitignored) is sourced first and uses the same form, so precedence + reads: exported by hand > this machine's intake > the defaults in `cuda-env.sh`. +- **Discover the machine, do not assume it.** A hardcoded path is what lets + `check-env` pass on a host that cannot compile. `scripts/setup.sh` scans for + toolkits and compilers and then *compiles a CUDA translation unit* with the pair + it picked — the only evidence that outranks a version comparison. New + prerequisites belong in that scan, not in a new hardcoded default. - **Paths derive from the repo root**, computed from the script's own location. Never reintroduce a hardcoded `$HOME/...` path — it breaks every clone that isn't yours. diff --git a/CHANGELOG.md b/CHANGELOG.md index 66bbea7..97b2979 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0. ## [Unreleased] +### Added + +- **`make setup` — environment intake.** Every prerequisite in v0.1.0 was a + hardcoded default that happened to be true on one workstation, and `check-env` + only validated those assumptions rather than discovering the machine. There was + nowhere to record a per-machine answer, so each new host rediscovered its own + paths by failing a build. `scripts/setup.sh` finds every CUDA toolkit and + gcc/g++ pair present — including toolchains built from source and off `PATH`, + via `CONCEIT_CC_SEARCH_PATH` — reads the gcc cap out of the chosen toolkit's own + `crt/host_config.h`, and then **compiles a real CUDA translation unit with the + pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a + toolkit and a host compiler agree; when it disagrees with a version table, the + compile wins. Answers persist to `conceit.env` (gitignored), sourced by + `cuda-env.sh` ahead of every default and written in the same `${VAR:-default}` + form, so precedence reads: exported by hand > this machine's intake > the + defaults. `make setup-auto` skips the prompts, `make setup-show` prints the + result. +- **`.agents/skills/env-intake/SKILL.md`** — the directed workflow around that + script, for the parts a scan cannot decide: which toolkit a machine should + standardize on, where an unpackaged toolchain lives, and whether a failed smoke + test means "pick another compiler" or "this machine needs a package." It also + says plainly not to reach for `-allow-unsupported-compiler`, which converts a + five-second failure into a corrupt build hours later. + ### Fixed - **The first build on a fresh clone always failed.** `emit_status` writes diff --git a/Makefile b/Makefile index 5a698d0..47dc10a 100644 --- a/Makefile +++ b/Makefile @@ -103,19 +103,31 @@ endef # ── Environment ─────────────────────────────────────────────────────────────── +.PHONY: setup +setup: ## Interactive intake: find CUDA + compiler, prove they compile, write conceit.env + @bash $(SCRIPTS_DIR)/setup.sh + +.PHONY: setup-auto +setup-auto: ## Same intake with no prompts — best candidate for each, still smoke tested + @bash $(SCRIPTS_DIR)/setup.sh --auto + +.PHONY: setup-show +setup-show: ## Print the conceit.env this machine is using + @bash $(SCRIPTS_DIR)/setup.sh --show + .PHONY: check-env check-env: ## Validate prerequisites: CUDA, Python, uv, GPU, disk - @[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: source scripts/cuda-env.sh"; exit 1; } + @[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: make setup && source scripts/cuda-env.sh"; exit 1; } @[ -d "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME=$(CUDA_HOME) does not exist"; exit 1; } @$(call require_bins,nvidia-smi python) - @[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Install a CUDA-supported gcc (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; } - @[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Install a CUDA-supported g++ (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; } + @[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; } + @[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; } @hdr="$(CUDA_HOME)/include/crt/host_config.h"; \ max=$$(grep -oE '__GNUC__ > [0-9]+' "$$hdr" 2>/dev/null | head -1 | grep -oE '[0-9]+$$' || true); \ have=$$("$(CXX)" -dumpversion 2>/dev/null | cut -d. -f1 || true); \ if [ -n "$$max" ] && [ -n "$$have" ] && [ "$$have" -gt "$$max" ]; then \ echo "ERROR: $(CXX) is gcc $$have, and this CUDA supports up to gcc $$max — nvcc will refuse to compile."; \ - echo " Install gcc $$max and re-source scripts/cuda-env.sh, or export CC/CXX to one."; \ + echo " Run 'make setup': it finds every compiler on this machine and smoke tests the one you pick."; \ exit 1; \ fi @[ -x "$(UV)" ] || { echo "ERROR: uv not found at $(UV). Install: curl -LsSf https://astral.sh/uv/install.sh | sh"; exit 1; } diff --git a/README.md b/README.md index 4ebec9d..58ae8f7 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ No wheel to wait for. No container to inherit. No blog post to follow line by line. ```bash +make setup source scripts/cuda-env.sh make check-env make build-pytorch @@ -29,6 +30,7 @@ conceit is that path, written down and made repeatable. | | | |---|---| +| **Intake** | `make setup` discovers this machine rather than assuming it: every CUDA toolkit and gcc/g++ pair present (including from-source toolchains off `PATH`), the gcc cap read out of the toolkit's own `crt/host_config.h`, then a real `nvcc` compile to prove the pair works. Answers persist to `conceit.env`. | | **Orchestrator** | `build-upstream.sh` clones, syncs submodules with backoff, runs preflight, creates the venv, compiles under a process-tree monitor, and produces wheels. Per-target presets encode the quirks: vLLM's uv + local-torch override that keeps `cuda-toolkit` out of the dep graph, PyTorch's `--no-build-isolation`, llama.cpp's cmake flags. | | **Build state** | `.build-status.json` per target — `init → clone → venv → build_start → build_done → wheel → done`, with the current ninja step and a live count of every descendant process. Poll it instead of grepping logs. | | **Patches** | Generated from the live source tree, never hand-written. Applying is idempotent; a patch that no longer applies is a hard error, not a warning. | @@ -75,20 +77,25 @@ Anything scoring 4–5 gets resolved before you compile. Every row cost a real b ```bash git clone https://github.com/TGPSKI/conceit.git cd conceit -source scripts/cuda-env.sh # CUDA_HOME, TORCH_CUDA_ARCH_LIST=12.0+PTX, host compiler -make check-env # fails fast on anything missing +make setup # find CUDA + compiler on THIS machine, compile-test the pair, write conceit.env +source scripts/cuda-env.sh # picks up conceit.env +make check-env # re-verifies everything, fails fast on anything missing ``` -`make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout. +`make setup` is the intake. It scans for every CUDA toolkit and gcc/g++ pair on the machine — including toolchains built from source and off `PATH`, via `CONCEIT_CC_SEARCH_PATH=/prefix/bin` — shows you what it found with versions, and then **compiles a real CUDA translation unit with the pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a toolkit and a host compiler agree; a version table goes stale one CUDA release later. The answers land in `conceit.env`, which every later shell inherits. + +`make setup-auto` takes the best candidate for each without prompting, still smoke tested. `make setup-show` prints what this machine resolved. `make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout. **Tested on:** Blackwell `sm_120` (RTX PRO 4500), CUDA 13.3, Python 3.14.6 via asdf, gcc/g++ 15, x86_64 Manjaro. Hopper and Ampere work by setting `TORCH_CUDA_ARCH_LIST`. Budget ~200 GB of disk and 32 GB of RAM for parallel nvcc. -**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `cuda-env.sh` picks the newest supported `gcc-*` it finds; `make check-env` fails if the result is missing or too new for the toolkit. +**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `make setup` finds all of these where they actually live on your machine and tells you which one is missing. ## Configure Every knob is an environment variable with a working default, resolved in `scripts/cuda-env.sh`. Export one to override it; nothing needs editing. +Precedence runs in one direction: **what you export by hand** beats **`conceit.env`** (what `make setup` found on this machine) beats **the detection defaults in `cuda-env.sh`**. Every line in `conceit.env` is itself a `${VAR:-default}`, which is what keeps that order honest. + ```bash TORCH_CUDA_ARCH_LIST=9.0+PTX # build for Hopper instead MAX_JOBS=8 # defaults to nproc @@ -99,10 +106,16 @@ CUDA_HOME=/usr/local/cuda-13.2 cuDNN, cuBLASMp, and cuDSS are found by globbing `cuda/` for their extracted archives, so upgrading one means extracting the new tarball and nothing else. +```bash +CONCEIT_CC_SEARCH_PATH=/opt/toolchains/bin # where make setup looks for compilers beyond PATH +CONCEIT_ENV_FILE=/etc/conceit.env # keep the machine's answers somewhere else +``` + ## Go deeper | you want to… | start here | then | |---|---|---| +| **set up** a new machine | [the env-intake skill](.agents/skills/env-intake/SKILL.md) — scan, choose, compile-test, persist | `make setup` · `make setup-show` | | **build** the stack | `make help` — every target, one line each | [AGENTS.md](AGENTS.md) for the invariants · `make env-show` for resolved paths | | **debug** a failed build | [the triage skill](.agents/skills/build-triage/SKILL.md) — intake, adversarial review, coordinate resolution | `make status` · `.build-status.json` per target | | **change** something | [CONTRIBUTING.md](CONTRIBUTING.md) — what `make check` gates and why patches are generated | [CHANGELOG.md](CHANGELOG.md) | diff --git a/scripts/cuda-env.sh b/scripts/cuda-env.sh index e0ed71a..11a7bd7 100755 --- a/scripts/cuda-env.sh +++ b/scripts/cuda-env.sh @@ -18,6 +18,17 @@ fi export CONCEIT_ROOT="${CONCEIT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" export CUDA_STAGE_ROOT="${CUDA_STAGE_ROOT:-$CONCEIT_ROOT/cuda}" +# Machine-specific answers from `make setup`, sourced before any default below. +# Every line in it is itself a ${VAR:-default}, so the precedence chain reads: +# what you exported by hand > what setup discovered on this machine > the +# guesses here. Without this layer every new machine has to rediscover its own +# paths by failing a build. +_conceit_env_file="${CONCEIT_ENV_FILE:-$CONCEIT_ROOT/conceit.env}" +if [[ -s "$_conceit_env_file" ]]; then + source "$_conceit_env_file" +fi +unset _conceit_env_file + if [[ -z "${CUDA_HOME:-}" ]]; then for _d in /usr/local/cuda-13.3 /usr/local/cuda; do [[ -d "$_d" ]] && { export CUDA_HOME="$_d"; break; } diff --git a/scripts/setup.sh b/scripts/setup.sh new file mode 100755 index 0000000..9c17d6e --- /dev/null +++ b/scripts/setup.sh @@ -0,0 +1,361 @@ +#!/usr/bin/env bash +# +# Environment intake. Finds every CUDA toolkit and host compiler on this +# machine, proves the pair you pick actually compiles a CUDA translation unit, +# and writes the answers to conceit.env so every future shell inherits them. +# +# scripts/setup.sh # interactive +# scripts/setup.sh --auto # no prompts: best candidate for each, still smoke tested +# scripts/setup.sh --show # print the current conceit.env and exit +# +# The point is that a machine is discovered, not assumed. Hardcoded defaults +# ("gcc-15 lives in /usr/bin") pass a prerequisite check on a machine that +# cannot compile, and you find out an hour into a build. +# +# Compilers built from source rarely sit on PATH. Point at them with: +# CONCEIT_CC_SEARCH_PATH=/opt/toolchains/bin:/srv/gcc-15/bin scripts/setup.sh + +set -euo pipefail +shopt -s nullglob + +CONCEIT_ROOT="${CONCEIT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +ENV_FILE="${CONCEIT_ENV_FILE:-$CONCEIT_ROOT/conceit.env}" + +interactive=1 + +die() { printf 'error: %s\n' "$*" >&2; exit 1; } +warn() { printf 'WARN %s\n' "$*" >&2; } +say() { printf '%s\n' "$*" >&2; } +hdr() { printf '\n── %s %s\n' "$1" "$(printf '─%.0s' $(seq 1 $((60 - ${#1}))))" >&2; } + +case "${1:-}" in + --auto) interactive=0 ;; + --show) + [[ -f "$ENV_FILE" ]] || die "no $ENV_FILE yet — run: make setup" + cat "$ENV_FILE" + exit 0 + ;; + --help|-h) + sed -n '2,18p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 0 + ;; + '') ;; + *) die "unknown argument: $1 (use --auto, --show, or --help)" ;; +esac + +# A pipe or a cron job has no one to answer the prompts. Fall back rather than +# block forever on a read that will never return. +if [[ ! -t 0 ]] && (( interactive )); then + warn "stdin is not a terminal — running as --auto" + interactive=0 +fi + +# prompt_choice ... +# Renders a menu on stderr and echoes the chosen 1-based index on stdout. +# In --auto mode it echoes the default without rendering anything. +prompt_choice() { + local default=$1; shift + local -a items=("$@") + local i marker reply + + if (( ! interactive )); then + printf '%s' "$default" + return 0 + fi + + for i in "${!items[@]}"; do + marker="" + (( i + 1 == default )) && marker=" <- default" + printf ' %2d) %s%s\n' "$((i + 1))" "${items[i]}" "$marker" >&2 + done + + while true; do + printf 'choice [%s]: ' "$default" >&2 + read -r reply || reply="" + [[ -z "$reply" ]] && reply=$default + if [[ "$reply" =~ ^[0-9]+$ ]] && (( reply >= 1 && reply <= ${#items[@]} )); then + printf '%s' "$reply" + return 0 + fi + say " not one of 1..${#items[@]}" + done +} + +# prompt_value +prompt_value() { + local prompt=$1 default=$2 reply + if (( ! interactive )); then + printf '%s' "$default" + return 0 + fi + printf '%s [%s]: ' "$prompt" "$default" >&2 + read -r reply || reply="" + printf '%s' "${reply:-$default}" +} + +# ── CUDA toolkits ───────────────────────────────────────────────────────────── + +hdr "CUDA toolkit" + +declare -A _seen_cuda=() +declare -a cuda_dirs=() +add_cuda() { + local d=$1 real + [[ -n "$d" && -x "$d/bin/nvcc" ]] || return 0 + real=$(cd "$d" && pwd -P) + [[ -n "${_seen_cuda[$real]:-}" ]] && return 0 + _seen_cuda[$real]=1 + cuda_dirs+=("$real") +} + +for _d in "${CUDA_HOME:-}" /usr/local/cuda /usr/local/cuda-* /opt/cuda /opt/cuda-*; do + add_cuda "$_d" +done +# An nvcc on PATH can live outside every conventional prefix. +if command -v nvcc >/dev/null 2>&1; then + add_cuda "$(cd "$(dirname "$(command -v nvcc)")/.." && pwd -P)" +fi +unset _d + +(( ${#cuda_dirs[@]} )) || die "no CUDA toolkit found (looked for bin/nvcc under /usr/local/cuda*, /opt/cuda*, \$CUDA_HOME, and PATH). + Install one with: scripts/install-cuda-toolkit.sh /path/to/cuda_*.run" + +cuda_version() { + "$1/bin/nvcc" --version 2>/dev/null | awk '/release/ {gsub(",", "", $(NF-1)); print $(NF-1); exit}' +} + +declare -a cuda_labels=() +for _d in "${cuda_dirs[@]}"; do + cuda_labels+=("$(printf 'CUDA %-6s %s' "$(cuda_version "$_d")" "$_d")") +done +unset _d + +# Default to the newest toolkit present. +cuda_default=1 +if (( ${#cuda_dirs[@]} > 1 )); then + _best=$(for i in "${!cuda_dirs[@]}"; do + printf '%s %s\n' "$(cuda_version "${cuda_dirs[i]}")" "$((i + 1))" + done | sort -V | tail -1 | awk '{print $2}') + cuda_default=$_best + unset _best +fi + +say "found ${#cuda_dirs[@]} toolkit(s):" +_pick=$(prompt_choice "$cuda_default" "${cuda_labels[@]}") +CUDA_HOME=${cuda_dirs[$((_pick - 1))]} +cuda_ver=$(cuda_version "$CUDA_HOME") +say "using CUDA $cuda_ver at $CUDA_HOME" + +# nvcc's own host_config.h is the authority on which gcc it will accept — not +# a table in a README that goes stale one CUDA release later. +gcc_cap=$(grep -oE '__GNUC__ > [0-9]+' "$CUDA_HOME/include/crt/host_config.h" 2>/dev/null \ + | head -1 | grep -oE '[0-9]+$' || true) +if [[ -n "$gcc_cap" ]]; then + say "this toolkit accepts host gcc up to $gcc_cap" +else + warn "could not read a gcc cap from $CUDA_HOME/include/crt/host_config.h — the smoke test still decides" +fi + +# ── Host compilers ──────────────────────────────────────────────────────────── + +hdr "host compiler" + +# PATH first, then the places a distro package or a from-source install puts a +# toolchain. A compiler you built yourself is exactly the one PATH does not know +# about, so CONCEIT_CC_SEARCH_PATH is part of the contract, not an escape hatch. +declare -a search_dirs=() +IFS=: read -ra search_dirs <<< "${CONCEIT_CC_SEARCH_PATH:+$CONCEIT_CC_SEARCH_PATH:}$PATH" +search_dirs+=( + /usr/bin /usr/local/bin /opt/bin + /opt/*/bin /opt/*/*/bin + /usr/local/*/bin + "$HOME/.local/bin" "$HOME/opt/"*/bin "$HOME/toolchains/"*/bin + "${ASDF_DATA_DIR:-$HOME/.asdf}/installs/gcc/"*/bin +) + +declare -A _seen_cc=() +declare -a cc_rows=() # "version|gcc|g++" +for _dir in "${search_dirs[@]}"; do + [[ -d "$_dir" ]] || continue + for _cc in "$_dir"/gcc "$_dir"/gcc-[0-9]*; do + [[ -x "$_cc" && -f "$_cc" ]] || continue + + # A gcc with no g++ beside it cannot build torch, so pair them up front. + _cxx="${_cc%/*}/$(basename "$_cc" | sed 's/^gcc/g++/')" + [[ -x "$_cxx" ]] || _cxx="${_cc%/*}/g++" + [[ -x "$_cxx" ]] || continue + + _key="$(readlink -f "$_cc"):$(readlink -f "$_cxx")" + [[ -n "${_seen_cc[$_key]:-}" ]] && continue + _seen_cc[$_key]=1 + + _ver=$("$_cc" -dumpfullversion 2>/dev/null || "$_cc" -dumpversion 2>/dev/null || true) + [[ -n "$_ver" ]] || continue + cc_rows+=("$_ver|$_cc|$_cxx") + done +done +unset _dir _cc _cxx _key _ver + +(( ${#cc_rows[@]} )) || die "no gcc/g++ pair found. + Install one (pacman -S gcc15 | apt install gcc-15 g++-15), or point at a + from-source toolchain: CONCEIT_CC_SEARCH_PATH=/prefix/bin scripts/setup.sh" + +# Newest first, and anything over the cap sinks to the bottom: it is present, +# it is selectable, and it is labelled as the thing nvcc will reject. +declare -a cc_sorted=() +mapfile -t cc_sorted < <( + for row in "${cc_rows[@]}"; do + ver=${row%%|*} + major=${ver%%.*} + rank=0 + if [[ -n "$gcc_cap" ]] && (( major > gcc_cap )); then rank=1; fi + printf '%s\t%s\n' "$rank" "$row" + done | sort -t$'\t' -k1,1n -k2,2Vr | cut -f2 +) + +declare -a cc_labels=() +for row in "${cc_sorted[@]}"; do + IFS='|' read -r _ver _cc _cxx <<< "$row" + note="" + if [[ -n "$gcc_cap" ]] && (( ${_ver%%.*} > gcc_cap )); then + note=" (too new — CUDA $cuda_ver stops at gcc $gcc_cap)" + fi + cc_labels+=("$(printf 'gcc %-10s %s%s' "$_ver" "$_cc" "$note")") +done +cc_labels+=("enter a path manually") +unset row note _ver _cc _cxx + +say "found ${#cc_sorted[@]} compiler(s):" + +CC=""; CXX="" +while true; do + _pick=$(prompt_choice 1 "${cc_labels[@]}") + + if (( _pick == ${#cc_labels[@]} )); then + CC=$(prompt_value " path to gcc" "") + [[ -x "$CC" ]] || { warn "$CC is not executable"; continue; } + CXX=$(prompt_value " path to g++" "${CC%/*}/$(basename "$CC" | sed 's/^gcc/g++/')") + [[ -x "$CXX" ]] || { warn "$CXX is not executable"; continue; } + else + IFS='|' read -r _ver CC CXX <<< "${cc_sorted[$((_pick - 1))]}" + fi + + # ── The gate: a real compile, not a version comparison ──────────────────── + # + # Everything above is inference. This is the only step that proves nvcc and + # this host compiler agree, and it costs about three seconds instead of the + # hour a build spends before failing on the same mismatch. + probe_dir=$(mktemp -d) + cat > "$probe_dir/probe.cu" <<'PROBE' +#include +__global__ void probe_kernel() {} +int main() { probe_kernel<<<1, 1>>>(); std::printf("ok\n"); return 0; } +PROBE + + say "" + say "smoke test: nvcc -ccbin $CXX" + if "$CUDA_HOME/bin/nvcc" -ccbin "$CXX" -c "$probe_dir/probe.cu" -o "$probe_dir/probe.o" \ + > "$probe_dir/out" 2>&1; then + say " compiled clean" + rm -rf "$probe_dir" + break + fi + + say " FAILED:" + sed 's/^/ /' "$probe_dir/out" | head -20 >&2 + rm -rf "$probe_dir" + + (( interactive )) || die "smoke test failed for $CXX and no terminal to pick another" + say "" + say "pick a different compiler:" +done + +say "using gcc $(basename "$CC") -> $CC" + +# ── GPU architecture ────────────────────────────────────────────────────────── + +hdr "GPU architecture" + +arch_default="${TORCH_CUDA_ARCH_LIST:-}" +if [[ -z "$arch_default" ]]; then + if command -v nvidia-smi >/dev/null 2>&1; then + # nvidia-smi prints its failures (a driver/library version mismatch, say) on + # stdout in the same shape as a result. Keep only well-formed compute caps, + # or an unusable driver silently becomes the arch list. + mapfile -t _caps < <(nvidia-smi --query-gpu=compute_cap --format=csv,noheader 2>/dev/null \ + | tr -d ' ' | grep -E '^[0-9]+\.[0-9]+$' | sort -u || true) + if (( ${#_caps[@]} )); then + arch_default=$(printf '%s+PTX;' "${_caps[@]}") + arch_default=${arch_default%;} + say "detected: $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | paste -sd',' -)" + else + warn "nvidia-smi returned no usable compute capability — is the driver healthy?" + fi + unset _caps + fi +fi +if [[ -z "$arch_default" ]]; then + arch_default="12.0+PTX" + warn "no GPU detected — defaulting to $arch_default (leaving this unset compiles every arch, ~9x the nvcc time)" +fi + +TORCH_CUDA_ARCH_LIST=$(prompt_value "TORCH_CUDA_ARCH_LIST" "$arch_default") + +# ── Supporting tools ────────────────────────────────────────────────────────── + +hdr "supporting tools" + +UV="${UV:-$HOME/.local/bin/uv}" +if [[ -x "$UV" ]]; then + say "uv $("$UV" --version)" +elif command -v uv >/dev/null 2>&1; then + UV=$(command -v uv) + say "uv $("$UV" --version) ($UV)" +else + UV="" + warn "uv not found — vLLM builds need it: curl -LsSf https://astral.sh/uv/install.sh | sh" +fi + +if command -v python >/dev/null 2>&1; then + say "python $(python --version 2>&1)" +else + warn "python not found on PATH" +fi + +# ── Write it down ───────────────────────────────────────────────────────────── + +hdr "conceit.env" + +if [[ -f "$ENV_FILE" ]] && (( interactive )); then + say "$ENV_FILE exists:" + sed 's/^/ /' "$ENV_FILE" >&2 + _ans=$(prompt_value "overwrite? [y/N]" "N") + [[ "$_ans" =~ ^[Yy] ]] || { say "left alone"; exit 0; } +fi + +# Every line keeps the ${VAR:-default} form the rest of the repo uses, so the +# precedence chain stays honest: something you exported by hand still beats this +# file, and this file beats the guesses in cuda-env.sh. +# The ${VAR:-...} has to reach the file literally — expanding it here would bake +# in this shell's values and destroy the precedence chain the file exists for. +# shellcheck disable=SC2016 +{ + printf '# Generated by scripts/setup.sh on %s.\n' "$(hostname)" + printf '# Machine-specific answers, sourced by scripts/cuda-env.sh. Not committed.\n' + printf '# Edit freely, or re-run `make setup` to regenerate. Delete to fall back to detection.\n' + printf '\n' + printf 'export CUDA_HOME="${CUDA_HOME:-%s}"\n' "$CUDA_HOME" + printf 'export CC="${CC:-%s}"\n' "$CC" + printf 'export CXX="${CXX:-%s}"\n' "$CXX" + printf 'export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-%s}"\n' "$TORCH_CUDA_ARCH_LIST" + if [[ -n "$UV" ]]; then + printf 'export UV="${UV:-%s}"\n' "$UV" + fi +} > "$ENV_FILE" + +sed 's/^/ /' "$ENV_FILE" >&2 + +hdr "next" +say " source scripts/cuda-env.sh # picks up conceit.env" +say " make check-env # re-verifies everything above" +say " make build-all"