diff --git a/.agents/skills/env-intake/SKILL.md b/.agents/skills/env-intake/SKILL.md new file mode 100644 index 0000000..6b026e4 --- /dev/null +++ b/.agents/skills/env-intake/SKILL.md @@ -0,0 +1,179 @@ +--- +name: env-intake +description: "Walk a user through conceit's environment intake on a new machine: discover every CUDA toolkit and host compiler present, resolve which pair nvcc will actually accept, prove it with a compile, and persist the answers to conceit.env. Use before the first build on any machine, when check-env fails on CUDA_HOME/CC/CXX, or when a toolchain moved and the build stopped finding it." +metadata: + author: conceit + version: "1.0" +compatibility: "bash, nvcc, gcc/g++, nvidia-smi, uv, make" +--- + +# conceit Environment Intake + +Establish a working build environment on this machine and write it down, so the +next shell and the next agent inherit it instead of rediscovering it. + +`scripts/setup.sh` does the mechanical work: scanning, smoke testing, writing +`conceit.env`. This workflow covers what the script cannot decide alone — which +toolkit a machine should standardize on, where an unpackaged toolchain lives, +whether a failed smoke test means "pick another compiler" or "this machine needs +a package." + +## Design Principles + +- **Discover, never assume.** A hardcoded path passes a prerequisite check on a + machine that cannot compile. Every value here comes from the filesystem or + from the user, never from what worked on another host. +- **The compile is the evidence.** Version tables go stale one CUDA release + later. `nvcc -ccbin ` on a three-line kernel is DEFINITIVE; a version + comparison is CONFIG. When they disagree, the compile wins. +- **Persist the answer.** An answer that lives only in this session is one the + next machine, shell, or agent pays for again. +- **Ask only what the disk cannot say.** Run the scan before the questions. + +## Prerequisites + +- A CUDA toolkit installed somewhere on this machine (`scripts/install-cuda-toolkit.sh` + installs one from an NVIDIA runfile, with a gcc shim on PATH). +- A gcc/g++ pair no newer than that toolkit accepts. +- Repo cloned; `make help` runs. + +--- + +## Step 1: Establish the starting state + +**Inspect**: + +```bash +ls -l conceit.env 2>/dev/null && make setup-show +``` + +| Status | Action | +|--------|--------| +| `conceit.env` missing | Continue to Step 2 | +| `conceit.env` exists, `make check-env` passes | Report the resolved values, stop. Nothing to do | +| `conceit.env` exists, `check-env` fails | Note which value is wrong, continue to Step 2 — the scan will re-resolve it | +| `conceit.env` exists, toolchain since moved | Continue to Step 2, regenerate | + +**Decide**: nothing yet. Do not ask the user anything before the scan runs. + +--- + +## Step 2: Scan the machine + +**Inspect**: + +```bash +bash scripts/setup.sh --auto # non-interactive: best candidate for each, still smoke tested +``` + +Read what it reports before interpreting anything: toolkits found, the gcc cap +parsed from that toolkit's own `crt/host_config.h`, compilers found with +versions, and the smoke-test result. + +| Status | Action | +|--------|--------| +| One toolkit, one supported compiler, smoke test clean | Accept it. Skip to Step 5 | +| Multiple toolkits found | Step 3 — the user picks | +| Compilers found but all over the cap | Step 4 — this machine needs a package or a from-source toolchain | +| No compiler found at all | Step 4 | +| Smoke test failed on the auto-picked pair | Step 4 — do not paper over it by raising the cap | + +**Never** reach for `nvcc -allow-unsupported-compiler` to make a failing smoke +test pass. It converts a clear five-second failure into a compile error hours +into a build, or into a binary that miscompiles at run time. + +--- + +## Step 3: Choose among what was found + +**Inspect**: the candidate list from Step 2. Versions and paths are already +resolved — do not re-derive them. + +**Decide**: + +1. "Which CUDA toolkit should this machine build against?" — default to the + newest present. A machine with 12.x and 13.x installed is usually mid-upgrade; + ask rather than assume the newest is the intended one. +2. "Which host compiler?" — default to the newest at or under the cap. + +Then run the interactive intake and make those selections: + +```bash +make setup +``` + +--- + +## Step 4: Resolve a missing or rejected compiler + +**Inspect**: + +| Status | Action | +|--------|--------| +| Distro packages a supported gcc | Offer the install command; do not run sudo without confirmation | +| A from-source toolchain exists but is off PATH | Re-scan with `CONCEIT_CC_SEARCH_PATH` | +| A from-source toolchain is too old for the rest of the stack | Say so plainly — rebuilding it is its own project, not a step here | + +**Decide**: + +1. "Where does your toolchain live?" — only if the scan found nothing usable. + A compiler built from source is precisely the one PATH does not know about. + +```bash +CONCEIT_CC_SEARCH_PATH=/prefix/bin:/other/prefix/bin make setup +``` + +Install commands, when the machine simply lacks one: + +| Distro | Command | +|--------|---------| +| Arch / Manjaro | `sudo pacman -S gcc15` | +| Debian / Ubuntu | `sudo apt install gcc-15 g++-15` | +| Fedora | `sudo dnf install gcc-15 gcc-c++-15` | + +Re-run Step 2 after any install. The smoke test, not the package manager's exit +code, is what says the machine is ready. + +--- + +## Step 5: Confirm and hand off + +**Generate**: `scripts/setup.sh` writes the file. Do not hand-write it. + +Every line keeps the `${VAR:-default}` form the rest of the repo uses, so +precedence stays readable: what the user exported by hand beats +`conceit.env`, which beats the detection defaults in `scripts/cuda-env.sh`. + +Write to: `conceit.env` (gitignored — it describes one machine) + +```bash +export CUDA_HOME="${CUDA_HOME:-/usr/local/cuda-13.3}" +export CC="${CC:-/usr/bin/gcc-15}" +export CXX="${CXX:-/usr/bin/g++-15}" +export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-12.0+PTX}" +export UV="${UV:-/home/you/.local/bin/uv}" +``` + +## Validate + +```bash +source scripts/cuda-env.sh +make check-env # every value above, re-verified from a clean shell +``` + +| Status | Action | +|--------|--------| +| `check-env` passes | Hand off to `make build-all`, or to the build-triage skill if a build then fails | +| `check-env` disagrees with `conceit.env` | The file is stale or something exported in this shell overrides it — `env \| grep -E 'CC\|CXX\|CUDA_HOME'` | + +## PR Checkpoint + +Intake produces no committed files — `conceit.env` is gitignored and describes +one machine. Open a PR only when the intake exposed a gap in the tooling itself: + +**Title**: `[env] ` + +**Files to include**: +- `scripts/setup.sh` — a search path or toolkit layout the scan should have found +- `scripts/cuda-env.sh` — a default that was wrong for this machine +- `Makefile` — a `check-env` gate that passed on an environment that could not build diff --git a/.cursor/skills/env-intake b/.cursor/skills/env-intake new file mode 120000 index 0000000..2632af4 --- /dev/null +++ b/.cursor/skills/env-intake @@ -0,0 +1 @@ +../../.agents/skills/env-intake \ No newline at end of file diff --git a/.gitignore b/.gitignore index 3f70e1d..d20cd91 100644 --- a/.gitignore +++ b/.gitignore @@ -15,6 +15,9 @@ venv/ **/*.egg-info/ **/.eggs/ +# ─── Machine-specific environment (written by `make setup`) ────────────────── +conceit.env + # ─── Runtime state files ────────────────────────────────────────────────────── **/.build-status.json **/.build-status.json.tmp diff --git a/AGENTS.md b/AGENTS.md index e9d5e17..087c3c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,11 @@ Repository: `github.com/TGPSKI/conceit` ## Before you build anything +On a machine that has never built here, run `make setup` first — it discovers +the CUDA toolkit and host compiler and proves they compile together, which is +the failure this repo has paid for most often. See +[`.agents/skills/env-intake/SKILL.md`](.agents/skills/env-intake/SKILL.md). + Read [`.agents/skills/build-triage/SKILL.md`](.agents/skills/build-triage/SKILL.md) first. It is not background reading — it is the procedure. The single highest-value rule in it: **coordinates before compilation.** Read the target repo's own @@ -51,12 +56,15 @@ wrong is measured in hours of nvcc, not seconds. ``` scripts/ + setup.sh # environment intake — scan, smoke test, write conceit.env cuda-env.sh # every env default; sourced by everything else build-upstream.sh # the orchestrator — presets, preflight, monitor, status gen-patches.sh # export live src/ diffs into patches/ install-cuda-toolkit.sh # CUDA runfile install behind a gcc-15 shim patches// # generated; committed; replayed by make patch-* .agents/skills/build-triage/SKILL.md +.agents/skills/env-intake/SKILL.md +conceit.env # gitignored — this machine's answers, written by make setup src/ # gitignored — cloned upstream trees live here ``` @@ -73,7 +81,14 @@ src/ # gitignored — cloned upstream trees live here and `clone` only show up on a re-run. Emitting it is best-effort by design: a status write must never take down a build that has been running for hours. - **Every env var takes the form `${VAR:-default}`**, so any single value can be - overridden without editing a file. Keep it that way. + overridden without editing a file. Keep it that way. `conceit.env` (written by + `make setup`, gitignored) is sourced first and uses the same form, so precedence + reads: exported by hand > this machine's intake > the defaults in `cuda-env.sh`. +- **Discover the machine, do not assume it.** A hardcoded path is what lets + `check-env` pass on a host that cannot compile. `scripts/setup.sh` scans for + toolkits and compilers and then *compiles a CUDA translation unit* with the pair + it picked — the only evidence that outranks a version comparison. New + prerequisites belong in that scan, not in a new hardcoded default. - **Paths derive from the repo root**, computed from the script's own location. Never reintroduce a hardcoded `$HOME/...` path — it breaks every clone that isn't yours. diff --git a/CHANGELOG.md b/CHANGELOG.md index 66bbea7..97b2979 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0. ## [Unreleased] +### Added + +- **`make setup` — environment intake.** Every prerequisite in v0.1.0 was a + hardcoded default that happened to be true on one workstation, and `check-env` + only validated those assumptions rather than discovering the machine. There was + nowhere to record a per-machine answer, so each new host rediscovered its own + paths by failing a build. `scripts/setup.sh` finds every CUDA toolkit and + gcc/g++ pair present — including toolchains built from source and off `PATH`, + via `CONCEIT_CC_SEARCH_PATH` — reads the gcc cap out of the chosen toolkit's own + `crt/host_config.h`, and then **compiles a real CUDA translation unit with the + pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a + toolkit and a host compiler agree; when it disagrees with a version table, the + compile wins. Answers persist to `conceit.env` (gitignored), sourced by + `cuda-env.sh` ahead of every default and written in the same `${VAR:-default}` + form, so precedence reads: exported by hand > this machine's intake > the + defaults. `make setup-auto` skips the prompts, `make setup-show` prints the + result. +- **`.agents/skills/env-intake/SKILL.md`** — the directed workflow around that + script, for the parts a scan cannot decide: which toolkit a machine should + standardize on, where an unpackaged toolchain lives, and whether a failed smoke + test means "pick another compiler" or "this machine needs a package." It also + says plainly not to reach for `-allow-unsupported-compiler`, which converts a + five-second failure into a corrupt build hours later. + ### Fixed - **The first build on a fresh clone always failed.** `emit_status` writes diff --git a/Makefile b/Makefile index 5a698d0..47dc10a 100644 --- a/Makefile +++ b/Makefile @@ -103,19 +103,31 @@ endef # ── Environment ─────────────────────────────────────────────────────────────── +.PHONY: setup +setup: ## Interactive intake: find CUDA + compiler, prove they compile, write conceit.env + @bash $(SCRIPTS_DIR)/setup.sh + +.PHONY: setup-auto +setup-auto: ## Same intake with no prompts — best candidate for each, still smoke tested + @bash $(SCRIPTS_DIR)/setup.sh --auto + +.PHONY: setup-show +setup-show: ## Print the conceit.env this machine is using + @bash $(SCRIPTS_DIR)/setup.sh --show + .PHONY: check-env check-env: ## Validate prerequisites: CUDA, Python, uv, GPU, disk - @[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: source scripts/cuda-env.sh"; exit 1; } + @[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: make setup && source scripts/cuda-env.sh"; exit 1; } @[ -d "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME=$(CUDA_HOME) does not exist"; exit 1; } @$(call require_bins,nvidia-smi python) - @[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Install a CUDA-supported gcc (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; } - @[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Install a CUDA-supported g++ (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; } + @[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; } + @[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; } @hdr="$(CUDA_HOME)/include/crt/host_config.h"; \ max=$$(grep -oE '__GNUC__ > [0-9]+' "$$hdr" 2>/dev/null | head -1 | grep -oE '[0-9]+$$' || true); \ have=$$("$(CXX)" -dumpversion 2>/dev/null | cut -d. -f1 || true); \ if [ -n "$$max" ] && [ -n "$$have" ] && [ "$$have" -gt "$$max" ]; then \ echo "ERROR: $(CXX) is gcc $$have, and this CUDA supports up to gcc $$max — nvcc will refuse to compile."; \ - echo " Install gcc $$max and re-source scripts/cuda-env.sh, or export CC/CXX to one."; \ + echo " Run 'make setup': it finds every compiler on this machine and smoke tests the one you pick."; \ exit 1; \ fi @[ -x "$(UV)" ] || { echo "ERROR: uv not found at $(UV). Install: curl -LsSf https://astral.sh/uv/install.sh | sh"; exit 1; } diff --git a/README.md b/README.md index 4ebec9d..58ae8f7 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ No wheel to wait for. No container to inherit. No blog post to follow line by line. ```bash +make setup source scripts/cuda-env.sh make check-env make build-pytorch @@ -29,6 +30,7 @@ conceit is that path, written down and made repeatable. | | | |---|---| +| **Intake** | `make setup` discovers this machine rather than assuming it: every CUDA toolkit and gcc/g++ pair present (including from-source toolchains off `PATH`), the gcc cap read out of the toolkit's own `crt/host_config.h`, then a real `nvcc` compile to prove the pair works. Answers persist to `conceit.env`. | | **Orchestrator** | `build-upstream.sh` clones, syncs submodules with backoff, runs preflight, creates the venv, compiles under a process-tree monitor, and produces wheels. Per-target presets encode the quirks: vLLM's uv + local-torch override that keeps `cuda-toolkit` out of the dep graph, PyTorch's `--no-build-isolation`, llama.cpp's cmake flags. | | **Build state** | `.build-status.json` per target — `init → clone → venv → build_start → build_done → wheel → done`, with the current ninja step and a live count of every descendant process. Poll it instead of grepping logs. | | **Patches** | Generated from the live source tree, never hand-written. Applying is idempotent; a patch that no longer applies is a hard error, not a warning. | @@ -75,20 +77,25 @@ Anything scoring 4–5 gets resolved before you compile. Every row cost a real b ```bash git clone https://github.com/TGPSKI/conceit.git cd conceit -source scripts/cuda-env.sh # CUDA_HOME, TORCH_CUDA_ARCH_LIST=12.0+PTX, host compiler -make check-env # fails fast on anything missing +make setup # find CUDA + compiler on THIS machine, compile-test the pair, write conceit.env +source scripts/cuda-env.sh # picks up conceit.env +make check-env # re-verifies everything, fails fast on anything missing ``` -`make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout. +`make setup` is the intake. It scans for every CUDA toolkit and gcc/g++ pair on the machine — including toolchains built from source and off `PATH`, via `CONCEIT_CC_SEARCH_PATH=/prefix/bin` — shows you what it found with versions, and then **compiles a real CUDA translation unit with the pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a toolkit and a host compiler agree; a version table goes stale one CUDA release later. The answers land in `conceit.env`, which every later shell inherits. + +`make setup-auto` takes the best candidate for each without prompting, still smoke tested. `make setup-show` prints what this machine resolved. `make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout. **Tested on:** Blackwell `sm_120` (RTX PRO 4500), CUDA 13.3, Python 3.14.6 via asdf, gcc/g++ 15, x86_64 Manjaro. Hopper and Ampere work by setting `TORCH_CUDA_ARCH_LIST`. Budget ~200 GB of disk and 32 GB of RAM for parallel nvcc. -**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `cuda-env.sh` picks the newest supported `gcc-*` it finds; `make check-env` fails if the result is missing or too new for the toolkit. +**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `make setup` finds all of these where they actually live on your machine and tells you which one is missing. ## Configure Every knob is an environment variable with a working default, resolved in `scripts/cuda-env.sh`. Export one to override it; nothing needs editing. +Precedence runs in one direction: **what you export by hand** beats **`conceit.env`** (what `make setup` found on this machine) beats **the detection defaults in `cuda-env.sh`**. Every line in `conceit.env` is itself a `${VAR:-default}`, which is what keeps that order honest. + ```bash TORCH_CUDA_ARCH_LIST=9.0+PTX # build for Hopper instead MAX_JOBS=8 # defaults to nproc @@ -99,10 +106,16 @@ CUDA_HOME=/usr/local/cuda-13.2 cuDNN, cuBLASMp, and cuDSS are found by globbing `cuda/` for their extracted archives, so upgrading one means extracting the new tarball and nothing else. +```bash +CONCEIT_CC_SEARCH_PATH=/opt/toolchains/bin # where make setup looks for compilers beyond PATH +CONCEIT_ENV_FILE=/etc/conceit.env # keep the machine's answers somewhere else +``` + ## Go deeper | you want to… | start here | then | |---|---|---| +| **set up** a new machine | [the env-intake skill](.agents/skills/env-intake/SKILL.md) — scan, choose, compile-test, persist | `make setup` · `make setup-show` | | **build** the stack | `make help` — every target, one line each | [AGENTS.md](AGENTS.md) for the invariants · `make env-show` for resolved paths | | **debug** a failed build | [the triage skill](.agents/skills/build-triage/SKILL.md) — intake, adversarial review, coordinate resolution | `make status` · `.build-status.json` per target | | **change** something | [CONTRIBUTING.md](CONTRIBUTING.md) — what `make check` gates and why patches are generated | [CHANGELOG.md](CHANGELOG.md) | diff --git a/scripts/cuda-env.sh b/scripts/cuda-env.sh index e0ed71a..11a7bd7 100755 --- a/scripts/cuda-env.sh +++ b/scripts/cuda-env.sh @@ -18,6 +18,17 @@ fi export CONCEIT_ROOT="${CONCEIT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" export CUDA_STAGE_ROOT="${CUDA_STAGE_ROOT:-$CONCEIT_ROOT/cuda}" +# Machine-specific answers from `make setup`, sourced before any default below. +# Every line in it is itself a ${VAR:-default}, so the precedence chain reads: +# what you exported by hand > what setup discovered on this machine > the +# guesses here. Without this layer every new machine has to rediscover its own +# paths by failing a build. +_conceit_env_file="${CONCEIT_ENV_FILE:-$CONCEIT_ROOT/conceit.env}" +if [[ -s "$_conceit_env_file" ]]; then + source "$_conceit_env_file" +fi +unset _conceit_env_file + if [[ -z "${CUDA_HOME:-}" ]]; then for _d in /usr/local/cuda-13.3 /usr/local/cuda; do [[ -d "$_d" ]] && { export CUDA_HOME="$_d"; break; } diff --git a/scripts/setup.sh b/scripts/setup.sh new file mode 100755 index 0000000..9c17d6e --- /dev/null +++ b/scripts/setup.sh @@ -0,0 +1,361 @@ +#!/usr/bin/env bash +# +# Environment intake. Finds every CUDA toolkit and host compiler on this +# machine, proves the pair you pick actually compiles a CUDA translation unit, +# and writes the answers to conceit.env so every future shell inherits them. +# +# scripts/setup.sh # interactive +# scripts/setup.sh --auto # no prompts: best candidate for each, still smoke tested +# scripts/setup.sh --show # print the current conceit.env and exit +# +# The point is that a machine is discovered, not assumed. Hardcoded defaults +# ("gcc-15 lives in /usr/bin") pass a prerequisite check on a machine that +# cannot compile, and you find out an hour into a build. +# +# Compilers built from source rarely sit on PATH. Point at them with: +# CONCEIT_CC_SEARCH_PATH=/opt/toolchains/bin:/srv/gcc-15/bin scripts/setup.sh + +set -euo pipefail +shopt -s nullglob + +CONCEIT_ROOT="${CONCEIT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +ENV_FILE="${CONCEIT_ENV_FILE:-$CONCEIT_ROOT/conceit.env}" + +interactive=1 + +die() { printf 'error: %s\n' "$*" >&2; exit 1; } +warn() { printf 'WARN %s\n' "$*" >&2; } +say() { printf '%s\n' "$*" >&2; } +hdr() { printf '\n── %s %s\n' "$1" "$(printf '─%.0s' $(seq 1 $((60 - ${#1}))))" >&2; } + +case "${1:-}" in + --auto) interactive=0 ;; + --show) + [[ -f "$ENV_FILE" ]] || die "no $ENV_FILE yet — run: make setup" + cat "$ENV_FILE" + exit 0 + ;; + --help|-h) + sed -n '2,18p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 0 + ;; + '') ;; + *) die "unknown argument: $1 (use --auto, --show, or --help)" ;; +esac + +# A pipe or a cron job has no one to answer the prompts. Fall back rather than +# block forever on a read that will never return. +if [[ ! -t 0 ]] && (( interactive )); then + warn "stdin is not a terminal — running as --auto" + interactive=0 +fi + +# prompt_choice ... +# Renders a menu on stderr and echoes the chosen 1-based index on stdout. +# In --auto mode it echoes the default without rendering anything. +prompt_choice() { + local default=$1; shift + local -a items=("$@") + local i marker reply + + if (( ! interactive )); then + printf '%s' "$default" + return 0 + fi + + for i in "${!items[@]}"; do + marker="" + (( i + 1 == default )) && marker=" <- default" + printf ' %2d) %s%s\n' "$((i + 1))" "${items[i]}" "$marker" >&2 + done + + while true; do + printf 'choice [%s]: ' "$default" >&2 + read -r reply || reply="" + [[ -z "$reply" ]] && reply=$default + if [[ "$reply" =~ ^[0-9]+$ ]] && (( reply >= 1 && reply <= ${#items[@]} )); then + printf '%s' "$reply" + return 0 + fi + say " not one of 1..${#items[@]}" + done +} + +# prompt_value +prompt_value() { + local prompt=$1 default=$2 reply + if (( ! interactive )); then + printf '%s' "$default" + return 0 + fi + printf '%s [%s]: ' "$prompt" "$default" >&2 + read -r reply || reply="" + printf '%s' "${reply:-$default}" +} + +# ── CUDA toolkits ───────────────────────────────────────────────────────────── + +hdr "CUDA toolkit" + +declare -A _seen_cuda=() +declare -a cuda_dirs=() +add_cuda() { + local d=$1 real + [[ -n "$d" && -x "$d/bin/nvcc" ]] || return 0 + real=$(cd "$d" && pwd -P) + [[ -n "${_seen_cuda[$real]:-}" ]] && return 0 + _seen_cuda[$real]=1 + cuda_dirs+=("$real") +} + +for _d in "${CUDA_HOME:-}" /usr/local/cuda /usr/local/cuda-* /opt/cuda /opt/cuda-*; do + add_cuda "$_d" +done +# An nvcc on PATH can live outside every conventional prefix. +if command -v nvcc >/dev/null 2>&1; then + add_cuda "$(cd "$(dirname "$(command -v nvcc)")/.." && pwd -P)" +fi +unset _d + +(( ${#cuda_dirs[@]} )) || die "no CUDA toolkit found (looked for bin/nvcc under /usr/local/cuda*, /opt/cuda*, \$CUDA_HOME, and PATH). + Install one with: scripts/install-cuda-toolkit.sh /path/to/cuda_*.run" + +cuda_version() { + "$1/bin/nvcc" --version 2>/dev/null | awk '/release/ {gsub(",", "", $(NF-1)); print $(NF-1); exit}' +} + +declare -a cuda_labels=() +for _d in "${cuda_dirs[@]}"; do + cuda_labels+=("$(printf 'CUDA %-6s %s' "$(cuda_version "$_d")" "$_d")") +done +unset _d + +# Default to the newest toolkit present. +cuda_default=1 +if (( ${#cuda_dirs[@]} > 1 )); then + _best=$(for i in "${!cuda_dirs[@]}"; do + printf '%s %s\n' "$(cuda_version "${cuda_dirs[i]}")" "$((i + 1))" + done | sort -V | tail -1 | awk '{print $2}') + cuda_default=$_best + unset _best +fi + +say "found ${#cuda_dirs[@]} toolkit(s):" +_pick=$(prompt_choice "$cuda_default" "${cuda_labels[@]}") +CUDA_HOME=${cuda_dirs[$((_pick - 1))]} +cuda_ver=$(cuda_version "$CUDA_HOME") +say "using CUDA $cuda_ver at $CUDA_HOME" + +# nvcc's own host_config.h is the authority on which gcc it will accept — not +# a table in a README that goes stale one CUDA release later. +gcc_cap=$(grep -oE '__GNUC__ > [0-9]+' "$CUDA_HOME/include/crt/host_config.h" 2>/dev/null \ + | head -1 | grep -oE '[0-9]+$' || true) +if [[ -n "$gcc_cap" ]]; then + say "this toolkit accepts host gcc up to $gcc_cap" +else + warn "could not read a gcc cap from $CUDA_HOME/include/crt/host_config.h — the smoke test still decides" +fi + +# ── Host compilers ──────────────────────────────────────────────────────────── + +hdr "host compiler" + +# PATH first, then the places a distro package or a from-source install puts a +# toolchain. A compiler you built yourself is exactly the one PATH does not know +# about, so CONCEIT_CC_SEARCH_PATH is part of the contract, not an escape hatch. +declare -a search_dirs=() +IFS=: read -ra search_dirs <<< "${CONCEIT_CC_SEARCH_PATH:+$CONCEIT_CC_SEARCH_PATH:}$PATH" +search_dirs+=( + /usr/bin /usr/local/bin /opt/bin + /opt/*/bin /opt/*/*/bin + /usr/local/*/bin + "$HOME/.local/bin" "$HOME/opt/"*/bin "$HOME/toolchains/"*/bin + "${ASDF_DATA_DIR:-$HOME/.asdf}/installs/gcc/"*/bin +) + +declare -A _seen_cc=() +declare -a cc_rows=() # "version|gcc|g++" +for _dir in "${search_dirs[@]}"; do + [[ -d "$_dir" ]] || continue + for _cc in "$_dir"/gcc "$_dir"/gcc-[0-9]*; do + [[ -x "$_cc" && -f "$_cc" ]] || continue + + # A gcc with no g++ beside it cannot build torch, so pair them up front. + _cxx="${_cc%/*}/$(basename "$_cc" | sed 's/^gcc/g++/')" + [[ -x "$_cxx" ]] || _cxx="${_cc%/*}/g++" + [[ -x "$_cxx" ]] || continue + + _key="$(readlink -f "$_cc"):$(readlink -f "$_cxx")" + [[ -n "${_seen_cc[$_key]:-}" ]] && continue + _seen_cc[$_key]=1 + + _ver=$("$_cc" -dumpfullversion 2>/dev/null || "$_cc" -dumpversion 2>/dev/null || true) + [[ -n "$_ver" ]] || continue + cc_rows+=("$_ver|$_cc|$_cxx") + done +done +unset _dir _cc _cxx _key _ver + +(( ${#cc_rows[@]} )) || die "no gcc/g++ pair found. + Install one (pacman -S gcc15 | apt install gcc-15 g++-15), or point at a + from-source toolchain: CONCEIT_CC_SEARCH_PATH=/prefix/bin scripts/setup.sh" + +# Newest first, and anything over the cap sinks to the bottom: it is present, +# it is selectable, and it is labelled as the thing nvcc will reject. +declare -a cc_sorted=() +mapfile -t cc_sorted < <( + for row in "${cc_rows[@]}"; do + ver=${row%%|*} + major=${ver%%.*} + rank=0 + if [[ -n "$gcc_cap" ]] && (( major > gcc_cap )); then rank=1; fi + printf '%s\t%s\n' "$rank" "$row" + done | sort -t$'\t' -k1,1n -k2,2Vr | cut -f2 +) + +declare -a cc_labels=() +for row in "${cc_sorted[@]}"; do + IFS='|' read -r _ver _cc _cxx <<< "$row" + note="" + if [[ -n "$gcc_cap" ]] && (( ${_ver%%.*} > gcc_cap )); then + note=" (too new — CUDA $cuda_ver stops at gcc $gcc_cap)" + fi + cc_labels+=("$(printf 'gcc %-10s %s%s' "$_ver" "$_cc" "$note")") +done +cc_labels+=("enter a path manually") +unset row note _ver _cc _cxx + +say "found ${#cc_sorted[@]} compiler(s):" + +CC=""; CXX="" +while true; do + _pick=$(prompt_choice 1 "${cc_labels[@]}") + + if (( _pick == ${#cc_labels[@]} )); then + CC=$(prompt_value " path to gcc" "") + [[ -x "$CC" ]] || { warn "$CC is not executable"; continue; } + CXX=$(prompt_value " path to g++" "${CC%/*}/$(basename "$CC" | sed 's/^gcc/g++/')") + [[ -x "$CXX" ]] || { warn "$CXX is not executable"; continue; } + else + IFS='|' read -r _ver CC CXX <<< "${cc_sorted[$((_pick - 1))]}" + fi + + # ── The gate: a real compile, not a version comparison ──────────────────── + # + # Everything above is inference. This is the only step that proves nvcc and + # this host compiler agree, and it costs about three seconds instead of the + # hour a build spends before failing on the same mismatch. + probe_dir=$(mktemp -d) + cat > "$probe_dir/probe.cu" <<'PROBE' +#include +__global__ void probe_kernel() {} +int main() { probe_kernel<<<1, 1>>>(); std::printf("ok\n"); return 0; } +PROBE + + say "" + say "smoke test: nvcc -ccbin $CXX" + if "$CUDA_HOME/bin/nvcc" -ccbin "$CXX" -c "$probe_dir/probe.cu" -o "$probe_dir/probe.o" \ + > "$probe_dir/out" 2>&1; then + say " compiled clean" + rm -rf "$probe_dir" + break + fi + + say " FAILED:" + sed 's/^/ /' "$probe_dir/out" | head -20 >&2 + rm -rf "$probe_dir" + + (( interactive )) || die "smoke test failed for $CXX and no terminal to pick another" + say "" + say "pick a different compiler:" +done + +say "using gcc $(basename "$CC") -> $CC" + +# ── GPU architecture ────────────────────────────────────────────────────────── + +hdr "GPU architecture" + +arch_default="${TORCH_CUDA_ARCH_LIST:-}" +if [[ -z "$arch_default" ]]; then + if command -v nvidia-smi >/dev/null 2>&1; then + # nvidia-smi prints its failures (a driver/library version mismatch, say) on + # stdout in the same shape as a result. Keep only well-formed compute caps, + # or an unusable driver silently becomes the arch list. + mapfile -t _caps < <(nvidia-smi --query-gpu=compute_cap --format=csv,noheader 2>/dev/null \ + | tr -d ' ' | grep -E '^[0-9]+\.[0-9]+$' | sort -u || true) + if (( ${#_caps[@]} )); then + arch_default=$(printf '%s+PTX;' "${_caps[@]}") + arch_default=${arch_default%;} + say "detected: $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | paste -sd',' -)" + else + warn "nvidia-smi returned no usable compute capability — is the driver healthy?" + fi + unset _caps + fi +fi +if [[ -z "$arch_default" ]]; then + arch_default="12.0+PTX" + warn "no GPU detected — defaulting to $arch_default (leaving this unset compiles every arch, ~9x the nvcc time)" +fi + +TORCH_CUDA_ARCH_LIST=$(prompt_value "TORCH_CUDA_ARCH_LIST" "$arch_default") + +# ── Supporting tools ────────────────────────────────────────────────────────── + +hdr "supporting tools" + +UV="${UV:-$HOME/.local/bin/uv}" +if [[ -x "$UV" ]]; then + say "uv $("$UV" --version)" +elif command -v uv >/dev/null 2>&1; then + UV=$(command -v uv) + say "uv $("$UV" --version) ($UV)" +else + UV="" + warn "uv not found — vLLM builds need it: curl -LsSf https://astral.sh/uv/install.sh | sh" +fi + +if command -v python >/dev/null 2>&1; then + say "python $(python --version 2>&1)" +else + warn "python not found on PATH" +fi + +# ── Write it down ───────────────────────────────────────────────────────────── + +hdr "conceit.env" + +if [[ -f "$ENV_FILE" ]] && (( interactive )); then + say "$ENV_FILE exists:" + sed 's/^/ /' "$ENV_FILE" >&2 + _ans=$(prompt_value "overwrite? [y/N]" "N") + [[ "$_ans" =~ ^[Yy] ]] || { say "left alone"; exit 0; } +fi + +# Every line keeps the ${VAR:-default} form the rest of the repo uses, so the +# precedence chain stays honest: something you exported by hand still beats this +# file, and this file beats the guesses in cuda-env.sh. +# The ${VAR:-...} has to reach the file literally — expanding it here would bake +# in this shell's values and destroy the precedence chain the file exists for. +# shellcheck disable=SC2016 +{ + printf '# Generated by scripts/setup.sh on %s.\n' "$(hostname)" + printf '# Machine-specific answers, sourced by scripts/cuda-env.sh. Not committed.\n' + printf '# Edit freely, or re-run `make setup` to regenerate. Delete to fall back to detection.\n' + printf '\n' + printf 'export CUDA_HOME="${CUDA_HOME:-%s}"\n' "$CUDA_HOME" + printf 'export CC="${CC:-%s}"\n' "$CC" + printf 'export CXX="${CXX:-%s}"\n' "$CXX" + printf 'export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-%s}"\n' "$TORCH_CUDA_ARCH_LIST" + if [[ -n "$UV" ]]; then + printf 'export UV="${UV:-%s}"\n' "$UV" + fi +} > "$ENV_FILE" + +sed 's/^/ /' "$ENV_FILE" >&2 + +hdr "next" +say " source scripts/cuda-env.sh # picks up conceit.env" +say " make check-env # re-verifies everything above" +say " make build-all"