Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions .agents/skills/env-intake/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
---
name: env-intake
description: "Walk a user through conceit's environment intake on a new machine: discover every CUDA toolkit and host compiler present, resolve which pair nvcc will actually accept, prove it with a compile, and persist the answers to conceit.env. Use before the first build on any machine, when check-env fails on CUDA_HOME/CC/CXX, or when a toolchain moved and the build stopped finding it."
metadata:
author: conceit
version: "1.0"
compatibility: "bash, nvcc, gcc/g++, nvidia-smi, uv, make"
---

# conceit Environment Intake

Establish a working build environment on this machine and write it down, so the
next shell and the next agent inherit it instead of rediscovering it.

`scripts/setup.sh` does the mechanical work: scanning, smoke testing, writing
`conceit.env`. This workflow covers what the script cannot decide alone — which
toolkit a machine should standardize on, where an unpackaged toolchain lives,
whether a failed smoke test means "pick another compiler" or "this machine needs
a package."

## Design Principles

- **Discover, never assume.** A hardcoded path passes a prerequisite check on a
machine that cannot compile. Every value here comes from the filesystem or
from the user, never from what worked on another host.
- **The compile is the evidence.** Version tables go stale one CUDA release
later. `nvcc -ccbin <cxx>` on a three-line kernel is DEFINITIVE; a version
comparison is CONFIG. When they disagree, the compile wins.
- **Persist the answer.** An answer that lives only in this session is one the
next machine, shell, or agent pays for again.
- **Ask only what the disk cannot say.** Run the scan before the questions.

## Prerequisites

- A CUDA toolkit installed somewhere on this machine (`scripts/install-cuda-toolkit.sh`
installs one from an NVIDIA runfile, with a gcc shim on PATH).
- A gcc/g++ pair no newer than that toolkit accepts.
- Repo cloned; `make help` runs.

---

## Step 1: Establish the starting state

**Inspect**:

```bash
ls -l conceit.env 2>/dev/null && make setup-show
```

| Status | Action |
|--------|--------|
| `conceit.env` missing | Continue to Step 2 |
| `conceit.env` exists, `make check-env` passes | Report the resolved values, stop. Nothing to do |
| `conceit.env` exists, `check-env` fails | Note which value is wrong, continue to Step 2 — the scan will re-resolve it |
| `conceit.env` exists, toolchain since moved | Continue to Step 2, regenerate |

**Decide**: nothing yet. Do not ask the user anything before the scan runs.

---

## Step 2: Scan the machine

**Inspect**:

```bash
bash scripts/setup.sh --auto # non-interactive: best candidate for each, still smoke tested
```

Read what it reports before interpreting anything: toolkits found, the gcc cap
parsed from that toolkit's own `crt/host_config.h`, compilers found with
versions, and the smoke-test result.

| Status | Action |
|--------|--------|
| One toolkit, one supported compiler, smoke test clean | Accept it. Skip to Step 5 |
| Multiple toolkits found | Step 3 — the user picks |
| Compilers found but all over the cap | Step 4 — this machine needs a package or a from-source toolchain |
| No compiler found at all | Step 4 |
| Smoke test failed on the auto-picked pair | Step 4 — do not paper over it by raising the cap |

**Never** reach for `nvcc -allow-unsupported-compiler` to make a failing smoke
test pass. It converts a clear five-second failure into a compile error hours
into a build, or into a binary that miscompiles at run time.

---

## Step 3: Choose among what was found

**Inspect**: the candidate list from Step 2. Versions and paths are already
resolved — do not re-derive them.

**Decide**:

1. "Which CUDA toolkit should this machine build against?" — default to the
newest present. A machine with 12.x and 13.x installed is usually mid-upgrade;
ask rather than assume the newest is the intended one.
2. "Which host compiler?" — default to the newest at or under the cap.

Then run the interactive intake and make those selections:

```bash
make setup
```

---

## Step 4: Resolve a missing or rejected compiler

**Inspect**:

| Status | Action |
|--------|--------|
| Distro packages a supported gcc | Offer the install command; do not run sudo without confirmation |
| A from-source toolchain exists but is off PATH | Re-scan with `CONCEIT_CC_SEARCH_PATH` |
| A from-source toolchain is too old for the rest of the stack | Say so plainly — rebuilding it is its own project, not a step here |

**Decide**:

1. "Where does your toolchain live?" — only if the scan found nothing usable.
A compiler built from source is precisely the one PATH does not know about.

```bash
CONCEIT_CC_SEARCH_PATH=/prefix/bin:/other/prefix/bin make setup
```

Install commands, when the machine simply lacks one:

| Distro | Command |
|--------|---------|
| Arch / Manjaro | `sudo pacman -S gcc15` |
| Debian / Ubuntu | `sudo apt install gcc-15 g++-15` |
| Fedora | `sudo dnf install gcc-15 gcc-c++-15` |

Re-run Step 2 after any install. The smoke test, not the package manager's exit
code, is what says the machine is ready.

---

## Step 5: Confirm and hand off

**Generate**: `scripts/setup.sh` writes the file. Do not hand-write it.

Every line keeps the `${VAR:-default}` form the rest of the repo uses, so
precedence stays readable: what the user exported by hand beats
`conceit.env`, which beats the detection defaults in `scripts/cuda-env.sh`.

Write to: `conceit.env` (gitignored — it describes one machine)

```bash
export CUDA_HOME="${CUDA_HOME:-/usr/local/cuda-13.3}"
export CC="${CC:-/usr/bin/gcc-15}"
export CXX="${CXX:-/usr/bin/g++-15}"
export TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST:-12.0+PTX}"
export UV="${UV:-/home/you/.local/bin/uv}"
```

## Validate

```bash
source scripts/cuda-env.sh
make check-env # every value above, re-verified from a clean shell
```

| Status | Action |
|--------|--------|
| `check-env` passes | Hand off to `make build-all`, or to the build-triage skill if a build then fails |
| `check-env` disagrees with `conceit.env` | The file is stale or something exported in this shell overrides it — `env \| grep -E 'CC\|CXX\|CUDA_HOME'` |

## PR Checkpoint

Intake produces no committed files — `conceit.env` is gitignored and describes
one machine. Open a PR only when the intake exposed a gap in the tooling itself:

**Title**: `[env] <what the scan could not resolve>`

**Files to include**:
- `scripts/setup.sh` — a search path or toolkit layout the scan should have found
- `scripts/cuda-env.sh` — a default that was wrong for this machine
- `Makefile` — a `check-env` gate that passed on an environment that could not build
1 change: 1 addition & 0 deletions .cursor/skills/env-intake
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ venv/
**/*.egg-info/
**/.eggs/

# ─── Machine-specific environment (written by `make setup`) ──────────────────
conceit.env

# ─── Runtime state files ──────────────────────────────────────────────────────
**/.build-status.json
**/.build-status.json.tmp
Expand Down
17 changes: 16 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ Repository: `github.com/TGPSKI/conceit`

## Before you build anything

On a machine that has never built here, run `make setup` first — it discovers
the CUDA toolkit and host compiler and proves they compile together, which is
the failure this repo has paid for most often. See
[`.agents/skills/env-intake/SKILL.md`](.agents/skills/env-intake/SKILL.md).

Read [`.agents/skills/build-triage/SKILL.md`](.agents/skills/build-triage/SKILL.md)
first. It is not background reading — it is the procedure. The single highest-value
rule in it: **coordinates before compilation.** Read the target repo's own
Expand Down Expand Up @@ -51,12 +56,15 @@ wrong is measured in hours of nvcc, not seconds.

```
scripts/
setup.sh # environment intake — scan, smoke test, write conceit.env
cuda-env.sh # every env default; sourced by everything else
build-upstream.sh # the orchestrator — presets, preflight, monitor, status
gen-patches.sh # export live src/ diffs into patches/
install-cuda-toolkit.sh # CUDA runfile install behind a gcc-15 shim
patches/<target>/ # generated; committed; replayed by make patch-*
.agents/skills/build-triage/SKILL.md
.agents/skills/env-intake/SKILL.md
conceit.env # gitignored — this machine's answers, written by make setup
src/ # gitignored — cloned upstream trees live here
```

Expand All @@ -73,7 +81,14 @@ src/ # gitignored — cloned upstream trees live here
and `clone` only show up on a re-run. Emitting it is best-effort by design:
a status write must never take down a build that has been running for hours.
- **Every env var takes the form `${VAR:-default}`**, so any single value can be
overridden without editing a file. Keep it that way.
overridden without editing a file. Keep it that way. `conceit.env` (written by
`make setup`, gitignored) is sourced first and uses the same form, so precedence
reads: exported by hand > this machine's intake > the defaults in `cuda-env.sh`.
- **Discover the machine, do not assume it.** A hardcoded path is what lets
`check-env` pass on a host that cannot compile. `scripts/setup.sh` scans for
toolkits and compilers and then *compiles a CUDA translation unit* with the pair
it picked — the only evidence that outranks a version comparison. New
prerequisites belong in that scan, not in a new hardcoded default.
- **Paths derive from the repo root**, computed from the script's own location.
Never reintroduce a hardcoded `$HOME/...` path — it breaks every clone that
isn't yours.
Expand Down
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,30 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.

## [Unreleased]

### Added

- **`make setup` — environment intake.** Every prerequisite in v0.1.0 was a
hardcoded default that happened to be true on one workstation, and `check-env`
only validated those assumptions rather than discovering the machine. There was
nowhere to record a per-machine answer, so each new host rediscovered its own
paths by failing a build. `scripts/setup.sh` finds every CUDA toolkit and
gcc/g++ pair present — including toolchains built from source and off `PATH`,
via `CONCEIT_CC_SEARCH_PATH` — reads the gcc cap out of the chosen toolkit's own
`crt/host_config.h`, and then **compiles a real CUDA translation unit with the
pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a
toolkit and a host compiler agree; when it disagrees with a version table, the
compile wins. Answers persist to `conceit.env` (gitignored), sourced by
`cuda-env.sh` ahead of every default and written in the same `${VAR:-default}`
form, so precedence reads: exported by hand > this machine's intake > the
defaults. `make setup-auto` skips the prompts, `make setup-show` prints the
result.
- **`.agents/skills/env-intake/SKILL.md`** — the directed workflow around that
script, for the parts a scan cannot decide: which toolkit a machine should
standardize on, where an unpackaged toolchain lives, and whether a failed smoke
test means "pick another compiler" or "this machine needs a package." It also
says plainly not to reach for `-allow-unsupported-compiler`, which converts a
five-second failure into a corrupt build hours later.

### Fixed

- **The first build on a fresh clone always failed.** `emit_status` writes
Expand Down
20 changes: 16 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -103,19 +103,31 @@ endef

# ── Environment ───────────────────────────────────────────────────────────────

.PHONY: setup
setup: ## Interactive intake: find CUDA + compiler, prove they compile, write conceit.env
@bash $(SCRIPTS_DIR)/setup.sh

.PHONY: setup-auto
setup-auto: ## Same intake with no prompts — best candidate for each, still smoke tested
@bash $(SCRIPTS_DIR)/setup.sh --auto

.PHONY: setup-show
setup-show: ## Print the conceit.env this machine is using
@bash $(SCRIPTS_DIR)/setup.sh --show

.PHONY: check-env
check-env: ## Validate prerequisites: CUDA, Python, uv, GPU, disk
@[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: source scripts/cuda-env.sh"; exit 1; }
@[ -n "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME not set. Run: make setup && source scripts/cuda-env.sh"; exit 1; }
@[ -d "$(CUDA_HOME)" ] || { echo "ERROR: CUDA_HOME=$(CUDA_HOME) does not exist"; exit 1; }
@$(call require_bins,nvidia-smi python)
@[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Install a CUDA-supported gcc (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; }
@[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Install a CUDA-supported g++ (pacman -S gcc15 | apt install gcc-15 g++-15), or export CC/CXX"; exit 1; }
@[ -x "$(CC)" ] || { echo "ERROR: CC=$(CC) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; }
@[ -x "$(CXX)" ] || { echo "ERROR: CXX=$(CXX) is not an executable compiler. Run 'make setup' to find one on this machine, or export CC/CXX"; exit 1; }
@hdr="$(CUDA_HOME)/include/crt/host_config.h"; \
max=$$(grep -oE '__GNUC__ > [0-9]+' "$$hdr" 2>/dev/null | head -1 | grep -oE '[0-9]+$$' || true); \
have=$$("$(CXX)" -dumpversion 2>/dev/null | cut -d. -f1 || true); \
if [ -n "$$max" ] && [ -n "$$have" ] && [ "$$have" -gt "$$max" ]; then \
echo "ERROR: $(CXX) is gcc $$have, and this CUDA supports up to gcc $$max — nvcc will refuse to compile."; \
echo " Install gcc $$max and re-source scripts/cuda-env.sh, or export CC/CXX to one."; \
echo " Run 'make setup': it finds every compiler on this machine and smoke tests the one you pick."; \
exit 1; \
fi
@[ -x "$(UV)" ] || { echo "ERROR: uv not found at $(UV). Install: curl -LsSf https://astral.sh/uv/install.sh | sh"; exit 1; }
Expand Down
21 changes: 17 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
No wheel to wait for. No container to inherit. No blog post to follow line by line.

```bash
make setup
source scripts/cuda-env.sh
make check-env
make build-pytorch
Expand All @@ -29,6 +30,7 @@ conceit is that path, written down and made repeatable.

| | |
|---|---|
| **Intake** | `make setup` discovers this machine rather than assuming it: every CUDA toolkit and gcc/g++ pair present (including from-source toolchains off `PATH`), the gcc cap read out of the toolkit's own `crt/host_config.h`, then a real `nvcc` compile to prove the pair works. Answers persist to `conceit.env`. |
| **Orchestrator** | `build-upstream.sh` clones, syncs submodules with backoff, runs preflight, creates the venv, compiles under a process-tree monitor, and produces wheels. Per-target presets encode the quirks: vLLM's uv + local-torch override that keeps `cuda-toolkit` out of the dep graph, PyTorch's `--no-build-isolation`, llama.cpp's cmake flags. |
| **Build state** | `.build-status.json` per target — `init → clone → venv → build_start → build_done → wheel → done`, with the current ninja step and a live count of every descendant process. Poll it instead of grepping logs. |
| **Patches** | Generated from the live source tree, never hand-written. Applying is idempotent; a patch that no longer applies is a hard error, not a warning. |
Expand Down Expand Up @@ -75,20 +77,25 @@ Anything scoring 4–5 gets resolved before you compile. Every row cost a real b
```bash
git clone https://github.com/TGPSKI/conceit.git
cd conceit
source scripts/cuda-env.sh # CUDA_HOME, TORCH_CUDA_ARCH_LIST=12.0+PTX, host compiler
make check-env # fails fast on anything missing
make setup # find CUDA + compiler on THIS machine, compile-test the pair, write conceit.env
source scripts/cuda-env.sh # picks up conceit.env
make check-env # re-verifies everything, fails fast on anything missing
```

`make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout.
`make setup` is the intake. It scans for every CUDA toolkit and gcc/g++ pair on the machine — including toolchains built from source and off `PATH`, via `CONCEIT_CC_SEARCH_PATH=/prefix/bin` — shows you what it found with versions, and then **compiles a real CUDA translation unit with the pair you pick**. That three-second `nvcc -ccbin` is the only check that proves a toolkit and a host compiler agree; a version table goes stale one CUDA release later. The answers land in `conceit.env`, which every later shell inherits.

`make setup-auto` takes the best candidate for each without prompting, still smoke tested. `make setup-show` prints what this machine resolved. `make help` lists every target. Nothing is installed system-wide; the source trees and venvs live under `src/` in the checkout.

**Tested on:** Blackwell `sm_120` (RTX PRO 4500), CUDA 13.3, Python 3.14.6 via asdf, gcc/g++ 15, x86_64 Manjaro. Hopper and Ampere work by setting `TORCH_CUDA_ARCH_LIST`. Budget ~200 GB of disk and 32 GB of RAM for parallel nvcc.

**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `cuda-env.sh` picks the newest supported `gcc-*` it finds; `make check-env` fails if the result is missing or too new for the toolkit.
**You need:** the CUDA 13.3 toolkit (`scripts/install-cuda-toolkit.sh` handles the gcc-15 shim), a gcc no newer than your CUDA supports (13.3 caps at 15 — `pacman -S gcc15`, `apt install gcc-15 g++-15`), `uv`, and — for the full torchvision/torchaudio builds — `libjpeg-turbo libpng ffmpeg sox`. `make setup` finds all of these where they actually live on your machine and tells you which one is missing.

## Configure

Every knob is an environment variable with a working default, resolved in `scripts/cuda-env.sh`. Export one to override it; nothing needs editing.

Precedence runs in one direction: **what you export by hand** beats **`conceit.env`** (what `make setup` found on this machine) beats **the detection defaults in `cuda-env.sh`**. Every line in `conceit.env` is itself a `${VAR:-default}`, which is what keeps that order honest.

```bash
TORCH_CUDA_ARCH_LIST=9.0+PTX # build for Hopper instead
MAX_JOBS=8 # defaults to nproc
Expand All @@ -99,10 +106,16 @@ CUDA_HOME=/usr/local/cuda-13.2

cuDNN, cuBLASMp, and cuDSS are found by globbing `cuda/` for their extracted archives, so upgrading one means extracting the new tarball and nothing else.

```bash
CONCEIT_CC_SEARCH_PATH=/opt/toolchains/bin # where make setup looks for compilers beyond PATH
CONCEIT_ENV_FILE=/etc/conceit.env # keep the machine's answers somewhere else
```

## Go deeper

| you want to… | start here | then |
|---|---|---|
| **set up** a new machine | [the env-intake skill](.agents/skills/env-intake/SKILL.md) — scan, choose, compile-test, persist | `make setup` · `make setup-show` |
| **build** the stack | `make help` — every target, one line each | [AGENTS.md](AGENTS.md) for the invariants · `make env-show` for resolved paths |
| **debug** a failed build | [the triage skill](.agents/skills/build-triage/SKILL.md) — intake, adversarial review, coordinate resolution | `make status` · `.build-status.json` per target |
| **change** something | [CONTRIBUTING.md](CONTRIBUTING.md) — what `make check` gates and why patches are generated | [CHANGELOG.md](CHANGELOG.md) |
Expand Down
Loading