From 9bd2e4accbe1dc302a54f9cc2957ed4861f53d7a Mon Sep 17 00:00:00 2001 From: TGPSKI Date: Tue, 4 Aug 2026 00:21:30 -0700 Subject: [PATCH] Fix the first build on a fresh clone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `emit_status` writes .build-status.json into the target's source tree, but the init phase fires before the clone has created that directory. On any checkout that had never been built, `make build-pytorch` died instantly with `src/pytorch/.build-status.json.tmp: No such file or directory` — v0.1.0 was unbuildable for every new user, and only worked here because src/ already existed. Status emission now skips until the tree is a git repo, and never fails the build if the write does not land. Two reasons for the skip rather than an mkdir: the directory has to stay empty until the clone lands, because git refuses to clone into a directory that already has files in it, and a status write must never take down a build that has been running for hours. That also unbreaks the case where the destination exists but is empty — the old code seeded a status file into it and the clone then refused the now-non-empty directory. The pytorch preset was missing two build requirements behind that failure. Upstream builds through scikit-build-core now, and --no-build-isolation installs nothing on our behalf, so numpy and scikit-build-core have to be seeded into the venv alongside the rest of [build-system] requires. Without them the build fails at backend import, before a single file compiles. Verified against a local throwaway repo on all three paths: fresh clone, re-run/pull, and pre-existing empty destination. `make check` passes with the CI-pinned shellcheck 0.11.0. --- AGENTS.md | 5 ++++- CHANGELOG.md | 17 +++++++++++++++++ scripts/build-upstream.sh | 24 ++++++++++++++++++++---- 3 files changed, 41 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7e60ad3..e9d5e17 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,7 +68,10 @@ src/ # gitignored — cloned upstream trees live here - **`.build-status.json`** is written atomically (`tmp` → `mv`) so a reader never sees a half-written file. Phases: `init → clone → venv → prebuild → build_start → building → build_done → wheel → done`, or `failed`. Anything - interpolated into it goes through `json_escape`. + interpolated into it goes through `json_escape`. It lives inside the source + tree, so on a first build nothing is written until the clone lands — `init` + and `clone` only show up on a re-run. Emitting it is best-effort by design: + a status write must never take down a build that has been running for hours. - **Every env var takes the form `${VAR:-default}`**, so any single value can be overridden without editing a file. Keep it that way. - **Paths derive from the repo root**, computed from the script's own location. diff --git a/CHANGELOG.md b/CHANGELOG.md index bbbf757..8537979 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,23 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0. ## [Unreleased] +### Fixed + +- **The first build on a fresh clone always failed.** `emit_status` writes + `.build-status.json` into the target's source tree, but the `init` phase fires + before the clone has created that directory — so `make build-pytorch` died + instantly with `src/pytorch/.build-status.json.tmp: No such file or directory` + on any checkout that had never been built. Status emission now skips until the + tree is a git repo, and never fails the build if the write does not land. This + also unbreaks the case where the destination directory exists but is empty: + the old code seeded a status file into it, and `git clone` then refused the + now-non-empty directory. +- **The pytorch preset was missing two build requirements.** Upstream builds + through `scikit-build-core` now, and `--no-build-isolation` installs nothing on + our behalf, so `numpy` and `scikit-build-core` have to be seeded into the venv + alongside the rest of `[build-system] requires`. Without them the build fails + at backend import, before a single file compiles. + ## [0.1.0] - 2026-08-03 First public release. conceit began as the private build orchestration behind a diff --git a/scripts/build-upstream.sh b/scripts/build-upstream.sh index 43189c5..39f9f9d 100755 --- a/scripts/build-upstream.sh +++ b/scripts/build-upstream.sh @@ -59,11 +59,23 @@ json_escape() { emit_status() { local phase=$1 step=${2:-null} jobs=${3:-0} detail=${4:-} - [[ -z "$_status_file" ]] && return - printf '{"phase":"%s","ts":"%s","step":"%s","jobs":%s,"detail":"%s"}\n' \ + [[ -z "$_status_file" ]] && return 0 + + # The status file lives inside the source tree, which does not exist yet on a + # fresh checkout — and must stay empty until the clone lands, because git + # refuses to clone into a directory that already has files in it. So skip the + # write until the tree is a repo: the init/clone phases have nowhere to go. + # Never fail here either; a status write must not kill a three-hour build. + [[ -d "${_status_file%/*}/.git" ]] || return 0 + + if printf '{"phase":"%s","ts":"%s","step":"%s","jobs":%s,"detail":"%s"}\n' \ "$(json_escape "$phase")" "$(ts_iso)" "$(json_escape "$step")" \ "$jobs" "$(json_escape "$detail")" \ - > "${_status_file}.tmp" && mv "${_status_file}.tmp" "$_status_file" + > "${_status_file}.tmp" 2>/dev/null + then + mv -f "${_status_file}.tmp" "$_status_file" 2>/dev/null || true + fi + return 0 } # Count all descendants of a PID (not just direct children). @@ -219,7 +231,11 @@ case "$preset" in pytorch) build_cmd=${BUILD_CMD:-"MAX_JOBS=${MAX_JOBS} NVCC_THREADS=${NVCC_THREADS} python -m pip install -e . -v --no-build-isolation"} build_dist_cmd=${BUILD_DIST_CMD:-"python -m pip wheel --no-build-isolation --no-deps -w dist ."} - extra_pip_deps=${EXTRA_PIP_DEPS:-"cmake ninja packaging pyyaml typing_extensions six"} + # --no-build-isolation means pip installs nothing on our behalf: every entry + # in pytorch's [build-system] requires has to already be in the venv. numpy + # and scikit-build-core (the build backend since the setup.py retirement) + # are load-bearing — without them the build dies before the first compile. + extra_pip_deps=${EXTRA_PIP_DEPS:-"cmake ninja packaging pyyaml typing_extensions six numpy scikit-build-core"} ;; vllm) # vllm uses uv (per AGENTS.md). We override the torch==2.11.0 pin with our