fix(tool-installer): 幂等安装 cargo-install 工具 + setup-new.sh 自动更新#9
Merged
Conversation
…ecture)
Layer 0 (bootstrap): scripts/layer0-bootstrap.sh
- Install Python, gh CLI, tool-installer single-file artifact
- Prompt gh auth login for GitHub API rate limit avoidance
Layer 1 (tool-installer): tools.toml + manifest.toml
- 50 tools across 6 modules: build-base, cargo-tools, editors,
languages, lsp-servers, extras
- 6 manager types: rustup, cargo-install, github-release, mise,
npm-global, uv-tool
- [_github-release] mirror config + token auto-detection
- All cargo tools pinned to exact versions from install-functions.sh
Layer 2 (post): scripts/layer2-post.sh
- Helix runtime compilation, yazi plugins, font cache refresh
CI: .github/workflows/tool-installer-verify.yml
- Dry-run verification, bootstrap test, single-module apply test
vendor/tool-installer: 32KB single-file Python artifact (3.8+)
Each module gets its own CI job running in parallel: - build-base: Rust toolchain (30min) - cargo-tools: 29 crates (120min, the bottleneck) - editors: neovim/helix/zellij/yq (20min) - languages: go/node/pnpm/zig/yazi via mise (30min) - lsp-servers: npm/uv/github-release LSPs (30min) - extras: xdotter/cargo-binstall (20min) - full-install: end-to-end after all modules pass
…tests - languages: install mise via curl before tool-installer - editors: add ~/.local/bin to PATH for github-release installs - lsp-servers: use actions/setup-node + pip install uv - full-install: combine all prerequisites - All jobs: export PATH for ~/.local/bin and ~/.cargo/bin
Editors (neovim, helix, zellij, yq) are all github-release downloads that don't need Rust. Removing the dependency lets them install independently on CI runners without rustup.
- helix: bin path 'hx' → 'helix-{version}-x86_64-linux/hx' (archive has directory)
- lsp-servers: remove depends=[languages] (CI installs node/uv separately)
- extras: remove depends=[build-base] (independent github-release tools)
- CI: add mise shims to PATH for languages job
Helix archive directory contains the version number (e.g.,
helix-25.07.1-x86_64-linux/hx). The bin field now supports the same
placeholders as asset: {tool}, {version}, {os}, {arch}.
Also rebuilds vendor/tool-installer with the fix.
…ame step Mise activation doesn't persist across GitHub Actions steps. Combine install + verify in a single step with eval mise activate.
- lsp-servers: pre-install npm packages directly, then tool-installer handles github-release tools (marksman, zls, lua-lsp) - languages: use mise trust + explicit shims PATH
Same fix as lsp-servers job - npm-global check queries registry which fails in CI. Pre-install packages so tool-installer skips check.
Add binstall_first = true to all cargo-install entries in manifest.toml. This enables cargo-binstall precompiled binary downloads with automatic fallback to source compilation, reducing install time from ~48 minutes to an estimated 8-15 minutes. CI will verify the acceleration is effective in the cargo-tools job.
The previous commit added binstall_first to manifest.toml, but the vendor artifact was still the old version that does not recognize the new field. This update replaces the vendor executable with the latest build that includes binstall_first support.
The previous binstall_first implementation caused CI cargo-tools to take 1h2m (worse than 48m without binstall) because cargo binstall hung on tools without prebuilt binaries. This update adds a 120s timeout to the binstall subprocess so it falls back to cargo install when it hangs.
Previous 120s timeout caused 1h2m CI time (worse than 48m without binstall). Reduced to 30s to limit worst-case overhead to 14.5m while still allowing binstall to succeed for tools with prebuilt binaries.
binstall_first optimization did not work in CI: - Previous run with binstall_first: 1h2m (worse than 48m without it) - With 30s timeout: ~50m and still running (same as 48m baseline) Root cause: In GitHub Actions runners, cargo binstall either fails quickly for tools without prebuilt binaries, or hangs for tools with binaries due to network/QuickInstall issues. The 30s timeout per tool adds up to 14.5m worst-case overhead for 29 tools, negating any speedup from successful downloads. The feature remains available for local use where users have binstall installed and good network connectivity. CI should use pure cargo-install for consistency.
Swatinem/rust-cache@v2 caches cargo registry, git db, and build artifacts between CI runs. This should significantly speed up subsequent cargo-tools jobs after the initial cold build.
With the gh release download fix in tool-installer, binstall_first should now work in CI because gh has GITHUB_TOKEN auth (5000 req/hr rate limit) instead of the previous urllib which was rate-limited to 60 req/hr. Expected: cargo-tools drops from 48m (pure cargo install) to ~5-15m (binstall download for tools with prebuilt binaries).
cargo-binstall subprocess now inherits GITHUB_TOKEN from gh auth (if not already set by user), enabling authenticated GitHub API requests in both CI and local environments.
Split build-base into two tools: - rustup-install: uses vendor rustup-init.sh via script manager - rust: uses rustup manager for toolchain/component configuration Changes: - tools.toml: add rustup-install tool in build-base module - manifest.toml: configure rustup-install (script) and rust (rustup) - scripts/install-rustup.sh: wrapper that calls vendor script with -y flag - vendor/tool-installer: update with PATH auto-include for ~/.cargo/bin Verified: independent user test shows rustup installed, components configured.
… github-release - Move cargo-binstall from extras to standalone layer before cargo-tools - Make cargo-tools depend on cargo-binstall so binstall_first works - Move starship from cargo-tools to new shell-tools layer - Change starship manager from cargo-install to github-release (avoids rsproxy SSL issues, uses ghproxy mirror) - Update CI: add apply-cargo-binstall and apply-shell-tools jobs - Update dry-run loop to include new modules
Fix four blocking issues in setup-new.sh that prevented it from completing like the main branch's setup.sh: - ensure_xdotter: three-layer fallback (tool-installer → curl → vendor) so xdotter install never fails from GitHub rate limits - install_fonts: add font installation in do_deploy, matching setup.sh - cargo config patch: temporarily disable sccache/wild during do_install to prevent "rustc-wrapper not found" blocking all subsequent cargo tools - add DEBIAN_FRONTEND/TZ exports to prevent apt interactive prompts Pin all tool versions in tools.toml and mise/config.toml (46 tools, zero 'latest' remaining). Fix v-prefix mismatches for github-release tools whose release tags use v prefix (neovim, zellij, yq, starship, xdotter, cargo-binstall). Vendor cargo-binstall and xdotter binaries to bypass GitHub download from slow networks. Change uv from cargo-install to script manager using astral.sh CDN. Change zola from cargo-install (git source, blocked by CHECK_ERROR bug) to github-release (pre-built binary). Update tool-installer vendor with CHECK_ERROR parser fix for git-source cargo-install checks. Add rsproxy.cn mirror for rustup. Add ghfast.top as preferred GitHub mirror. Fix Dockerfile sources.list.d cleanup order (must precede apt-get update). Update Dockerfile and CI workflow to use setup-new.sh. Co-authored-by: tool-installer dev team
- ensure_xdotter: only use vendor/xdotter on Linux x86_64, avoiding 'cannot execute binary file' on macOS (exit code 126) - pyright: mark as allow_fail (uv-tool check fails in CI)
- ensure_xdotter: wrap curl download and vendor copy in Linux x86_64 guards to prevent downloading/running Linux binaries on macOS (exit code 126) - ensure_xdotter: return 0 with warning on macOS when all install methods fail (tool-installer API failure in CI) - do_deploy: check for xd existence before running deploy, skip gracefully when xdotter unavailable on macOS
macOS runner failed with 'Missing macos strategy for tool: rustup-install'. Add macos strategy using the same vendor script (which already handles macOS via curl + official rustup installer).
- manifest.toml: add [rust.macos] strategy matching [rust.linux] - runner-verify.yml: temporarily disable macos-latest in CI matrix (~43 tools in dev module only have .linux strategies; adding .macos for all is a separate scope item tracked as TODO) This prevents the CI from failing one-by-one for each tool missing a macos strategy.
- Remove vendored cargo-binstall binary; tool-installer now handles binstall bootstrapping itself (upstream fix: use -V for verification). - Update vendor/tool-installer to upstream commit 8e6d95a which fixes cargo-binstall verification CLI compatibility. - Remove cargo-binstall preinstall logic from layer0-bootstrap.sh. - Update scripts/vendor/README.md with vendor scope/policy. - Add docs/tool-installer-migration-plan.md documenting the three-layer architecture and vendor strategy.
- Add .macos strategies to manifest.toml for all 50+ tools - Enable macos-latest in E2E Full Install CI workflow - Fix sed portability (BSD vs GNU) in setup-new.sh - Add fzf/ripgrep/tree/git to macOS bootstrap via Homebrew - Compute SHA256 for macOS aarch64 GitHub Release assets
yq outputs 'version v4.52.4' but regex expected 'version 4.52.4', causing CHECK_ERROR that blocks the entire install pipeline.
- Add scripts/mise-install.sh (MISE_YES=1 mise trust --silent && mise install) - Remove [languages] module from tools.toml - Add [mise-install] script job to manifest.toml - [dev] depends now includes mise-install instead of languages
setup-new.sh's sudo_run was designed for CI (NOPASSWD only), but broke local usage where users expect interactive password prompt. Match setup.sh behavior: use plain sudo and let the system prompt.
Bootstrap was a separate script with duplicated sudo_run helper that also had the NOPASSWD-only design (now fixed). Merge all layer0 logic (system packages, wezterm, gh login, tool-installer) directly into setup-new.sh --bootstrap. Delete scripts/layer0-bootstrap.sh and update CI workflow.
cp fails with 'same file' error when the target is a symlink or hard link to the source. install -m 755 handles this case reliably and is POSIX-standard across Linux and macOS.
- Add [system-packages] module with script manager (apt/brew) - Add [wezterm] module with script manager (apt.fury.io/brew cask) - Thin bootstrap: only installs tool-installer binary from vendor - Move system packages + WezTerm out of do_bootstrap() - Update dev depends to include system-packages - Both scripts use 'exit 0' on failure (non-blocking) This follows the principle: bootstrap should only install tool-installer itself; all other packages are managed declaratively.
- Add [fonts] module to tools.toml and manifest.toml - Create scripts/install-fonts.sh (apt/brew + FiraCode Nerd Font) - Remove install_fonts() and sudo_run() from setup-new.sh - Update bootstrap comment to reflect thin bootstrap (only tool-installer binary) Fonts now follow the same pattern as system-packages and WezTerm: managed declaratively by tool-installer, not inline in setup-new.sh.
'local' is only valid inside functions. Using it at the top level of a script causes 'can only be used in a function' error with set -u (nounset) enabled.
zls --version outputs '0.15.1', not 'zls 0.15.1'. The regex 'zls ...' would never match, causing 'Check failed for zls' on every install attempt.
Runs 'tool-installer install dev' a second time on a system where all tools are already installed. Catches bugs like the zls version_probe regex that only manifest when version_probe is executed (i.e., when the binary already exists).
When a mirror returns content with a different SHA256 than expected, the installer now discards the file and tries the next source (mirror or official GitHub) instead of failing immediately. Changes: - github_release.py: _download_asset() now accepts expected_sha256 parameter; verifies checksum after each download and raises InstallationError on mismatch (caught by outer loop, triggers fallback to next URL) - install(): passes SHA256 to _download_asset(), removes separate _verify_checksum() call - _verify_checksum() preserved for backward compatibility This fixes the issue where Chinese mirrors may serve altered/cached release assets, causing installation to abort even though the official source has the correct file. Also adds docs/ci-issue-tracker.md documenting all CI/local issues found during this migration.
- scripts/layer2-post.sh: add GitHub mirror fallback for Helix runtime download, dpkg lock wait for LLVM install, graceful failure handling - scripts/install-fonts.sh: dpkg lock wait, retry loop, remove sudo for root execution - shells/common/install-functions.sh: Yazi plugins install with 3-retry and git mirror fallback - setup-new.sh: set MISE_HTTP_TIMEOUT=300 to prevent pnpm download timeout - langs/rust/cargo/config.toml: restore sccache/wild config (was temporarily disabled during install) - vendor/tool-installer: rebuilt with npm-global/uv-tool check fixes and --tool flag support
- install-fonts.sh: 用户级字体安装，plain sudo 替代 sudo -n，GitHub 镜像回退 - install-lua-lsp.sh: 新增脚本，完整解压 archive 解决 lua-language-server 运行时依赖缺失 - manifest.toml: lua-lsp 改为 script manager - install-functions.sh: Yazi 插件用 GIT_CONFIG_COUNT 零副作用注入镜像规则 + 多镜像 fallback - layer2-post.sh: GITHUB_MIRRORS 默认值统一加入 ghfast.top
Rewrite _ensure_tool_installer() to handle all edge cases: 1. Detect hardlink (-ef test): source and target same inode 2. Detect version mismatch (cmp -s comparison) 3. Multi-level write fallback: - install -m 755 (preferred, handles hardlink/symlink) - rm + install (handles file-in-use) - cat + chmod (last resort) 4. Post-write verification: cmp -s confirms file integrity 5. Detailed logging for each scenario This ensures ./setup-new.sh works correctly on both new and old machines, regardless of entry point (--bootstrap, --install, or no args). Container verification confirmed: - Old version detected and auto-updated - Check() correctly skips already-installed tools
…vate) - env.sh: keep mise hook-env for non-interactive shells (PATH immediately available) - inter.sh: add mise activate for interactive shells (loads [env] vars) - CI: add verify step for STEP_API_KEY in new interactive shell Fixes STEP_API_KEY not visible in WSL new terminals
mise is installed via cargo-install but may not be ready during cargo-tools verification. The proper verification belongs in the languages module where mise-install runs.
…se || true masking rust-cache restores .crates.toml from previous runs, causing cargo-binstall to skip installation (trusts metadata without verifying binary exists). Fix: rm .crates.toml/.crates2.json before cargo-tools install in all 3 jobs. Also: remove mise --version || true (should pass now), fix kondo verification to use --help (tool has no --version flag).
Verify tool-installer works on both Linux and macOS via E2E tests. Individual module jobs stay Linux-only (sufficient for pinpointing failures). - full-install: matrix [ubuntu-latest, macos-latest] - re-install: matrix [ubuntu-latest, macos-latest] - fail-fast: false so both platforms run independently
Pass --yes to brew install in install-system-packages.sh and install-fonts.sh to avoid interactive confirmation prompts during automated setup. Update vendor/tool-installer with matching fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
重新运行
setup-new.sh时，通过cargo binstall安装的工具（如wild-linker、bat）报错：根因：
CargoInstallManager.check()使用cargo install --list检查，但该命令不跟踪cargo binstall安装的工具，导致 check() 返回 NOT_SATISFIED，install() 被调用后报错。修复
1. tool-installer 源码修复 (
vendor/tool-installer)CargoInstallManager.check()重写：从cargo install --list改为文件系统检查~/.cargo/bin/<binary>+<binary> --version解析_parse_binary_version()：新增，处理多种--version输出格式_latest_registry_version()：添加--registry crates-io参数，绕过rsproxy-sparse替换导致的cargo search失败_cargo_bin_dirs()：新增，返回~/.cargo/bin和~/.local/bin_fallback_check_via_cargo_list()保留旧逻辑作为兜底2. manifest.toml 添加
bin字段5 个 crate 名与二进制名不一致的工具需要显式声明：
fd-find→bin = "fd"gen-mdbook-summary→bin = "gms"parallel-disk-usage→bin = "pdu"tree-sitter-cli→bin = "tree-sitter"wild-linker→bin = "wild"3. setup-new.sh 自动更新
_ensure_tool_installer()-ef）、版本不匹配（cmp -s）install -m 755→rm + install→cat + chmodcmp -s校验CI 验证目标
full-install：首次安装全量 dev 模块re-install：已安装环境再次 install，所有工具应被跳过（不应报 "binary already exists"）提交记录
d0efc32fix(cargo-install): redesign check() with filesystem-based binary detection9774efffix(setup-new): robust _ensure_tool_installer with multi-level fallback