From 6460a628e93921a2bb5aa32587f1f96133fdc9a9 Mon Sep 17 00:00:00 2001 From: Christopher Maher Date: Sat, 23 May 2026 16:44:29 -0700 Subject: [PATCH 1/2] feat(homebrew,kubernetes): switch default Docker runtime from docker-desktop to colima Docker Desktop's cask postinstall hook tries to sudo-write to multiple /usr/local/* directories that do not exist on a fresh Apple Silicon Mac. PR #7 fixed one (/usr/local/cli-plugins); the very next bootstrap run hit the next (/usr/local/bin). Patching each path one at a time is whack-a-mole. Switch the default Docker runtime to **colima** (#8): - colima runs entirely in user space (Lima VM at ~/.colima/), no /usr/local/* writes, no sudo prompts during cask install - Matches the convention already in use on other LLMKube hosts - Smaller resident memory footprint than Docker Desktop -- useful on Macs that also serve a 30B-class model locally via mlx-server Changes: - roles/homebrew/defaults/main.yml: homebrew_casks defaults to []. Add colima + docker + docker-compose + docker-buildx to the homebrew_cli_packages list. Docs callout in the comment explains the switch + how to opt back into docker-desktop via group_vars. - roles/kubernetes/defaults/main.yml: new file. colima_cpu (4), colima_memory_gib (6), colima_disk_gib (60) defaults sized for a single-node kind cluster plus a couple of small workloads. Each overridable via group_vars or --extra-vars. - roles/kubernetes/tasks/main.yml: replace 'Ensure Docker Desktop is running' (open -ga Docker) with a colima status check + a conditional that respects the new sizing defaults. Idempotent: skips when colima is already running. - README.md: 'What it sets up' section now describes colima as the default Docker provider. New 'Docker Desktop on Apple Silicon (opt-in only)' callout documents how to opt back in and the two caveats (cli-plugins handled by the pre-task; /usr/local/bin needs manual sudo mkdir + chown). Backward compat: - PR #7's pre-create-cli-plugins task stays in place (its clause already no-ops when homebrew_casks lacks docker / docker-desktop). Users opting into the cask still get that workaround for free. Verification: - ansible-playbook --syntax-check: clean - CI's macos-homebrew job exercises the homebrew role end-to-end on a fresh GitHub-hosted Apple Silicon runner -- the changed cli list + the cleared casks list both get installed cleanly Fixes #8 --- README.md | 56 ++++++++++++++++++++++++++---- roles/homebrew/defaults/main.yml | 21 +++++++++-- roles/kubernetes/defaults/main.yml | 20 +++++++++++ roles/kubernetes/tasks/main.yml | 29 ++++++++++++---- 4 files changed, 110 insertions(+), 16 deletions(-) create mode 100644 roles/kubernetes/defaults/main.yml diff --git a/README.md b/README.md index c3b3c7d..efb596a 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,14 @@ After a clean bootstrap you have: - Homebrew + the LLMKube-supporting package set (`kubectl`, `helm`, `kind`, `jq`, `yq`, `gh`, `k9s`, `tmux`, `ripgrep`, `fzf`). -- Docker Desktop, started, with a `kind` cluster named - `llmkube-local` running the LLMKube operator from the published - Helm chart. +- `colima` started with a user-space Lima VM (4 CPU, 6 GiB, 60 GiB + disk by default; override via `--extra-vars`). Provides the Docker + socket `kind` uses. `docker` / `docker-compose` / `docker-buildx` + CLIs installed via Homebrew, no Docker Desktop required. To use + Docker Desktop instead, see the "Docker Desktop on Apple Silicon" + callout near the end of this README. +- A `kind` cluster named `llmkube-local` running the LLMKube + operator from the published Helm chart. - A clone of `defilantech/LLMKube` at `~/src/LLMKube`, and the `metal-agent` daemon installed + running under `launchd`. - A starter model (`phi-4-mini-instruct`, Q4_K_M) deployed via @@ -97,7 +102,7 @@ group_vars/all.yml # tunables (memory_fraction, src_dir, model_repo, etc.) roles/ system/ # macOS sanity asserts, Xcode CLT homebrew/ # brew + per-package installs - kubernetes/ # Docker Desktop, kind, kubectl, helm, k9s + kubernetes/ # colima, kind, kubectl, helm, k9s llmkube_core/ # clone LLMKube, helm install, install metal-agent model_starter/ # download phi-4-mini, apply Model + InferenceService opencode/ # opencode binary + sanitized config template @@ -119,11 +124,48 @@ Every PR runs three linters: - `shellcheck` — `bootstrap.sh` and `teardown.sh` These run on an Ubuntu runner in seconds. They do not exercise the -playbook end-to-end — macOS-bound bugs in homebrew / launchd / Docker -Desktop will only surface when a human runs the bootstrap on a real -Mac. A future change may add a nightly macOS runner that does the +playbook end-to-end — macOS-bound bugs in homebrew / launchd / colima +will only surface when a human runs the bootstrap on a real Mac. A +future change may add a nightly macOS runner that does the full install; that's not in v0.1. +## Docker Desktop on Apple Silicon (opt-in only) + +The default Docker runtime is **colima**. If you specifically want +Docker Desktop instead (for the GUI, for parity with an existing +team setup, or to test something Docker Desktop-only), opt in via +your `group_vars`: + +```yaml +# group_vars/all.yml or a per-host override +homebrew_casks: + - docker # the cask is named `docker`; brew resolves to docker-desktop +``` + +Two caveats specific to Apple Silicon: + +1. Docker Desktop's cask postinstall hook writes CLI plugin symlinks + into `/usr/local/cli-plugins`. The pre-task in + `roles/homebrew/tasks/main.yml` handles that automatically when + `homebrew_casks` contains `docker` / `docker-desktop`. +2. Docker Desktop also writes the `docker`, `docker-compose`, and + `docker-buildx` binaries into `/usr/local/bin`. On a fresh + Apple Silicon Mac that directory may not exist (Homebrew lives + at `/opt/homebrew/bin`), and `/usr/local/` is root-owned. If + bootstrap fails here, run once: + + ```sh + sudo mkdir -p /usr/local/bin + sudo chown -R "$(whoami)":staff /usr/local/bin + ./bootstrap.sh + ``` + + Subsequent runs no-op; Docker Desktop will not try to recreate + the directory once it has user-write perms. + +If neither caveat sounds appealing, the default colima path is +strictly easier. + ## Contributing This repo is small but opinionated. Two rules: diff --git a/roles/homebrew/defaults/main.yml b/roles/homebrew/defaults/main.yml index 56a16b3..80bc618 100644 --- a/roles/homebrew/defaults/main.yml +++ b/roles/homebrew/defaults/main.yml @@ -10,6 +10,18 @@ homebrew_cli_packages: - k9s - kustomize + # Docker via colima. colima boots a user-space Lima VM and exposes + # a Docker socket; the docker / docker-compose / docker-buildx CLIs + # talk to that socket. No /usr/local writes, no sudo dance, smaller + # resident footprint than Docker Desktop. To use Docker Desktop + # instead, add `docker` to `homebrew_casks` in your group_vars and + # be ready to run sudo for /usr/local/* directory creation -- see + # README "Docker Desktop on Apple Silicon" callout. + - colima + - docker + - docker-compose + - docker-buildx + # llama.cpp for Metal-accelerated inference (metal-agent shells to it) - llama.cpp @@ -26,5 +38,10 @@ homebrew_cli_packages: - go - make -homebrew_casks: - - docker # Docker Desktop -- container runtime kind depends on +# Default casks: empty. The previous default included Docker Desktop, +# but its postinstall hook sudo-writes to multiple /usr/local/* +# paths that do not exist on a fresh Apple Silicon Mac, and brew +# runs under `become: false` so the sudo prompts have no TTY. We +# switched to colima (above) as the default Docker runtime. Add +# casks here in your group_vars when you want GUI apps. +homebrew_casks: [] diff --git a/roles/kubernetes/defaults/main.yml b/roles/kubernetes/defaults/main.yml new file mode 100644 index 0000000..85a970d --- /dev/null +++ b/roles/kubernetes/defaults/main.yml @@ -0,0 +1,20 @@ +--- +# kubernetes role defaults. +# +# The Docker runtime is colima (user-space Lima VM) rather than +# Docker Desktop. Reasons: +# - No /usr/local/* writes -> no sudo prompts during cask install +# - Matches the convention already in use on other LLMKube hosts +# - Smaller resident memory footprint, useful on Macs that also +# serve a 30B-class model locally +# +# Override any of the below in group_vars/all.yml or via +# --extra-vars when the defaults do not fit your hardware. + +# colima VM sizing. Defaults are tuned for a single-node kind cluster +# running LLMKube core plus a couple of small workloads -- enough +# headroom that `kind create cluster` does not OOM, not so much that +# we steal RAM from mlx-server. +colima_cpu: 4 +colima_memory_gib: 6 +colima_disk_gib: 60 diff --git a/roles/kubernetes/tasks/main.yml b/roles/kubernetes/tasks/main.yml index cedc14f..8c2c5a5 100644 --- a/roles/kubernetes/tasks/main.yml +++ b/roles/kubernetes/tasks/main.yml @@ -1,16 +1,31 @@ --- # kubernetes role -- bring up the local cluster the operator runs on. # -# Docker Desktop is installed by the homebrew role; we just need to -# make sure it is running, then create the kind cluster + verify the -# operator can talk to it. +# Docker is provided by colima (user-space Lima VM), installed by the +# homebrew role. We make sure colima is running with enough resources +# for a kind cluster, then create the cluster + verify connectivity. +# Colima's daemon stays up across reboots once started; `colima start` +# is idempotent (no-ops when the VM is already up). -- name: Ensure Docker Desktop is running - ansible.builtin.command: open -ga Docker +- name: Check colima status + ansible.builtin.command: colima status + register: colima_status changed_when: false - # `open -ga` is idempotent: silently no-ops when Docker is already up. + failed_when: false + +- name: Start colima with the configured VM size + ansible.builtin.command: >- + colima start + --cpu {{ colima_cpu }} + --memory {{ colima_memory_gib }} + --disk {{ colima_disk_gib }} + # colima writes the 'is it running?' line to stderr, hence the + # stderr match below. rc != 0 covers the first-ever invocation + # where the VM has not been created yet. + when: colima_status.rc != 0 or 'Running' not in (colima_status.stderr | default('')) + changed_when: true -- name: Wait for Docker daemon to accept commands +- name: Wait for Docker daemon (via colima) to accept commands ansible.builtin.command: docker info register: docker_info retries: 30 From 7dd74cfd3c9f38208dbbbee983e48064c74465a6 Mon Sep 17 00:00:00 2001 From: Christopher Maher Date: Sat, 23 May 2026 16:54:17 -0700 Subject: [PATCH 2/2] fix(homebrew): skip cask task when homebrew_casks is empty CI's macos-homebrew job caught it: community.general.homebrew_cask errors with 'You must select a cask to install' when given an empty list. With this PR's switch to colima the default homebrew_casks list is [], so the task needs a 'when:' guard to skip when nothing is requested. Add 'when: homebrew_casks | length > 0' on the cask install task. Users who opt back into Docker Desktop (or any other cask) via group_vars continue to get the full install path. --- roles/homebrew/tasks/main.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/roles/homebrew/tasks/main.yml b/roles/homebrew/tasks/main.yml index c979060..3e10340 100644 --- a/roles/homebrew/tasks/main.yml +++ b/roles/homebrew/tasks/main.yml @@ -49,3 +49,9 @@ # Desktop manually. accept_external_apps: true become: false + # community.general.homebrew_cask errors with "You must select a + # cask to install" when handed an empty list. The default + # homebrew_casks is now [] (colima replaces docker-desktop), so + # skip the task entirely unless a user has opted into one or more + # casks via group_vars / --extra-vars. + when: homebrew_casks | length > 0