Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@

Rent a GPU, run vLLM on it, and stop paying once the daily budget is gone. Works against Vast.ai and RunPod, with Tailscale handling the network so the pod is reachable by hostname from anywhere.

| What you get | How |
| ------------------- | ---------------------------------------------------------------------------- |
| Vendors | Vast.ai, RunPod (one CLI, identical command surface) |
| Inference server | vLLM, OpenAI-compatible at `:8000` — any client that speaks OpenAI works |
| Network | Tailscale hostname (e.g. `vllm-qwen27b-vast:8000`) — no port-forward, no SSH tunnel per session |
| Cost control | Hard `DAILY_BUDGET` ($/day) and optional `DAILY_HOURS` cap with grace countdown before terminate |
| Reproducibility | Per-model + per-GPU profiles in `src/profiles/*.env` |
| Adding a vendor | One file: `src/vendors/<name>.sh` implementing `create`/`ssh`/`terminate`/`status`/`logs` |

> **New here?** There's a course that walks you from zero to a personal coding agent running on a rented GPU, reachable from your laptop and your phone. Start at [`docs/course/README.md`](docs/course/README.md).
| What you get | How |
| ---------------- | ------------------------------------------------------------------------------------------------ |
| Vendors | Vast.ai, RunPod (one CLI, identical command surface) |
| Inference server | vLLM, OpenAI-compatible at `:8000` — any client that speaks OpenAI works |
| Network | Tailscale hostname (e.g. `vllm-qwen27b-vast:8000`) — no port-forward, no SSH tunnel per session |
| Cost control | Hard `DAILY_BUDGET` ($/day) and optional `DAILY_HOURS` cap with grace countdown before terminate |
| Reproducibility | Per-model + per-GPU profiles in `src/profiles/*.env` |
| Adding a vendor | One file: `src/vendors/<name>.sh` implementing `create`/`ssh`/`terminate`/`status`/`logs` |

> **New here?** There's a course that walks you from zero to a personal coding agent running on a rented GPU, reachable from your laptop and your phone. Start at [`docs/course/README.md`](docs/course/README.md). On Windows + VS Code + RunPod, take the parallel on-ramp at [`docs/course/00-vscode-runpod.md`](docs/course/00-vscode-runpod.md) instead.

## Quickstart

Expand Down
208 changes: 208 additions & 0 deletions docs/course/00-vscode-runpod.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
# Lesson 0 — Running Hopper from VS Code on Windows with RunPod

This is the zero-to-working path if you're on **Windows**, want to drive everything from **VS Code** (including GitHub Copilot Chat in agent mode), and prefer **RunPod** over Vast.ai.

The rest of the course (lessons 01–06) is macOS + Vast.ai oriented. This lesson is a parallel on-ramp: by the end of it you will have a RunPod pod running vLLM, joined to your tailnet, and reachable from VS Code via the Ollama-compatible bridge.

If you already use macOS, skip this and go to [the course README](README.md).

## What you'll have at the end

- A Windows machine that can run `./src/gpu.sh runpod create` from a Git Bash terminal in VS Code.
- A RunPod pod running vLLM (OpenAI-compatible) on port `8000`, plus the in-pod Ollama bridge on `11434`.
- Both reachable privately over Tailscale by hostname — no public ports, no SSH tunnel for everyday use.
- GitHub Copilot Chat configured to use your pod as an Ollama provider in agent mode.

## Prerequisites

- Windows 10 or 11 with admin rights to install software.
- A GitHub account (for VS Code + Copilot Chat).
- A credit card with **$5–$10** for RunPod top-up.
- A Tailscale account (free tier is enough).

## 1. Install the toolbelt

You need: Git Bash (shell + ssh + scp + curl), jq, Tailscale for Windows, runpodctl, and VS Code. The easiest path is [winget](https://learn.microsoft.com/windows/package-manager/winget/), bundled with Windows 11 and recent Windows 10.

Open **PowerShell as Administrator** and run:

```powershell
winget install --id Git.Git -e
winget install --id stedolan.jq -e
winget install --id tailscale.tailscale -e
winget install --id Microsoft.VisualStudioCode -e
winget install --id Microsoft.OpenSSH.Beta -e # only if `ssh` is missing
```

`runpodctl` is not in winget. Install it from the official release into a folder on `PATH`:

```powershell
$dest = "$env:USERPROFILE\bin"
New-Item -ItemType Directory -Force -Path $dest | Out-Null
Invoke-WebRequest -Uri "https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-windows-amd64.exe" `
-OutFile "$dest\runpodctl.exe"
# Add to PATH for new shells:
setx PATH "$env:PATH;$dest"
```

Close PowerShell. Open a **new** Git Bash terminal (Start menu → "Git Bash") and verify:

```bash
git --version
bash --version
ssh -V
jq --version
tailscale version
runpodctl version
code --version
```

If any command is "not found", fix the install before continuing. PATH changes only apply to **new** terminals — close and reopen if needed.

> **Why Git Bash, not WSL or PowerShell?** `gpu.sh` is bash and uses POSIX tools (`awk`, `sed`, `scp`, `mktemp`). Git Bash ships all of them and stays out of WSL's filesystem-translation issues. If you already live in WSL, that works too — just clone the repo inside WSL so paths are Linux-native.

## 2. Open VS Code with the right defaults

Install these extensions (in VS Code or via the command line):

```powershell
code --install-extension GitHub.copilot
code --install-extension GitHub.copilot-chat
code --install-extension ms-vscode-remote.remote-ssh
```

Set Git Bash as the default integrated terminal so every terminal you open in VS Code is the same shell `gpu.sh` runs in. In VS Code:

1. `Ctrl+Shift+P` → **Terminal: Select Default Profile** → **Git Bash**.
2. Open a new terminal (`Ctrl+\``) and confirm the prompt looks like `user@host MINGW64 ~`.

## 3. Set up Tailscale

1. Sign in to the Tailscale tray app on Windows (`Tailscale → Log in`). Your laptop joins your tailnet.
2. In the [admin console](https://login.tailscale.com/admin/dns), turn on **MagicDNS**. This lets you reach pods by hostname (e.g. `vllm-qwen27b:8000`) instead of IP.
3. Create a reusable auth key at [`/admin/settings/keys`](https://login.tailscale.com/admin/settings/keys):
- **Reusable**: ON
- **Ephemeral**: ON (so destroyed pods leave no stale device entries)
- **Pre-approved**: ON
4. Copy the key. You'll paste it into `src/.env` in step 5.

## 4. Set up RunPod

1. Create an account at [runpod.io](https://www.runpod.io/) and add credit ($5–$10 is fine to start).
2. In **Console → Settings → API Keys**, create a key with read/write access. Copy it.
3. In a Git Bash terminal, configure `runpodctl`:

```bash
runpodctl config --apiKey <paste-your-key>
runpodctl pod list # should return an empty list (or your existing pods), not an auth error
```

This writes `~/.runpod/config.toml` and generates an SSH key at `~/.runpod/ssh/RunPod-Key-Go`. Hopper auto-detects that key, so you don't need to copy it anywhere.

## 5. Clone the repo and configure `.env`

In Git Bash:

```bash
mkdir -p ~/code && cd ~/code
git clone https://github.com/ugudlado/hopper.git
cd hopper
cp src/.env.example src/.env
```

Open `src/.env` in VS Code (`code src/.env`) and set:

```env
TAILSCALE_AUTHKEY=tskey-auth-... # from step 3
RUNPOD_API_KEY=... # from step 4

# Daily caps. The supervisor terminates the pod when either is hit.
DAILY_BUDGET=5
DAILY_HOURS=5

# Enable the in-pod Ollama-compatible bridge for VS Code Copilot Chat agent mode.
OLLAMA_SHIM_ENABLED=1
```

Leave `VLLM_API_KEY` and `RUNPOD_EXPOSE_HTTP` empty — Tailscale is the access path, and public RunPod ports are not needed.

Save the file. `.env` is gitignored; double-check before any `git add`.

## 6. Create the pod

Still in Git Bash, from the repo root:

```bash
PROFILE=qwen3.6-27b-4b ./src/gpu.sh runpod create
```

What this does, in order:

1. Picks an A100 80GB on RunPod, attaches a 60GB network volume named `vllm-cache` (created on first run).
2. SSHs in, copies `setup-vllm.sh` and `vscode-ollama-bridge.py` to the pod.
3. Installs vLLM, joins Tailscale as `vllm-qwen27b`, starts vLLM on `:8000` and the bridge on `:11434`.
4. Records spend start and launches the supervisor in a tmux session named `guard`.

First boot takes 8–15 minutes (model download + JIT compile). On reruns with the network volume attached, it's 1–2 minutes.

Watch progress in another Git Bash terminal:

```bash
PROFILE=qwen3.6-27b-4b ./src/gpu.sh runpod logs
```

When it's serving, this works from your Windows machine:

```bash
curl http://vllm-qwen27b:8000/v1/models
curl http://vllm-qwen27b:11434/api/tags
```

If the curl fails: see the [recovery section](#recovery-from-a-broken-create) below.

## 7. Wire VS Code Copilot Chat to your pod

Copilot Chat's agent mode supports BYOK via the Ollama API. The in-pod bridge speaks Ollama and proxies to vLLM, so VS Code treats your pod like a local Ollama install.

1. In VS Code, open the Copilot Chat side panel.
2. Click the **model picker** at the bottom of the chat box → **Manage Models** → **Ollama**.
3. Set the Ollama endpoint to:

```text
http://vllm-qwen27b:11434
```

4. VS Code lists the available models (you'll see whatever `VLLM_MODEL` resolves to in the profile, e.g. `cyankiwi/Qwen3.6-27B-AWQ-INT4`). Tick it.
5. Switch the chat to **Agent** mode and pick that model.

Sanity check: ask Copilot Chat to "list the files in this repo." It should pick up the workspace and call its filesystem tool against your pod-served model.

## 8. Tear it down

When you're done for the day:

```bash
PROFILE=qwen3.6-27b-4b ./src/gpu.sh runpod terminate
```

The network volume survives, so the next `create` reuses the downloaded model. The supervisor would also terminate automatically once `DAILY_BUDGET` or `DAILY_HOURS` is hit.

## Recovery from a broken create

If `runpod create` exits after the pod is created but before setup finishes (network blip, SSH timeout, etc.) — the pod is still billing, but vLLM and the bridge are not running. Re-run setup in place instead of recreating:

```bash
PROFILE=qwen3.6-27b-4b ./src/gpu.sh runpod restart-vllm
```

This re-copies the setup script and bridge, re-joins Tailscale, restarts vLLM, and (re)starts the supervisor. See [`gpu-runbook.md`](../gpu-runbook.md) for the underlying behavior and known RunPod-on-Git-Bash quirks (CRLF in `runpodctl` output, SSH endpoint lag, etc.).

## Where to go next

You now have the same architecture lessons 01–06 build up to, just from a Windows + VS Code starting point. Skim these if you want the conceptual background:

- [Lesson 01 — tmux](01-tmux.md): why the agent loop survives terminal closes (works the same in Git Bash).
- [Lesson 02 — Tailscale](02-tailscale.md): how MagicDNS turns `vllm-qwen27b` into a routable hostname.
- [Lesson 04 — `gpu.sh`](04-gpu-sh.md): the dispatcher, profiles, and supervisor in detail.

The Pi lesson ([05](05-pi.md)) is optional on Windows — Copilot Chat agent mode covers the same "agent talks to vLLM" loop without installing Pi.
22 changes: 12 additions & 10 deletions docs/course/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ If `command -v remote-session` prints nothing, add `export PATH="$HOME/.local/bi

**Ready?** [**Start with lesson 01 — tmux →**](01-tmux.md)

> **On Windows, or want to drive everything from VS Code with RunPod?** Take the parallel on-ramp first: [**Lesson 0 — Running Hopper from VS Code on Windows with RunPod**](00-vscode-runpod.md). It installs every Windows prerequisite (Git Bash, jq, Tailscale, runpodctl) and ends with Copilot Chat agent mode pointed at your pod.

## Lessons

1. **[tmux](01-tmux.md)**: Sessions that outlive the terminal you started them in. _This is what makes the device handoff possible._
Expand All @@ -121,14 +123,14 @@ If `command -v remote-session` prints nothing, add `export PATH="$HOME/.local/bi

**Plan on roughly $1/hr** for a verified A100 SXM4 80GB on Vast with a recent driver (CUDA 12.6+). It can be a bit less or more depending on the day and what's available. With `DAILY_HOURS=5`, that's about $5/day, or ~$100/month if you run the daily envelope every weekday.

| Lesson | Time | GPU cost | Device needed |
| -------------- | --------- | ---------------------------------- | ---------------- |
| 01 — tmux | 15–20 min | $0 | Laptop |
| 02 — Tailscale | 20–30 min | $0 | Laptop + phone |
| 03 — Vast.ai | 20–30 min | $0 until you rent | Laptop |
| 04 — `gpu.sh` | 30–45 min | ~$1 for a create + idle window | Laptop + GPU pod |
| 05 — Pi | 30–45 min | GPU running while you test | Laptop + phone |
| 06 — Capstone | 20–30 min | GPU running during handoff | Laptop + phone |
| Lesson | Time | GPU cost | Device needed |
| -------------- | --------- | ------------------------------ | ---------------- |
| 01 — tmux | 15–20 min | $0 | Laptop |
| 02 — Tailscale | 20–30 min | $0 | Laptop + phone |
| 03 — Vast.ai | 20–30 min | $0 until you rent | Laptop |
| 04 — `gpu.sh` | 30–45 min | ~$1 for a create + idle window | Laptop + GPU pod |
| 05 — Pi | 30–45 min | GPU running while you test | Laptop + phone |
| 06 — Capstone | 20–30 min | GPU running during handoff | Laptop + phone |

The main cost risk is not the course itself; it is forgetting to terminate the pod. Keep the supervisor running whenever the GPU is up.

Expand All @@ -149,5 +151,5 @@ The main cost risk is not the course itself; it is forgetting to terminate the p

---

| | | **[Lesson 01 — tmux →](01-tmux.md)** |
| :--- | :---: | ---: |
| | | **[Lesson 01 — tmux →](01-tmux.md)** |
| :-- | :-: | -----------------------------------: |
85 changes: 77 additions & 8 deletions docs/gpu-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ The Vast CLI has returned false zero-result searches for `gpu_ram>=N` filters. `

Common Vast create failures:

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Pod stays in `loading` | Fresh host is pulling a large image | Wait up to 15 minutes before retrying. |
| `manifest unknown` or invalid image | Bad Docker image tag | Use the known-good Vast profile image. |
| SSH permission denied | Key missing from Vast account | Run `vastai show ssh-keys`; register `~/.ssh/id_ed25519.pub` if missing. |
| Instance created but never starts | Vast `create instance --ssh` does not start instances | Harness calls `vastai start instance <id>` after create. |
| Destroy hangs | Vast prompts interactively | Harness pipes confirmation for destroy. |
| SSH proxy logs remote forwarding errors | Vast platform proxy issue | Retry later or consider a direct-port refactor if it becomes frequent. |
| Symptom | Likely cause | Fix |
| --------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------ |
| Pod stays in `loading` | Fresh host is pulling a large image | Wait up to 15 minutes before retrying. |
| `manifest unknown` or invalid image | Bad Docker image tag | Use the known-good Vast profile image. |
| SSH permission denied | Key missing from Vast account | Run `vastai show ssh-keys`; register `~/.ssh/id_ed25519.pub` if missing. |
| Instance created but never starts | Vast `create instance --ssh` does not start instances | Harness calls `vastai start instance <id>` after create. |
| Destroy hangs | Vast prompts interactively | Harness pipes confirmation for destroy. |
| SSH proxy logs remote forwarding errors | Vast platform proxy issue | Retry later or consider a direct-port refactor if it becomes frequent. |

When a create attempt fails, check the provider state before layering on local fixes:

Expand All @@ -31,6 +31,75 @@ vastai show instances --raw | jq '.[0] | {id, actual_status, status_msg, ssh_hos

## vLLM Runtime Fixes

## RunPod SSH On Git Bash

RunPod creates public ingress for pod `ports` entries beyond `22/tcp`, including
`*.proxy.runpod.net` URLs for `*/http` ports. Hopper defaults to `22/tcp` only,
so vLLM is reached via Tailscale or an explicit SSH tunnel.

Public RunPod ports are blocked unless both conditions are true:

```env
RUNPOD_PUBLIC_PORTS_ACK=I_UNDERSTAND_THIS_IS_PUBLIC
VLLM_API_KEY=<strong random token>
```

Generate the API key from Git Bash with OpenSSL:

```bash
openssl rand -hex 32
```

With `VLLM_API_KEY` set, `setup-vllm.sh` starts vLLM with `--api-key`, and
clients must send `Authorization: Bearer <VLLM_API_KEY>`. `RUNPOD_EXPOSE_HTTP=1`
and custom `RUNPOD_PORTS` are for short manual tests only.

On Windows Git Bash, `runpodctl pod get ... | jq -r ...` can leave a trailing
carriage return on the parsed SSH port. That turns `22008` into `$'22008\r'`,
so the harness fails with `Bad port` even though the same endpoint works when
typed manually from `runpodctl` output. `get_pod_endpoint` strips `\r` from the
parsed IP and port before calling `ssh`.

RunPod can also publish a pod as `RUNNING` before the SSH endpoint is usable.
The harness waits for `runpodctl pod get <id>` to expose `.ssh.ip` and
`.ssh.port` before connecting.

## VS Code Ollama Agent Mode

Hopper's primary contract is still an OpenAI-compatible vLLM endpoint on `:8000`.
For VS Code clients that only speak the Ollama API, enable the in-pod shim:

```env
OLLAMA_SHIM_ENABLED=1
OLLAMA_SHIM_VERSION=0.6.4
OLLAMA_SHIM_PORT=11434
```

This keeps vLLM as the only model server and starts a sibling compatibility
process inside the pod that proxies Ollama requests to `http://127.0.0.1:8000/v1`.
Use Tailscale to reach it privately:

```text
http://<tailnet-host>:11434
```

Do not route this through OpenRouter unless you explicitly want a third-party
broker in the path. The intended Hopper pattern is local vLLM + local shim,
both private to the tailnet.

If a create command exits after the pod is created but before setup completes,
the pod may be reachable by SSH with an empty `/workspace` and no vLLM or
Tailscale process yet. Recover in place instead of recreating:

```bash
PROFILE=qwen3.6-27b-4b ./src/gpu.sh runpod restart-vllm
```

That command re-copies `src/setup-vllm.sh`, joins Tailscale if configured, and
starts vLLM on the existing pod. It also records an active spend-log entry and
starts the budget supervisor if the original create command exited before those
steps ran.

### Qwen3.x GDN On CUDA < 12.6

Qwen3.5/Qwen3.6 can fail on first inference when flashinfer tries to JIT Gated Delta Net kernels against CUDA 12.4:
Expand Down
Loading