feat(runpod): private-by-default ports, VS Code Ollama bridge#1
Open
RCSnyder wants to merge 1 commit into
Open
feat(runpod): private-by-default ports, VS Code Ollama bridge#1RCSnyder wants to merge 1 commit into
RCSnyder wants to merge 1 commit into
Conversation
…s on-ramp Adds a Windows + VS Code + RunPod path to hopper without changing the existing Vast.ai/macOS contract. The pod stays private-by-default; the new bridge lets VS Code Copilot Chat agent mode talk to vLLM over Tailscale by pretending to be an Ollama server. Vendor changes (src/vendors/runpod.sh): * Ports are now derived from runpod_ports(). Default is 22/tcp only; 8000/http requires both RUNPOD_PUBLIC_PORTS_ACK=I_UNDERSTAND_THIS_IS_PUBLIC and a non-empty VLLM_API_KEY so we never accidentally publish an unauthenticated vLLM. RUNPOD_EXPOSE_HTTP=1 / RUNPOD_PORTS opt back in. * record_spend_start + autosupervise_start now fire before run_remote_setup so a setup failure still leaves an active spend-log entry and a running guard. * runpod_restart_vllm re-runs run_remote_setup (re-copies scripts, re-joins Tailscale, restarts vLLM) and calls runpod_ensure_spend_tracking_started so recovery from a half-finished create doesn't silently stop billing tracking. * get_pod_endpoint strips CRLF from POD_IP/POD_SSH_PORT — runpodctl|jq on Git Bash leaves '\r' on parsed values and ssh rejected it as "Bad port". * runpod_status / runpod_logs surface the Ollama bridge when enabled. * Bumped ephemeral container disk to 80GB when no network volume is attached (30GB fills with vLLM install + HF cache for the 27B profile). * Replaced an `rg` shellout in runpod_ssh with grep (not always installed). Shared lib (src/lib/gpu-common.sh): * load_env_file sources via a CRLF-stripped temp copy so Windows-saved .env files don't leak '\r' into ssh -i paths, curl URLs, etc. * run_remote_setup scp's setup-vllm.sh through a CRLF-stripped temp copy too, and also copies vscode-ollama-bridge.py to /workspace when present. * build_remote_env_prefix forwards VLLM_API_KEY and the OLLAMA_SHIM_* vars. * Auto-detect SSH key: prefer ~/.ssh/id_ed25519, ~/.ssh/id_rsa, then ~/.runpod/ssh/RunPod-Key-Go (runpodctl-generated, already registered with RunPod), and warn if none exist. * DEFAULT_VOLUME_CREATE_DC moved to US-KS-2 (EU-SE-1 has been rejecting new volume creation in my account; flip back per .env if you prefer EU). vLLM setup (src/setup-vllm.sh): * Adds --api-key when VLLM_API_KEY is set, with a redacted echo. * Optional Ollama-compatible bridge controlled by OLLAMA_SHIM_ENABLED. The bridge runs in its own venv under /workspace/ollama-shim, started via uvicorn on 11434 (bound to 0.0.0.0 when Tailscale is up, else 127.0.0.1). Separate install/stop/serve/wait functions mirror the vLLM ones; logs go to /workspace/ollama-shim-<hostname>.log. * uv install runs with UV_NO_CACHE=1 and clears uv/pip caches afterward to keep ephemeral disk usage down. VS Code Ollama bridge (src/vscode-ollama-bridge.py): * Minimal FastAPI app — only the endpoints Copilot Chat's Ollama BYOK provider actually calls: /api/version, /api/tags, /api/show, and a transparent stream-proxy for /v1/chat/completions. /api/show reports capabilities:["tools"] so the model stays in the agent-mode picker, and context_length comes from the vLLM model entry. Reads BACKEND_BASE_URL, BACKEND_API_KEY, REPORTED_VERSION from env. .env.example: * Documents RUNPOD_EXPOSE_HTTP, RUNPOD_PUBLIC_PORTS_ACK, VLLM_API_KEY, and the OLLAMA_SHIM_* knobs with the same private-by-default framing. Docs: * docs/gpu-runbook.md: new sections "RunPod SSH On Git Bash" (CRLF + endpoint-lag quirks, public-port acknowledgement, openssl rand recipe) and "VS Code Ollama Agent Mode" (when to enable the bridge, restart-vllm as the in-place recovery path). Reflowed the failure table. * docs/course/00-vscode-runpod.md: new lesson 0 — full Windows on-ramp with winget installs (Git Bash, jq, Tailscale, OpenSSH, VS Code), runpodctl direct-download, Copilot Chat + Remote-SSH extensions, Git Bash as default VS Code terminal, Tailscale auth-key setup, .env walkthrough, create/teardown flow, and Copilot Chat agent-mode wiring via http://<tailnet>:11434. * README.md + docs/course/README.md link the new lesson without disturbing the existing macOS/Vast path. No vendor contract changes: the dispatcher and KNOWN_COMMANDS are untouched, runpod_* function names are stable, and the spend-log format is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey cool project, I finally got this working on my Windows machine with Github Copilot and I thought I'd throw a MR together for it. Not sure what the contributing guidelines are for this, open to suggestions/requests for change.
Adds a Windows + VS Code + RunPod path to hopper without changing the existing Vast.ai/macOS contract. The pod stays private-by-default; the new bridge lets VS Code Copilot Chat agent mode talk to vLLM over Tailscale by pretending to be an Ollama server.
Vendor changes (src/vendors/runpod.sh):
rgshellout in runpod_ssh with grep (not always installed).Shared lib (src/lib/gpu-common.sh):
vLLM setup (src/setup-vllm.sh):
VS Code Ollama bridge (src/vscode-ollama-bridge.py):
.env.example:
Docs:
No vendor contract changes: the dispatcher and KNOWN_COMMANDS are untouched, runpod_* function names are stable, and the spend-log format is unchanged.