Skip to content

feat(runpod): private-by-default ports, VS Code Ollama bridge#1

Open
RCSnyder wants to merge 1 commit into
ugudlado:mainfrom
RCSnyder:ghcp_vscode_connection
Open

feat(runpod): private-by-default ports, VS Code Ollama bridge#1
RCSnyder wants to merge 1 commit into
ugudlado:mainfrom
RCSnyder:ghcp_vscode_connection

Conversation

@RCSnyder

Copy link
Copy Markdown

Hey cool project, I finally got this working on my Windows machine with Github Copilot and I thought I'd throw a MR together for it. Not sure what the contributing guidelines are for this, open to suggestions/requests for change.


Adds a Windows + VS Code + RunPod path to hopper without changing the existing Vast.ai/macOS contract. The pod stays private-by-default; the new bridge lets VS Code Copilot Chat agent mode talk to vLLM over Tailscale by pretending to be an Ollama server.

Vendor changes (src/vendors/runpod.sh):

  • Ports are now derived from runpod_ports(). Default is 22/tcp only; 8000/http requires both RUNPOD_PUBLIC_PORTS_ACK=I_UNDERSTAND_THIS_IS_PUBLIC and a non-empty VLLM_API_KEY so we never accidentally publish an unauthenticated vLLM. RUNPOD_EXPOSE_HTTP=1 / RUNPOD_PORTS opt back in.
  • record_spend_start + autosupervise_start now fire before run_remote_setup so a setup failure still leaves an active spend-log entry and a running guard.
  • runpod_restart_vllm re-runs run_remote_setup (re-copies scripts, re-joins Tailscale, restarts vLLM) and calls runpod_ensure_spend_tracking_started so recovery from a half-finished create doesn't silently stop billing tracking.
  • get_pod_endpoint strips CRLF from POD_IP/POD_SSH_PORT — runpodctl|jq on Git Bash leaves '\r' on parsed values and ssh rejected it as "Bad port".
  • runpod_status / runpod_logs surface the Ollama bridge when enabled.
  • Bumped ephemeral container disk to 80GB when no network volume is attached (30GB fills with vLLM install + HF cache for the 27B profile).
  • Replaced an rg shellout in runpod_ssh with grep (not always installed).

Shared lib (src/lib/gpu-common.sh):

  • load_env_file sources via a CRLF-stripped temp copy so Windows-saved .env files don't leak '\r' into ssh -i paths, curl URLs, etc.
  • run_remote_setup scp's setup-vllm.sh through a CRLF-stripped temp copy too, and also copies vscode-ollama-bridge.py to /workspace when present.
  • build_remote_env_prefix forwards VLLM_API_KEY and the OLLAMA_SHIM_* vars.
  • Auto-detect SSH key: prefer ~/.ssh/id_ed25519, ~/.ssh/id_rsa, then ~/.runpod/ssh/RunPod-Key-Go (runpodctl-generated, already registered with RunPod), and warn if none exist.
  • DEFAULT_VOLUME_CREATE_DC moved to US-KS-2 (EU-SE-1 has been rejecting new volume creation in my account; flip back per .env if you prefer EU).

vLLM setup (src/setup-vllm.sh):

  • Adds --api-key when VLLM_API_KEY is set, with a redacted echo.
  • Optional Ollama-compatible bridge controlled by OLLAMA_SHIM_ENABLED. The bridge runs in its own venv under /workspace/ollama-shim, started via uvicorn on 11434 (bound to 0.0.0.0 when Tailscale is up, else 127.0.0.1). Separate install/stop/serve/wait functions mirror the vLLM ones; logs go to /workspace/ollama-shim-.log.
  • uv install runs with UV_NO_CACHE=1 and clears uv/pip caches afterward to keep ephemeral disk usage down.

VS Code Ollama bridge (src/vscode-ollama-bridge.py):

  • Minimal FastAPI app — only the endpoints Copilot Chat's Ollama BYOK provider actually calls: /api/version, /api/tags, /api/show, and a transparent stream-proxy for /v1/chat/completions. /api/show reports capabilities:["tools"] so the model stays in the agent-mode picker, and context_length comes from the vLLM model entry. Reads BACKEND_BASE_URL, BACKEND_API_KEY, REPORTED_VERSION from env.

.env.example:

  • Documents RUNPOD_EXPOSE_HTTP, RUNPOD_PUBLIC_PORTS_ACK, VLLM_API_KEY, and the OLLAMA_SHIM_* knobs with the same private-by-default framing.

Docs:

  • docs/gpu-runbook.md: new sections "RunPod SSH On Git Bash" (CRLF + endpoint-lag quirks, public-port acknowledgement, openssl rand recipe) and "VS Code Ollama Agent Mode" (when to enable the bridge, restart-vllm as the in-place recovery path). Reflowed the failure table.
  • docs/course/00-vscode-runpod.md: new lesson 0 — full Windows on-ramp with winget installs (Git Bash, jq, Tailscale, OpenSSH, VS Code), runpodctl direct-download, Copilot Chat + Remote-SSH extensions, Git Bash as default VS Code terminal, Tailscale auth-key setup, .env walkthrough, create/teardown flow, and Copilot Chat agent-mode wiring via http://:11434.
  • README.md + docs/course/README.md link the new lesson without disturbing the existing macOS/Vast path.

No vendor contract changes: the dispatcher and KNOWN_COMMANDS are untouched, runpod_* function names are stable, and the spend-log format is unchanged.

…s on-ramp

Adds a Windows + VS Code + RunPod path to hopper without changing the
existing Vast.ai/macOS contract. The pod stays private-by-default; the new
bridge lets VS Code Copilot Chat agent mode talk to vLLM over Tailscale by
pretending to be an Ollama server.

Vendor changes (src/vendors/runpod.sh):
* Ports are now derived from runpod_ports(). Default is 22/tcp only; 8000/http
  requires both RUNPOD_PUBLIC_PORTS_ACK=I_UNDERSTAND_THIS_IS_PUBLIC and a
  non-empty VLLM_API_KEY so we never accidentally publish an unauthenticated
  vLLM. RUNPOD_EXPOSE_HTTP=1 / RUNPOD_PORTS opt back in.
* record_spend_start + autosupervise_start now fire before run_remote_setup so
  a setup failure still leaves an active spend-log entry and a running guard.
* runpod_restart_vllm re-runs run_remote_setup (re-copies scripts, re-joins
  Tailscale, restarts vLLM) and calls runpod_ensure_spend_tracking_started so
  recovery from a half-finished create doesn't silently stop billing tracking.
* get_pod_endpoint strips CRLF from POD_IP/POD_SSH_PORT — runpodctl|jq on
  Git Bash leaves '\r' on parsed values and ssh rejected it as "Bad port".
* runpod_status / runpod_logs surface the Ollama bridge when enabled.
* Bumped ephemeral container disk to 80GB when no network volume is attached
  (30GB fills with vLLM install + HF cache for the 27B profile).
* Replaced an `rg` shellout in runpod_ssh with grep (not always installed).

Shared lib (src/lib/gpu-common.sh):
* load_env_file sources via a CRLF-stripped temp copy so Windows-saved .env
  files don't leak '\r' into ssh -i paths, curl URLs, etc.
* run_remote_setup scp's setup-vllm.sh through a CRLF-stripped temp copy too,
  and also copies vscode-ollama-bridge.py to /workspace when present.
* build_remote_env_prefix forwards VLLM_API_KEY and the OLLAMA_SHIM_* vars.
* Auto-detect SSH key: prefer ~/.ssh/id_ed25519, ~/.ssh/id_rsa, then
  ~/.runpod/ssh/RunPod-Key-Go (runpodctl-generated, already registered with
  RunPod), and warn if none exist.
* DEFAULT_VOLUME_CREATE_DC moved to US-KS-2 (EU-SE-1 has been rejecting new
  volume creation in my account; flip back per .env if you prefer EU).

vLLM setup (src/setup-vllm.sh):
* Adds --api-key when VLLM_API_KEY is set, with a redacted echo.
* Optional Ollama-compatible bridge controlled by OLLAMA_SHIM_ENABLED. The
  bridge runs in its own venv under /workspace/ollama-shim, started via uvicorn
  on 11434 (bound to 0.0.0.0 when Tailscale is up, else 127.0.0.1). Separate
  install/stop/serve/wait functions mirror the vLLM ones; logs go to
  /workspace/ollama-shim-<hostname>.log.
* uv install runs with UV_NO_CACHE=1 and clears uv/pip caches afterward to
  keep ephemeral disk usage down.

VS Code Ollama bridge (src/vscode-ollama-bridge.py):
* Minimal FastAPI app — only the endpoints Copilot Chat's Ollama BYOK
  provider actually calls: /api/version, /api/tags, /api/show, and a
  transparent stream-proxy for /v1/chat/completions. /api/show reports
  capabilities:["tools"] so the model stays in the agent-mode picker, and
  context_length comes from the vLLM model entry. Reads BACKEND_BASE_URL,
  BACKEND_API_KEY, REPORTED_VERSION from env.

.env.example:
* Documents RUNPOD_EXPOSE_HTTP, RUNPOD_PUBLIC_PORTS_ACK, VLLM_API_KEY, and
  the OLLAMA_SHIM_* knobs with the same private-by-default framing.

Docs:
* docs/gpu-runbook.md: new sections "RunPod SSH On Git Bash" (CRLF +
  endpoint-lag quirks, public-port acknowledgement, openssl rand recipe) and
  "VS Code Ollama Agent Mode" (when to enable the bridge, restart-vllm as the
  in-place recovery path). Reflowed the failure table.
* docs/course/00-vscode-runpod.md: new lesson 0 — full Windows on-ramp with
  winget installs (Git Bash, jq, Tailscale, OpenSSH, VS Code), runpodctl
  direct-download, Copilot Chat + Remote-SSH extensions, Git Bash as default
  VS Code terminal, Tailscale auth-key setup, .env walkthrough, create/teardown
  flow, and Copilot Chat agent-mode wiring via http://<tailnet>:11434.
* README.md + docs/course/README.md link the new lesson without disturbing
  the existing macOS/Vast path.

No vendor contract changes: the dispatcher and KNOWN_COMMANDS are untouched,
runpod_* function names are stable, and the spend-log format is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant