Skip to content
This repository was archived by the owner on Jun 25, 2026. It is now read-only.

deployment.md

Latest commit

 

History

History
199 lines (153 loc) · 7.77 KB

deployment.md

File metadata and controls

199 lines (153 loc) · 7.77 KB
Raw

Deployment

This doc describes how Rascal deploys rascald to a single host using blue/green slots, what is running where, and what happens during cutover and drain.

Detached runner containers now preserve in-flight runs across deploys and process restarts. Blue/green remains in place for readiness-checked cutover, webhook/API continuity, and rollback safety rather than for keeping active runs alive.

What Gets Deployed

Rascal deploy uploads/builds these artifacts on the server:

  • /opt/rascal/rascald (Linux binary)
  • /opt/rascal/runner/rascal-runner (Linux runner binary copied into image build context)
  • /etc/systemd/system/rascal@.service (slot unit)
  • /etc/rascal/rascal.env (shared runtime env)
  • /etc/rascal/rascal-blue.env and /etc/rascal/rascal-green.env (slot env)
  • /etc/caddy/Caddyfile + /etc/caddy/rascal-upstream.caddy (proxy target)
  • Docker images for the configured runner tags (defaults: rascal-runner-goose-codex:latest, rascal-runner-codex:latest, rascal-runner-claude:latest, and rascal-runner-goose-claude:latest)

It also writes:

  • /etc/rascal/active_slot with blue or green

Phases

Rascal host setup is split into three phases:

  • provision: create the VM and network resources only
  • bootstrap: install OS-level dependencies and host prerequisites
  • deploy: upload Rascal artifacts, build runner images, switch slots, and reload services

The source of truth for host packages now lives in internal/deploy/assets/bootstrap_host.sh. Package additions should be made there, not inline in deploy.go.

Runtime Topology

  • rascal@blue listens on 127.0.0.1:18080
  • rascal@green listens on 127.0.0.1:18081
  • Caddy proxies external traffic to the currently selected slot
  • Caddy also enforces request-shape guardrails before rascald sees traffic: method/path/header gating, request body size caps, header-size caps, and server-side read/write timeouts
  • Legacy single-unit rascal service mode is not supported
  • Slot identity is set by env:
    • rascal@blue gets RASCAL_SLOT=blue
    • rascal@green gets RASCAL_SLOT=green

Default deployed env also includes agent session persistence knobs (enabled by default):

  • RASCAL_TASK_SESSION_MODE=all
  • RASCAL_TASK_SESSION_ROOT=/var/lib/rascal/agent-sessions
  • RASCAL_TASK_SESSION_TTL_DAYS=14
  • RASCAL_RUNNER_IMAGE_GOOSE_CODEX, RASCAL_RUNNER_IMAGE_CODEX, RASCAL_RUNNER_IMAGE_CLAUDE, and RASCAL_RUNNER_IMAGE_GOOSE_CLAUDE set the runtime-specific runner images
  • Docker runner hardening defaults to baseline with: RASCAL_RUNNER_DOCKER_SECURITY_MODE=baseline, RASCAL_RUNNER_DOCKER_CPUS=2, RASCAL_RUNNER_DOCKER_MEMORY=4g, RASCAL_RUNNER_DOCKER_PIDS_LIMIT=256
  • strict currently adds a size-bounded /tmp mount via RASCAL_RUNNER_DOCKER_TMPFS_TMP_SIZE
  • Runner secrets default to file-scoped injection: RASCAL_RUNNER_ALLOW_ENV_SECRETS=false mounts per-run secrets read-only at /run/rascal-secrets
  • RASCAL_AGENT_RUNTIME is optional and overrides the default runtime when set

Blue/Green Sequence

Given active slot A and inactive slot B, deploy does:

  1. Build rascald for Linux and upload artifacts.
  2. Build rascal-runner for Linux and upload artifacts.
  3. Upload bootstrap assets and execute bootstrap_host.sh to ensure base packages (docker, caddy, curl, sqlite3, ripgrep) and host layout.
  4. Install uploaded rascal-runner into /opt/rascal/runner/rascal-runner.
  5. Build/update runner images on host.
  6. Install/update systemd unit and env files.
  7. If slot B is still draining from the previous deploy, reclaim it first: cancel its remaining active runs, allow a short cleanup window, and stop it.
  8. Start/restart slot B.
  9. Wait for slot B readiness (/readyz on B port).
  10. Update Caddy upstream to slot B and reload Caddy.
  11. Verify proxy readiness via Caddy.
  12. Write /etc/rascal/active_slot = B.
  13. Signal old slot A into deploy-drain mode.
  14. Keep new slot unit enabled/active.

Important: deploy success is no longer coupled to waiting for old-slot run completion.

Detached execution means blue/green is no longer required to preserve active task execution during deploy. Its remaining value is:

  • readiness-checked cutover before traffic moves
  • rollback if proxy activation fails
  • overlap safety while both slots are briefly alive
  • avoiding API/webhook downtime during rascald replacement

Drain Behavior

Deploy cutover uses an explicit two-slot policy:

  • The immediately previous slot is allowed to drain indefinitely.
  • The next deploy may reclaim that draining slot if it is needed again.

Deploy drain (SIGUSR1) does this:

  1. Enters draining mode.
  2. Stops accepting new work and shuts down HTTP listeners.
  3. Keeps supervising any active runs already assigned to that slot.
  4. Uses no fixed timeout cancellation.
  5. Exits only after those active runs finish, unless a later deploy reclaims it.

Deploy reclaim (SIGUSR2) is used only when the inactive slot is still draining during the next deploy:

  1. Enter drain mode and stop HTTP listeners.
  2. Stop run supervisors (release in-memory cancel funcs) but do not touch detached containers.
  3. Exit immediately. Detached containers continue executing.
  4. The new slot adopts them via RecoverRunningRuns() on startup.
  5. Stop the slot so deploy can reuse it.

Generic host/service shutdown is separate from deploy drain and may still use a bounded wait on exit.

Runner Entrypoint

  • Container entrypoint script is intentionally minimal (runner/entrypoint.sh).
  • It only executes /usr/local/bin/rascal-runner.
  • Task workflow behavior (git/agent/PR/meta handling) is implemented in Go in cmd/rascal-runner.

Overlap Safety (Both Slots Alive Briefly)

During overlap, only active slot may process webhooks:

  • rascald reads /etc/rascal/active_slot per webhook request.
  • If instance slot (RASCAL_SLOT) does not match active slot, webhook is accepted-but-skipped.

Additional safeguards:

  • Webhook delivery dedupe is atomic claim/finalize (no check-then-insert race).
  • Run start is DB-atomic (queued -> running) with task-level exclusivity, so two instances cannot both start work for the same queued run/task.
  • Detached execution handles are persisted, so startup recovery can adopt active runs immediately after slot rotation.

Cancellation Semantics

  • Cancel intent is persisted in run_cancels.
  • Active cancellation stops the detached container by persisted execution handle, even after supervision handoff.
  • Final run state is written after terminal observation/finalization.
  • Containers are explicitly removed during terminal cleanup.

Rollback Behavior

If Caddy reload/readiness fails after switching upstream during deploy:

  • Deploy attempts rollback:
    • restore upstream to previous slot
    • reload/restart Caddy
    • stop new slot and restart previous slot

Quick Inspection Commands

ssh root@HOST 'cat /etc/rascal/active_slot'
ssh root@HOST 'systemctl status rascal@blue --no-pager'
ssh root@HOST 'systemctl status rascal@green --no-pager'
ssh root@HOST 'cat /etc/caddy/rascal-upstream.caddy'
ssh root@HOST 'curl -fsS http://127.0.0.1:18080/readyz || true'
ssh root@HOST 'curl -fsS http://127.0.0.1:18081/readyz || true'

End-to-End Example Flow

Example: blue is active and running a job, deploy is triggered.

  1. Deploy prepares green and passes green readiness.
  2. Caddy upstream switches to green.
  3. active_slot flips to green.
  4. blue gets a deploy-drain signal and stops accepting new work.
  5. blue keeps supervising its active run until that run finishes.
  6. Deploy returns success without waiting for blue to finish.
  7. If another deploy happens before blue is done, deploy reclaims blue, cancels its remaining work, and reuses that slot.
  8. Otherwise blue exits naturally after its active work completes.