Skip to content
This repository was archived by the owner on Jun 25, 2026. It is now read-only.

runbook.md

Latest commit

 

History

History
223 lines (160 loc) · 5.27 KB

runbook.md

File metadata and controls

223 lines (160 loc) · 5.27 KB
Raw

Operator Runbook

This is the fastest path for common production issues.

Set variables once:

HOST=your-server-host
DOMAIN=rascal.example.com
REPO=OWNER/REPO

0) Quick Triage

rascal doctor --host "$HOST"
rascal ps
rascal config view
rascal logs rascald --host "$HOST" --follow

1) Webhook Not Triggering Runs

Symptoms:

  • Label/comment in GitHub does nothing.
  • No new run appears in rascal ps.

Checks:

curl -fsS "https://${DOMAIN}/healthz"
rascal logs caddy-access --host "$HOST" --follow
rascal logs rascald --host "$HOST" --follow

Resync webhook/label (no deploy):

set -a; source .rascal.env; set +a
rascal init \
  --repo "$REPO" \
  --server-url "https://${DOMAIN}" \
  --skip-deploy \
  --api-token "$RASCAL_API_TOKEN" \
  --webhook-secret "$RASCAL_GITHUB_WEBHOOK_SECRET" \
  --github-admin-token "$GITHUB_ADMIN_TOKEN"

If behind Cloudflare, use:

  • SSL/TLS mode: Full (strict)
  • Proxy mode: DNS only while validating webhook behavior

See also: webhooks.md

2) Deploy Failing or Regressing

Blue/green deploy and rollback are primarily for restoring rascald API/webhook service safely. In-flight task execution is detached in Docker containers and should survive slot rotation while the active slot adopts supervision.

Run deploy directly:

set -a; source .rascal.env; set +a
rascal deploy \
  --host "$HOST" \
  --domain "$DOMAIN" \
  --codex-auth ~/.codex/auth.json \
  --github-runtime-token "$RASCAL_GITHUB_TOKEN"

Deploy now includes an explicit bootstrap phase. If a host package change is needed, update internal/deploy/assets/bootstrap_host.sh; deploy.go only orchestrates the upload/order of that bootstrap step plus the Rascal rollout.

To rotate only the server-side GitHub runtime token later, you can sync auth without re-supplying the webhook secret:

set -a; source .rascal.env; set +a
rascal auth sync --host "$HOST"

That preserves the existing RASCAL_GITHUB_WEBHOOK_SECRET already stored in /etc/rascal/rascal.env. If the server does not already have a webhook secret, the command fails instead of silently clearing it.

That --codex-auth value seeds or updates the shared stored codex credential used for Codex and Goose runs; it is not copied to a static server-side fallback file. For Claude and Goose-Claude runs, create a separate anthropic credential via rascal auth credentials create --provider anthropic --auth-file <path>.

Inspect remote services:

rascal logs rascald --host "$HOST" --lines 300
rascal logs caddy --host "$HOST" --lines 300
rascal logs caddy-access --host "$HOST" --lines 300

Confirm active slot and health:

ssh root@"$HOST" 'cat /etc/rascal/active_slot'
ssh root@"$HOST" 'curl -fsS http://127.0.0.1:18080/readyz || true'
ssh root@"$HOST" 'curl -fsS http://127.0.0.1:18081/readyz || true'
curl -fsS "https://${DOMAIN}/readyz"

See also: deployment.md

3) Run Appears Stuck

Identify run and follow logs:

rascal ps
rascal logs run RUN_ID --follow

If a run should stop:

rascal cancel RUN_ID
rascal logs run RUN_ID --follow

Requeue after fix:

rascal retry RUN_ID

If no progress across runs, inspect server:

rascal logs rascald --host "$HOST" --follow

Check detached execution state on host:

ssh root@"$HOST" "docker ps -a --format '{{.Names}} {{.Status}}' | rg '^rascal-' || true"

4) Cancel Does Not Take Effect Quickly

Request cancel:

rascal cancel RUN_ID
rascal logs run RUN_ID --follow

Verify container stop on remote host:

ssh root@"$HOST" "docker ps --format '{{.Names}}' | rg '^rascal-' || true"

If a deploy recently rotated slots, remember execution is detached: the new active slot should adopt supervision and continue cancellation/finalization.

If run remains active unexpectedly, capture:

rascal logs rascald --host "$HOST" --lines 300
rascal logs run RUN_ID --lines 300

If a recent deploy is involved, remember the two-slot interim policy:

  • the immediately previous slot may still be draining old work
  • the next deploy will reclaim that oldest draining slot if it needs to reuse it

5) Manual Rollback (Blue/Green)

Use only if automatic rollback did not recover service.

This restores the control plane to a known-good slot. It is not intended to preserve already-running work because active runs continue in detached containers and should be adopted by whichever slot becomes active.

  1. Determine slot state:
ssh root@"$HOST" 'cat /etc/rascal/active_slot; systemctl is-active rascal@blue; systemctl is-active rascal@green'
  1. Switch traffic back to known-good slot (example: blue):
ssh root@"$HOST" "cat >/etc/caddy/rascal-upstream.caddy <<'EOF'
reverse_proxy 127.0.0.1:18080
EOF
systemctl reload caddy || systemctl restart caddy
echo blue >/etc/rascal/active_slot
systemctl restart rascal@blue
if systemctl is-active --quiet rascal@green; then
  systemctl kill -s SIGUSR1 rascal@green || true
fi"
  1. Verify recovery:
curl -fsS "https://${DOMAIN}/readyz"
rascal doctor --host "$HOST"

6) Post-Incident Checklist

rascal doctor --host "$HOST"
rascal ps

Then:

  • Open an issue with failing run IDs and timestamps.
  • Include snippets from rascald, caddy, and run logs.