Skip to content

ci(deploy): cap max-instances and prune stale Cloud Run revisions - #50

Merged
amrtgaber merged 1 commit into
mainfrom
ci/cloud-run-connection-guardrails
Jun 1, 2026
Merged

ci(deploy): cap max-instances and prune stale Cloud Run revisions#50
amrtgaber merged 1 commit into
mainfrom
ci/cloud-run-connection-guardrails

Conversation

@amrtgaber

Copy link
Copy Markdown
Contributor

Follow-up to #47 (the infra hardening it flagged from aoe2-live-standings-api's 2026-06-01 launch incident). Mirrors aoe2 #171. Pairs with #49 (connection-pool sizing).

Problem

deploy.yml set no --max-instances (Cloud Run default = 100) and never pruned old revisions. Two compounding leaks toward Cloud SQL's max_connections:

  1. Unbounded scale — at the default cap, 100 instances × the per-instance pool can demand far more connections than any small Cloud SQL tier allows.
  2. Revision accumulation — Cloud Run never auto-prunes; a revision pinned by min-instances>0 keeps an instance and its DB pool alive at 0% traffic. Enough stale revisions silently saturate the cap (the morning connection-exhaustion outage in the referenced incident).

Change

  • Cap scale: MAX_INSTANCES=10 env var, passed as --max-instances on the service update. Bounds worst-case connections together with the pool sizing in feat(db): bound the connection pool with configurable sizing + pre-ping #49: (5+5)/instance × 10 = ~100, plus the migrate job + headroom — sized to be raised alongside the Cloud SQL tier.
  • Prune revisions: a best-effort post-deploy step that keeps the 2 newest revisions (current + one rollback) and deletes the rest. continue-on-error: true + per-delete || true so it can never fail a deploy that already succeeded.

Validation

  • deploy.yml parses as valid YAML; the new step is recognized by the workflow parser.
  • Prune shell script passes bash -n syntax check.
  • No untrusted github.event.* input is used in any run: block (only static env: vars + github.sha), so there's no workflow-injection surface.
  • Runtime behavior (instance cap applied, revisions pruned) is exercised on the next deploy to main — it can't be tested in CI.

Notes

  • This is breathing room, not a response to a current bottleneck. The remaining lever — verifying/raising the Cloud SQL tier's max_connections (aoe2 #173) — is infra (gcloud/Terraform) and out of scope here.
  • If the deploy service account lacks run.revisions.delete, the prune step no-ops harmlessly (best-effort); grant the permission to activate it.

Cloud Run defaulted to no instance cap (max 100) and never prunes old revisions. Each revision pinned by min-instances holds an instance + its DB connection pool even at 0% traffic, so unbounded scale and revision accumulation can saturate Cloud SQL's max_connections — the morning connection-exhaustion outage from the 2026-06-01 live event (aoe2-live-standings-api#171).

Cap the service at MAX_INSTANCES=10 (bounds worst-case connections with the pool sizing in the companion PR) and add a best-effort post-deploy step that keeps the 2 newest revisions and deletes the rest. Follow-up to #47.
@amrtgaber
amrtgaber merged commit d5de1da into main Jun 1, 2026
2 checks passed
@amrtgaber
amrtgaber deleted the ci/cloud-run-connection-guardrails branch June 1, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant