ci(deploy): cap max-instances and prune stale Cloud Run revisions - #50
Merged
Conversation
Cloud Run defaulted to no instance cap (max 100) and never prunes old revisions. Each revision pinned by min-instances holds an instance + its DB connection pool even at 0% traffic, so unbounded scale and revision accumulation can saturate Cloud SQL's max_connections — the morning connection-exhaustion outage from the 2026-06-01 live event (aoe2-live-standings-api#171). Cap the service at MAX_INSTANCES=10 (bounds worst-case connections with the pool sizing in the companion PR) and add a best-effort post-deploy step that keeps the 2 newest revisions and deletes the rest. Follow-up to #47.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #47 (the infra hardening it flagged from
aoe2-live-standings-api's 2026-06-01 launch incident). Mirrors aoe2#171. Pairs with #49 (connection-pool sizing).Problem
deploy.ymlset no--max-instances(Cloud Run default = 100) and never pruned old revisions. Two compounding leaks toward Cloud SQL'smax_connections:min-instances>0keeps an instance and its DB pool alive at 0% traffic. Enough stale revisions silently saturate the cap (the morning connection-exhaustion outage in the referenced incident).Change
MAX_INSTANCES=10env var, passed as--max-instanceson the service update. Bounds worst-case connections together with the pool sizing in feat(db): bound the connection pool with configurable sizing + pre-ping #49:(5+5)/instance × 10 = ~100, plus the migrate job + headroom — sized to be raised alongside the Cloud SQL tier.continue-on-error: true+ per-delete|| trueso it can never fail a deploy that already succeeded.Validation
deploy.ymlparses as valid YAML; the new step is recognized by the workflow parser.bash -nsyntax check.github.event.*input is used in anyrun:block (only staticenv:vars +github.sha), so there's no workflow-injection surface.Notes
max_connections(aoe2#173) — is infra (gcloud/Terraform) and out of scope here.run.revisions.delete, the prune step no-ops harmlessly (best-effort); grant the permission to activate it.