Skip to content

fix(db,deploy): fit connection budget inside Cloud SQL's max_connections=25 - #81

Merged
amrtgaber merged 1 commit into
mainfrom
fix/pool-fit-cloud-sql-cap
Apr 19, 2026
Merged

fix(db,deploy): fit connection budget inside Cloud SQL's max_connections=25#81
amrtgaber merged 1 commit into
mainfrom
fix/pool-fit-cloud-sql-cap

Conversation

@amrtgaber

Copy link
Copy Markdown
Contributor

Summary

Root cause evidence (from `gcloud sql instances describe criticalbit-db` + `gcloud run services describe vagrant-story-api`):

value
Cloud SQL tier `db-f1-micro`
Cloud SQL `max_connections` 25 (default, no override)
Previous pool (per process) 5 + 10 overflow = 15
Cloud Run `maxScale` 5
Previous peak demand 75 (3× the cap)

Any moderately parallel traffic could exhaust the DB; the Netlify sitemap generator's 18-call burst (separately fixed in vagrant-story-web#152) was the reliable trigger.

Changes

  • `app/database.py`: `pool_size=3`, `max_overflow=2` per process.
  • `.github/workflows/deploy.yml`: add `--max-instances=4` to the `gcloud run services update` step so `maxScale` and pool config stay in lockstep in git.

New peak: 4 × (3 + 2) = 20 connections, leaving ~5 for Postgres reserved roles + the migration job.

Trade-offs

  • Each instance has 5 DB connections available (vs 15 before). Endpoints issuing long-session-hold work (primarily `/v1/loadout`) could, under heavy burst, see `pool_timeout` queuing rather than immediate errors. That's an improvement: users wait briefly instead of failing.
  • `maxScale=4` trims horizontal headroom by 20%. Given observed typical traffic, this should be unnoticeable.

Future work (not in this PR)

  • `/v1/loadout` holds its session through the Python optimizer. Splitting query phase from compute phase would cut pool occupancy and make the cap feel much looser.
  • If traffic grows past what `db-f1-micro` supports, the cheapest upgrade is `db-g1-small` (~$7→$25/mo but raises `max_connections` to 50).

Test plan

  • `uv run ruff check .` clean
  • `uv run ruff format --check .` clean
  • `uv run pytest` — 21/21 pass
  • After deploy: `gcloud run services describe vagrant-story-api --format='value(spec.template.metadata.annotations."autoscaling.knative.dev/maxScale")'` shows `4`
  • Sentry: no new `TooManyConnectionsError` events under normal traffic

Closes #79

…ons=25

Cloud SQL db-f1-micro has max_connections=25; with the previous
pool_size=5 + max_overflow=10 at maxScale=5, peak demand was 75 —
3x the cap. The 2026-04-18 storm was driven by the Netlify sitemap
generator bursting 18 parallel calls into that tight headroom.

- app/database.py: pool_size=3, max_overflow=2 per process.
- .github/workflows/deploy.yml: --max-instances=4 on the service update
  so the budget stays in lockstep with the pool config.

Peak connections = 4 × 5 = 20, leaving ~5 slots for Postgres reserved
roles and the migration job.

Closes #79
@amrtgaber
amrtgaber merged commit ace62e1 into main Apr 19, 2026
2 checks passed
@amrtgaber
amrtgaber deleted the fix/pool-fit-cloud-sql-cap branch April 19, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

investigate: root cause of Postgres TooManyConnectionsError driving pool exhaustion

1 participant