Skip to content

feat(db): bound the connection pool with configurable sizing + pre-ping - #49

Merged
amrtgaber merged 1 commit into
mainfrom
feat/db-connection-pool
Jun 1, 2026
Merged

feat(db): bound the connection pool with configurable sizing + pre-ping#49
amrtgaber merged 1 commit into
mainfrom
feat/db-connection-pool

Conversation

@amrtgaber

Copy link
Copy Markdown
Contributor

Follow-up to #47 (the infra hardening it flagged from aoe2-live-standings-api's 2026-06-01 launch incident). Mirrors aoe2 #172.

Problem

app/database.py built the async engine with SQLAlchemy's defaults — pool_size=5, max_overflow=10 = up to 15 connections per instance — and no pool_pre_ping/pool_recycle. On Cloud Run (horizontal scaling) total DB connections = per-instance pool × running instances, and with no instance cap that can saturate Cloud SQL's max_connections (the morning connection-exhaustion outage in the referenced incident). Separately, Cloud SQL silently dropping idle connections surfaces as intermittent request errors without pool_pre_ping.

Change

Make the pool configurable via settings, with conservative defaults:

Setting Default Purpose
DB_POOL_SIZE 5 persistent connections per instance
DB_MAX_OVERFLOW 5 burst connections above pool_size (→ 10/instance, down from 15)
DB_POOL_TIMEOUT 30 seconds to wait for a free connection
DB_POOL_RECYCLE 1800 recycle connections older than N seconds
DB_POOL_PRE_PING true liveness check before use (handles idle drops)

Applied to Postgres only — SQLite (dev/test) uses a StaticPool that rejects these args, so _pool_kwargs() returns {} for sqlite URLs.

The sizing formula is documented in .env.example:

(DB_POOL_SIZE + DB_MAX_OVERFLOW) × max-instances + migrate-job + headroom ≤ Cloud SQL max_connections

Defaults (10/instance) fit a 200-connection tier at --max-instances=10 (added in the companion deploy PR). Pairs with that instance cap to bound worst-case connections.

Note: this isn't expected to be the bottleneck at current traffic — it's breathing room against a spike, and replaces the unbounded default. Final numbers can be tuned via env once the Cloud SQL tier's max_connections is confirmed.

Test plan

  • uv run pytest — 134 passed, 1 skipped
  • uv run ruff check . + ruff format --check — clean
  • New tests/test_db_pool.py: _pool_kwargs empty for sqlite / populated for postgres; a throwaway asyncpg engine reflects the configured pool.size(); a tripwire test that defaults stay ≤ 10/instance with pre-ping on.

app/database.py created the async engine with SQLAlchemy's defaults (pool_size=5, max_overflow=10 = up to 15 connections/instance) and no liveness check. On Cloud Run, which scales horizontally, total DB connections = per-instance pool × running instances, so an unbounded instance count can saturate Cloud SQL's max_connections — the morning-outage failure mode from the 2026-06-01 live event (aoe2-live-standings-api#171/#172). Idle connections dropped by Cloud SQL also surfaced as request errors without pre-ping.

Make pool_size, max_overflow, pool_timeout, pool_recycle, and pool_pre_ping configurable via settings with conservative defaults (10 connections/instance, pre-ping on, 30-min recycle). Apply them to Postgres only — SQLite (dev/test) uses a StaticPool that rejects these args. Document the sizing formula vs. Cloud SQL max_connections in .env.example. Follow-up to #47.
@amrtgaber
amrtgaber merged commit fb08167 into main Jun 1, 2026
2 checks passed
@amrtgaber
amrtgaber deleted the feat/db-connection-pool branch June 1, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant