Skip to content

fix: harden DB pool and rate-limit pool-exhaustion Sentry events - #77

Merged
amrtgaber merged 1 commit into
mainfrom
fix/db-pool-sentry-brake
Apr 19, 2026
Merged

fix: harden DB pool and rate-limit pool-exhaustion Sentry events#77
amrtgaber merged 1 commit into
mainfrom
fix/db-pool-sentry-brake

Conversation

@amrtgaber

Copy link
Copy Markdown
Contributor

Summary

  • DB pool resilience (`app/database.py`): adds `pool_pre_ping=True` and `pool_recycle=1800` to the async engine. Pool capacity is unchanged (defaults: 5 + 10 overflow = 15 per process). Pre-ping catches connections killed server-side (Cloud SQL idle timeout, failover) before they surface as errors; recycle proactively retires connections before those server-side timeouts fire.
  • Sentry rate limiter (`app/observability/sentry.py`): replaces the prior unfiltered capture with a `before_send` that allows 5 `TooManyConnectionsError` events per 60s per process, then drops subsequent ones. Storm visibility is preserved; structlog logs every occurrence regardless. Per-process state, so aggregate Sentry volume scales with Cloud Run instance count.
  • Walker handles direct match, `BaseExceptionGroup` members (asyncpg often arrives wrapped in anyio `TaskGroup` `ExceptionGroup`), and `cause`/`context` chains.

Context

On 2026-04-18 the Sentry monthly quota was exhausted by `TooManyConnectionsError` events originating from Cloud SQL. Root cause under investigation; this PR stops the quota bleed, makes the pool more resilient to transient connection issues, and preserves enough signal to confirm the storm has passed.

Rate-limit threshold (5/60s) and window are `_POOL_ERR_MAX_PER_WINDOW` / `_POOL_ERR_WINDOW_S` at the top of `sentry.py` if tuning becomes necessary.

Test plan

  • `uv run ruff check .` — clean
  • `uv run ruff format --check .` — clean
  • `uv run pytest` — 28 passed (7 new rate-limiter tests cover: unrelated errors pass through, missing `exc_info` passes through, first N pass / Nth+1 drops, dropped events don't extend the window, cause chain is walked, ExceptionGroup members are walked, window eviction allows new events after expiry)
  • Deploy to production
  • Observe Sentry: storm-grouped events at ≤ 5/min/instance; no fresh quota burn
  • Confirm backend Netlify builds for vagrant-story-web start succeeding again

pool_pre_ping + pool_recycle stop Cloud SQL idle kills and failovers
from surfacing as errors. A before_send rate limiter caps
TooManyConnectionsError reports at 5 per 60s per process — enough to
see a storm in Sentry without burning quota like on 2026-04-18.
Structlog continues to log every occurrence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant