Skip to content

Connection efficiency: host-level shared LISTEN (supervisor-owned NotifyListener, one direct connection per host) + explicit multi-queue read priority contract #381

Description

@mhenrixon

Problem

On transaction-pool PgBouncer platforms (PlanetScale Postgres and friends), every LISTEN connection must bypass the pooler and hit the direct port — and the direct slice of max_connections is the scarcest resource in the whole system (the pooler's server-side pools draw from the same ceiling, minus superuser-reserved slots).

pgbus's current LISTEN footprint scales with process count, not host count:

  • Worker NotifyListener (lib/pgbus/process/notify_listener.rb): one dedicated direct connection per worker fork.
  • Consumer NotifyListener (lib/pgbus/process/consumer.rb:366): one more per consumer fork.
  • Streamer Listener (lib/pgbus/web/streamer/listener.rb): one per web-server worker process.

A deployment with 5 worker capsules + 2 consumers on a job host and 2 web hosts × 4 Puma workers pins 7 + 8 = 15 direct connections at steady state — before migrations, consoles, or monitoring tools take theirs. On a small cluster (max_connections = 30) this collides with the pooler's own server pools and produces FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute.

The waste is structural: a single PG connection can LISTEN on any number of channels. N forks on one host listening to overlapping channel sets need one connection, not N.

Proposal

1. Supervisor-owned NotifyListener (job role): 1 direct connection per host

Move the NotifyListener up from Worker/Consumer forks into the Supervisor process:

Result: a job host's LISTEN footprint drops from worker_forks + consumer_forks to 1.

2. Streams listener consolidation (web role): 1 per host instead of 1 per Puma worker

Puma workers are forked from a master; the same self-pipe pattern applies:

  • A Puma plugin (or before_fork hook) starts the Streamer Listener in the master process; workers inherit read-pipes and get woken via IPC.
  • Single-mode / non-preforking servers keep the current per-process listener (scope config again).

Result: web hosts go from puma_workers LISTEN connections to 1.

3. Make multi-queue read priority an explicit, documented, tested contract

capsule_dsl.rb documents "list order = strict priority", but the actual mechanism is incidental: read_multi builds pgmq.read(q1) UNION ALL pgmq.read(q2) ... LIMIT n (pgmq-ruby client/multi_queue.rb), and the ordering only holds because Postgres's Append node fills the LIMIT from subqueries in order. Nothing asserts it, and a future planner change or pgmq-ruby refactor could silently break it. Also note a side effect worth documenting: subqueries that execute claim messages (set vt) even when the outer LIMIT discards their rows — those messages go invisible for a full visibility timeout without being processed.

  • Document the strict-priority semantics (and the vt-claim caveat) on read_multi and in the capsule DSL docs.
  • Add an integration test: enqueue on q1/q2/q3, read with a limit smaller than total, assert earlier-listed queues win.
  • Consider making the ordering first-class (e.g. per-queue reads in list order with a shared budget, like fetch_prioritized already does for priority sub-queues) so the contract doesn't depend on planner behavior.

4. Doctor: report the direct-connection budget

doctor already checks "Dedicated connections … connect OK". Extend it to count what the current config will pin (forks × listeners under :fork scope, 1 under :supervisor, streams scope, etc.) and print the number, so operators can do capacity math from the doctor output alone.

Why not "just turn worker_notify off"

Disabling NOTIFY wake-up reintroduces the empty-read polling storm the listener exists to prevent (tens of millions of pgmq.read calls/day at default polling intervals). The fix is to make wake-ups cheap, not to remove them.

Acceptance

  • Job host with 5 capsules + 2 consumers: exactly 1 direct LISTEN connection (supervisor scope).
  • Preforking web server with N workers: exactly 1 streams LISTEN connection per host.
  • Kill -9 on the supervisor's listener connection: forks degrade to polling within one health-check cycle and recover when the listener reconnects.
  • Priority contract test green against real PostgreSQL.
  • Doctor prints the pinned direct-connection count.

Metadata

Metadata

Assignees

No one assigned

    Labels

    connectionsConnection handling, failover, self-healingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions