Problem
On transaction-pool PgBouncer platforms (PlanetScale Postgres and friends), every LISTEN connection must bypass the pooler and hit the direct port — and the direct slice of max_connections is the scarcest resource in the whole system (the pooler's server-side pools draw from the same ceiling, minus superuser-reserved slots).
pgbus's current LISTEN footprint scales with process count, not host count:
- Worker NotifyListener (
lib/pgbus/process/notify_listener.rb): one dedicated direct connection per worker fork.
- Consumer NotifyListener (
lib/pgbus/process/consumer.rb:366): one more per consumer fork.
- Streamer Listener (
lib/pgbus/web/streamer/listener.rb): one per web-server worker process.
A deployment with 5 worker capsules + 2 consumers on a job host and 2 web hosts × 4 Puma workers pins 7 + 8 = 15 direct connections at steady state — before migrations, consoles, or monitoring tools take theirs. On a small cluster (max_connections = 30) this collides with the pooler's own server pools and produces FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute.
The waste is structural: a single PG connection can LISTEN on any number of channels. N forks on one host listening to overlapping channel sets need one connection, not N.
Proposal
1. Supervisor-owned NotifyListener (job role): 1 direct connection per host
Move the NotifyListener up from Worker/Consumer forks into the Supervisor process:
Result: a job host's LISTEN footprint drops from worker_forks + consumer_forks to 1.
2. Streams listener consolidation (web role): 1 per host instead of 1 per Puma worker
Puma workers are forked from a master; the same self-pipe pattern applies:
- A Puma plugin (or
before_fork hook) starts the Streamer Listener in the master process; workers inherit read-pipes and get woken via IPC.
- Single-mode / non-preforking servers keep the current per-process listener (scope config again).
Result: web hosts go from puma_workers LISTEN connections to 1.
3. Make multi-queue read priority an explicit, documented, tested contract
capsule_dsl.rb documents "list order = strict priority", but the actual mechanism is incidental: read_multi builds pgmq.read(q1) UNION ALL pgmq.read(q2) ... LIMIT n (pgmq-ruby client/multi_queue.rb), and the ordering only holds because Postgres's Append node fills the LIMIT from subqueries in order. Nothing asserts it, and a future planner change or pgmq-ruby refactor could silently break it. Also note a side effect worth documenting: subqueries that execute claim messages (set vt) even when the outer LIMIT discards their rows — those messages go invisible for a full visibility timeout without being processed.
- Document the strict-priority semantics (and the vt-claim caveat) on
read_multi and in the capsule DSL docs.
- Add an integration test: enqueue on q1/q2/q3, read with a limit smaller than total, assert earlier-listed queues win.
- Consider making the ordering first-class (e.g. per-queue reads in list order with a shared budget, like
fetch_prioritized already does for priority sub-queues) so the contract doesn't depend on planner behavior.
4. Doctor: report the direct-connection budget
doctor already checks "Dedicated connections … connect OK". Extend it to count what the current config will pin (forks × listeners under :fork scope, 1 under :supervisor, streams scope, etc.) and print the number, so operators can do capacity math from the doctor output alone.
Why not "just turn worker_notify off"
Disabling NOTIFY wake-up reintroduces the empty-read polling storm the listener exists to prevent (tens of millions of pgmq.read calls/day at default polling intervals). The fix is to make wake-ups cheap, not to remove them.
Acceptance
- Job host with 5 capsules + 2 consumers: exactly 1 direct LISTEN connection (supervisor scope).
- Preforking web server with N workers: exactly 1 streams LISTEN connection per host.
- Kill -9 on the supervisor's listener connection: forks degrade to polling within one health-check cycle and recover when the listener reconnects.
- Priority contract test green against real PostgreSQL.
- Doctor prints the pinned direct-connection count.
Problem
On transaction-pool PgBouncer platforms (PlanetScale Postgres and friends), every LISTEN connection must bypass the pooler and hit the direct port — and the direct slice of
max_connectionsis the scarcest resource in the whole system (the pooler's server-side pools draw from the same ceiling, minus superuser-reserved slots).pgbus's current LISTEN footprint scales with process count, not host count:
lib/pgbus/process/notify_listener.rb): one dedicated direct connection per worker fork.lib/pgbus/process/consumer.rb:366): one more per consumer fork.lib/pgbus/web/streamer/listener.rb): one per web-server worker process.A deployment with 5 worker capsules + 2 consumers on a job host and 2 web hosts × 4 Puma workers pins
7 + 8 = 15direct connections at steady state — before migrations, consoles, or monitoring tools take theirs. On a small cluster (max_connections = 30) this collides with the pooler's own server pools and producesFATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute.The waste is structural: a single PG connection can
LISTENon any number of channels. N forks on one host listening to overlapping channel sets need one connection, not N.Proposal
1. Supervisor-owned NotifyListener (job role): 1 direct connection per host
Move the NotifyListener up from Worker/Consumer forks into the Supervisor process:
NotifyListeneron the union of all capsules' + consumers' physical queue channels (channel names are already deterministic:pgmq.q_<prefix>_<queue>.INSERT).fork) fits the existingWakeSignalshape (lib/pgbus/process/wake_signal.rb); the fork's wait loop selects on the pipe instead of (or in addition to) its own LISTEN connection.NotifyListener failed to startpath.worker_notify_scope = :supervisor | :fork(default:forkfor one release, flip default after burn-in), so the change is opt-in and reversible.Result: a job host's LISTEN footprint drops from
worker_forks + consumer_forksto 1.2. Streams listener consolidation (web role): 1 per host instead of 1 per Puma worker
Puma workers are forked from a master; the same self-pipe pattern applies:
before_forkhook) starts the Streamer Listener in the master process; workers inherit read-pipes and get woken via IPC.Result: web hosts go from
puma_workersLISTEN connections to 1.3. Make multi-queue read priority an explicit, documented, tested contract
capsule_dsl.rbdocuments "list order = strict priority", but the actual mechanism is incidental:read_multibuildspgmq.read(q1) UNION ALL pgmq.read(q2) ... LIMIT n(pgmq-rubyclient/multi_queue.rb), and the ordering only holds because Postgres's Append node fills the LIMIT from subqueries in order. Nothing asserts it, and a future planner change or pgmq-ruby refactor could silently break it. Also note a side effect worth documenting: subqueries that execute claim messages (setvt) even when the outer LIMIT discards their rows — those messages go invisible for a full visibility timeout without being processed.read_multiand in the capsule DSL docs.fetch_prioritizedalready does for priority sub-queues) so the contract doesn't depend on planner behavior.4. Doctor: report the direct-connection budget
doctoralready checks "Dedicated connections … connect OK". Extend it to count what the current config will pin (forks × listeners under:forkscope, 1 under:supervisor, streams scope, etc.) and print the number, so operators can do capacity math from the doctor output alone.Why not "just turn worker_notify off"
Disabling NOTIFY wake-up reintroduces the empty-read polling storm the listener exists to prevent (tens of millions of
pgmq.readcalls/day at default polling intervals). The fix is to make wake-ups cheap, not to remove them.Acceptance