`connection_lost_error?` misses SSL-layer EOF and relies on fragile message matching

[mhenrixon]

Summary

PGMQ::Connection#connection_lost_error? (lib/pgmq/connection.rb:113)
matches exclusively on the exception message. A transient Postgres drop
surfaces at the SSL layer as
PQconsumeInput() SSL error: unexpected eof while reading — a message that
isn't in the list — and the single-retry path in with_connection
(line 79) is skipped. The error propagates as PGMQ::Errors::ConnectionError
even though the connection would recover on the very next checkout.

Observed with pgmq-ruby 0.6.1 behind pgbus 0.7.5 on Rails 8.1.3.
A parallel gap exists on the pgbus side — filed as
zoolutions/pgbus#143 — both fixes are needed for full coverage, but this
issue tracks the pgmq-ruby half.

Repro / production evidence

Three unrelated controllers hit the same enqueue-time exception in a
six-minute window after a brief Postgres/SSL blip. Identical message on all
three:

Database connection error: PQconsumeInput() SSL error: unexpected eof while reading

Stack tail (top frames are pgmq-ruby):

pgmq-ruby (0.6.1) lib/pgmq/connection.rb:81:in 'PGMQ::Connection#with_connection'
pgmq-ruby (0.6.1) lib/pgmq/client.rb:102:in 'PGMQ::Client#with_connection'
pgmq-ruby (0.6.1) lib/pgmq/client/queue_management.rb:23:in 'PGMQ::Client::QueueManagement#create'

An earlier cluster in the same app raised the same exception class with a
different message — PQsocket() can't get socket descriptor — which IS in
connection_lost_error? and was retried successfully. The machinery works;
only this specific message isn't covered.

Why verify_connection! doesn't catch it

verify_connection! runs before yield conn and checks
conn.finished? / conn.status == PG::CONNECTION_BAD. For an idle
connection silently killed by the SSL peer, PQstatus() often still
reports CONNECTION_OK until the next I/O, because libpq only flips the
status after a failed operation. The first I/O inside the yielded block is
what raises PQconsumeInput() SSL error: unexpected eof, and at that point
execution is already in the rescue PG::Error branch — where matching
depends on connection_lost_error?.

Root cause

lib/pgmq/connection.rb:

lost_connection_messages = [
  "server closed the connection",
  "connection not open",
  "connection is closed",
  "connection has been closed",
  "no connection to the server",
  "terminating connection",
  "connection to server was lost",
  "could not receive data from server",
  "pqsocket() can't get socket descriptor"
]

message = error.message.to_s.downcase
lost_connection_messages.any? { |pattern| message.include?(pattern) }

This list already includes mid-flight signals like
"server closed the connection" and
"could not receive data from server", so the "retry might duplicate a
partial INSERT" tradeoff is already accepted at this layer. The SSL EOF
variant fits the same category and should be covered.

Proposed fix

Two angles — I'd recommend doing both. (1) fixes today's production
symptom in the smallest patch. (2) makes detection robust to future
SSL/TLS/TCP/pooler message variants.

1. Extend the message list (narrow)

lost_connection_messages = [
  "server closed the connection",
  "connection not open",
  "connection is closed",
  "connection has been closed",
  "no connection to the server",
  "terminating connection",
  "connection to server was lost",
  "could not receive data from server",
  "pqsocket() can't get socket descriptor",
  "ssl error: unexpected eof",   # observed in production
  "ssl syscall error"            # related SSL-layer teardown
]

2. Prefer class-matching over message-matching (structural)

The pg gem already defines dedicated connection-failure classes:

PG::ConnectionBad < PG::Error
PG::UnableToSend  < PG::Error

Matching them directly removes the need to enumerate every OS/TLS/pooler
message variant:

def connection_lost_error?(error)
  return true if error.is_a?(PG::ConnectionBad) || error.is_a?(PG::UnableToSend)

  message = error.message.to_s.downcase
  LOST_CONNECTION_PATTERNS.any? { |pattern| message.include?(pattern) }
end

Caveat worth verifying: confirm the pg gem raises
PG::ConnectionBad/PG::UnableToSend for the SSL EOF path on modern
versions. If libpq raises a bare PG::Error there, the class check is a
no-op on this specific case and the message fallback still handles it — so
the combination is strictly safer than message-only.

Optional last-resort signal: error.connection&.status == PG::CONNECTION_BAD.
PG::Error#connection is nil for some errors, so this is a fallback rather
than primary.

Test strategy

• connection_lost_error? returns true for a PG::Error with each new
  message ("...SSL error: unexpected eof...", "...SSL SYSCALL error...").
• connection_lost_error? returns true for a raised PG::ConnectionBad
  and PG::UnableToSend regardless of message (including empty / non-
  matching messages).
• connection_lost_error? returns false for a generic
  PG::Error.new("syntax error") — no behavior change for query-level
  errors.
• with_connection performs exactly one retry when the first yield raises
  each new case, and raise PGMQ::Errors::ConnectionError still fires when
  the retry also fails.

Version info

• pgmq-ruby: 0.6.1
• pg (gem): modern release that ships PG::ConnectionBad / PG::UnableToSend
• Ruby: 4.0.2
• Postgres: managed, pooler in front
• Observed: three clustered failures in 6 minutes followed by automatic
  recovery — consistent with transient SSL teardown on idle pooled
  connections.

Cross-reference

• Enqueue path does not retry on SSL EOF and ensure_queue is outside the stale-connection retry zoolutions/pgbus#143 — pgbus has a parallel gap (same pattern missing
  from its own STALE_CONNECTION_PATTERNS, plus ensure_queue called
  outside the retry). Fixing only one of the two gems leaves a narrower
  but still real window.

[mensfeld]

@mhenrixon shouldnt we close it?

[mhenrixon]

@mhenrixon shouldnt we close it?

Let me have a look first, want to make sure it isn't something I forgot to upstream.