[pgmq-framework] Implement stuck job detection

[mensfeld]

Summary

Implement stuck job detection for the planned pgmq-framework gem, inspired by RiverQueue's implementation.

Context

This feature was considered for pgmq-ruby but is a framework-level concern, not appropriate for the low-level client. It requires:

• Periodic monitoring/scanning of queues
• Threshold configuration
• Logging/alerting infrastructure
• Job execution lifecycle tracking

Feature Description

Detect "stuck" jobs - messages that fail to progress due to worker crashes, deadlocks, or other issues. A message could be considered stuck when:

1. High read count - Message has been read many times (read_ct) but never deleted/archived (indicating repeated failures)
2. Long processing time - Message visibility timeout keeps expiring without completion
3. Old age - Message has been in queue beyond expected processing time

Proposed Components

StuckJobDetector

Periodic scanner that identifies stuck messages based on configurable thresholds:

detector = PGMQ::Framework::StuckJobDetector.new(
  client: pgmq_client,
  max_read_count: 5,           # Messages read more than 5 times
  max_age_seconds: 3600,       # Messages older than 1 hour
  scan_interval: 60            # Check every minute
)
detector.on_stuck { |message, reason| Logger.warn("Stuck: #{message.msg_id}") }
detector.start

Dead Letter Queue (DLQ) Support

Automatically move stuck messages to a DLQ for manual inspection:

detector.on_stuck do |message, reason|
  client.archive(message.queue_name, message.msg_id)
  client.send("#{message.queue_name}_dlq", message.message)
end

Instrumentation Hooks

Integration with observability tools:

PGMQ::Framework.configure do |config|
  config.on_stuck_job = ->(msg, reason) { StatsD.increment("pgmq.stuck_jobs") }
end

Low-level primitives already available in pgmq-ruby

Primitive	How it helps
metrics(queue_name)	Returns queue length, oldest message age
Message#read_ct	Count of times message was read
Message#vt	Visibility timeout expiration
Message#enqueued_at	When message was added
read_batch with conditional:	Could filter for high read_ct messages

References

• RiverQueue PR #1097 - Basic stuck job detection
• Related patterns: Job monitoring, observability in distributed systems, dead letter queues

---

This issue tracks a feature for the planned pgmq-framework gem, not pgmq-ruby.