feat(deploy): health-checked rolling restarts — container-local /readyz, aligned shutdown budget, pgbus-health probe

[mhenrixon]

Summary

Implements #386 — makes the job container a correct participant in health-gated rolling restarts (Kamal dash-style: start new → poll healthcheck → stop old).

• Container-local /readyz on the standalone health server (lib/pgbus/web/health_app.rb, lib/pgbus/process/supervisor.rb, new lib/pgbus/process/readiness_snapshot.rb). ⚠️ Behavior change: the health_port server no longer serves the cluster-wide verdict — a new container can't pass the deploy gate on the old container's workers. States: BOOTING / OK / DEGRADED / DRAINING; the supervisor swaps an immutable snapshot per monitor pass, so the accept thread never touches @forks. The Rails-mounted HealthApp keeps the cluster verdict.
• Alignable shutdown budget (lib/pgbus/configuration.rb, supervisor.rb, worker.rb, consumer.rb). New config.shutdown_timeout, default drain_timeout + 5, replaces the supervisor's hardcoded 30s SIGKILL deadline; Consumer's drain wait follows drain_timeout; Worker's post-drain residual wait drops 30s → 5s. Documented rule: orchestrator stop_timeout > shutdown_timeout > drain_timeout.
• pgbus-health probe (exe/pgbus-health, lib/pgbus/health_probe.rb): stdlib-only, loads neither Bundler nor the gem (spec-enforced), exit 0/1/2 — cheap enough for 1–5s docker HEALTHCHECK intervals, no curl needed.
• Docs: README "Rolling restarts (Kamal, docker)" guide + probe-table split; docs-site "Rolling restarts" Operations page; CHANGELOG (Added + Changed, behavior change flagged).

Closes #386

Test plan

• bundle exec rake rubocop — 546 files, no offenses (root task; docs app rake lint also clean)
• Touched specs green together: 747 examples, 0 failures
• Full suite vs main, same seed (--seed 20260803, excluding the locally-unloadable spec/rubocop cop spec): identical 2 pre-existing i18n failures on both, zero regressions; branch adds 26 examples
• Docs app: 81 specs green (config-reference drift spec covers the new shutdown_timeout)
• End-to-end smoke: real exe/pgbus-health process against a real HealthServer — exit 0 on a ready snapshot, exit 1 on a draining one

Deviations & judgment calls

• expected children = fork-table size after boot_processes, stamped once. A child whose fork itself failed at boot (EAGAIN/ENOMEM) is not counted, so readiness could report OK without it — accepted because deriving the expectation from config is wrong in legitimate cases (scheduler role declines to boot when no recurring tasks exist). Rare failure, conservative direction (over-ready only for a child that also crash-looped at fork time).
• Readiness refresh is called directly from graceful_shutdown/immediate_shutdown, not only from the monitor loop, so DRAINING is visible to the very next probe instead of up to 1s later.
• Probe grew --timeout and --path flags beyond the issue's minimum (--port/env); parsing is hand-rolled (~15 lines) rather than optparse to keep the load profile at exactly socket.
• HealthApp's local-mode error path mirrors the cluster path (503 {"status":"ERROR"} + logged) — the issue specified only the happy-path body shape.
• Suite baseline discovery: the memory'd "~97–100 pre-existing failures" is stale — current baseline on main is 2 i18n failures plus one load error (spec/rubocop/cop/pgbus/no_ruby_timeout_spec.rb can't require "rubocop/pgbus" locally; pre-existing, excluded from the comparison run).
• Issue's manual smoke (probe against a live pgbus start) replaced with an equivalent that needs no DB: the probe exe against a real HealthServer + local snapshot, both readiness polarities. The DB-dependent boot path is covered by the existing supervisor #run specs.

Summary by CodeRabbit

• New Features

  • Added container-local readiness reporting with booting, healthy, degraded, and draining states.
  • Added the lightweight pgbus-health probe for deployment health checks.
  • Added configurable shutdown timing for graceful worker and supervisor termination.

• Documentation

  • Added guidance for rolling restarts, readiness endpoints, shutdown budgets, overlap safety, and forced termination impacts.
  • Clarified the difference between cluster-wide and container-local health checks.

• Bug Fixes

  • Replaced fixed shutdown waits with configurable drain and shutdown timeouts.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ae31ccca-2ecc-4fe0-a06a-221892958166

📥 Commits

Reviewing files that changed from the base of the PR and between 1e2041d and 28950b6.

📒 Files selected for processing (21)

• CHANGELOG.md
• README.md
• docs/app/models/config_reference.rb
• docs/app/models/doc.rb
• docs/app/views/docs/pages/observability.rb
• docs/app/views/docs/pages/rolling_restarts.rb
• exe/pgbus-health
• lib/pgbus/configuration.rb
• lib/pgbus/health_probe.rb
• lib/pgbus/process/consumer.rb
• lib/pgbus/process/readiness_snapshot.rb
• lib/pgbus/process/supervisor.rb
• lib/pgbus/process/worker.rb
• lib/pgbus/web/health_app.rb
• spec/pgbus/configuration_spec.rb
• spec/pgbus/health_probe_spec.rb
• spec/pgbus/process/consumer_spec.rb
• spec/pgbus/process/readiness_snapshot_spec.rb
• spec/pgbus/process/supervisor_spec.rb
• spec/pgbus/process/worker_spec.rb
• spec/pgbus/web/health_app_spec.rb

---

📝 Walkthrough

Walkthrough

The change adds container-local supervisor readiness, the pgbus-health probe, configurable shutdown budgets, bounded worker and consumer termination waits, rolling-restart documentation, and comprehensive tests.

Changes

Health and shutdown lifecycle

Layer / File(s)	Summary
Supervisor readiness flow lib/pgbus/process/readiness_snapshot.rb, lib/pgbus/process/supervisor.rb, lib/pgbus/web/health_app.rb, spec/pgbus/process/*, spec/pgbus/web/health_app_spec.rb	Supervisors publish boot, child-count, and shutdown state. Local /readyz returns readiness data without cluster analysis.
Configurable shutdown budget lib/pgbus/configuration.rb, lib/pgbus/process/{consumer,supervisor,worker}.rb, spec/pgbus/configuration_spec.rb, spec/pgbus/process/*_spec.rb	shutdown_timeout supports derived defaults, validation, warnings, and supervisor escalation. Consumers use drain_timeout, and workers use a five-second residual wait.
Standalone health probe lib/pgbus/health_probe.rb, exe/pgbus-health, spec/pgbus/health_probe_spec.rb	The dependency-free probe checks localhost HTTP readiness and returns separate healthy, unhealthy, and usage exit codes.
Rolling restart documentation README.md, CHANGELOG.md, docs/app/models/config_reference.rb, docs/app/models/doc.rb, docs/app/views/docs/pages/observability.rb, docs/app/views/docs/pages/rolling_restarts.rb	Documentation describes readiness states, probe configuration, shutdown timing, overlap behavior, and hard-kill effects.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Supervisor
  participant HealthApp
  participant HealthProbe
  Supervisor->>HealthApp: provide local readiness callback
  HealthProbe->>HealthApp: request /readyz
  HealthApp->>Supervisor: read readiness snapshot
  Supervisor-->>HealthApp: return status and child counts
  HealthApp-->>HealthProbe: return 200 or 503

Loading

Possibly related PRs

• mhenrixon/pgbus#112: Overlaps in consumer, worker, and supervisor execution-pool shutdown behavior.
• mhenrixon/pgbus#189: Relates to supervisor and worker drain handling.
• mhenrixon/pgbus#303: Relates to configurable shutdown timing in configuration and worker lifecycle code.

Suggested labels: enhancement, documentation

Poem

A rabbit checks readiness at dawn,
Workers drain before they’re gone.
The probe hops through localhost air,
Snapshots show the children there.
Timed shutdowns keep queues bright.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 32.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes container-local readiness, aligned shutdown timing, and the pgbus-health probe.
Linked Issues check	✅ Passed	The changes implement the coding objectives in direct issue [#386], including readiness, shutdown budgets, probe support, documentation, and tests.
Out of Scope Changes check	✅ Passed	The changes stay within issue [#386] and cover implementation, tests, documentation, configuration, and changelog updates.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-386-rolling-restart-readiness

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ast-grep (0.45.0)

spec/pgbus/configuration_spec.rb

ast-grep timed out on this file

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

lib/pgbus/process/supervisor.rb (1)

919-929: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Validate shutdown_timeout before Supervisor#run enters its ensure path. eager_validation = false allows invalid values to reach shutdown; Time.now + config.shutdown_timeout can then raise TypeError and mask the boot error.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/pgbus/process/supervisor.rb` around lines 919 - 929, Validate
config.shutdown_timeout before Supervisor#run can enter its ensure-driven
shutdown path, rejecting invalid values during startup rather than allowing
shutdown to perform Time.now arithmetic on them. Update the existing Supervisor
startup/config validation flow and preserve the shutdown deadline behavior for
valid timeout values.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/app/models/doc.rb`:
- Line 24: In docs/app/models/doc.rb, keep only the docs declaration for
“Rolling restarts” at line 45 under the Operations group and remove the
duplicate declarations at lines 24, 37, 49, 56, and 61, so the navigation
registers that page once.

In `@lib/pgbus/configuration.rb`:
- Around line 785-790: Update validate_shutdown_timeout! to accept nil only or a
finite, positive real numeric value, rejecting Float::INFINITY and non-real
Numeric implementations before shutdown calculations; add coverage for infinite
and non-real Numeric inputs while preserving the existing ConfigurationError and
message behavior.

In `@lib/pgbus/health_probe.rb`:
- Around line 45-52: Update the validation flow in HealthProbe#run to reject
ports outside 1–65535 and return usage_failure before calling probe. Also
validate the raw timeout input before conversion so non-numeric --timeout values
such as "abc" return the usage exit code 2 instead of becoming 0.0; preserve
valid timeout handling.

In `@lib/pgbus/process/supervisor.rb`:
- Around line 131-139: Update the boot lifecycle around mark_booted,
boot_processes, and fork_worker/fork_consumer to track the configured child-fork
intent separately from `@forks.size`, incrementing only for roles that should
actually boot and excluding legitimately disabled or empty roles. Set
`@expected_children` from this intent count at boot completion, then compare it
with the fork table so failed or rescued forks keep readiness unhealthy rather
than lowering the baseline.

In `@README.md`:
- Line 1202: Update the fenced code block in README.md to specify the text
language identifier, using ```text while preserving the existing timeline
content and formatting.

---

Outside diff comments:
In `@lib/pgbus/process/supervisor.rb`:
- Around line 919-929: Validate config.shutdown_timeout before Supervisor#run
can enter its ensure-driven shutdown path, rejecting invalid values during
startup rather than allowing shutdown to perform Time.now arithmetic on them.
Update the existing Supervisor startup/config validation flow and preserve the
shutdown deadline behavior for valid timeout values.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1269691f-8c85-4b10-be8c-56d10b9736d4

📥 Commits

Reviewing files that changed from the base of the PR and between 1ab300f and 1e2041d.

📒 Files selected for processing (21)

• CHANGELOG.md
• README.md
• docs/app/models/config_reference.rb
• docs/app/models/doc.rb
• docs/app/views/docs/pages/observability.rb
• docs/app/views/docs/pages/rolling_restarts.rb
• exe/pgbus-health
• lib/pgbus/configuration.rb
• lib/pgbus/health_probe.rb
• lib/pgbus/process/consumer.rb
• lib/pgbus/process/readiness_snapshot.rb
• lib/pgbus/process/supervisor.rb
• lib/pgbus/process/worker.rb
• lib/pgbus/web/health_app.rb
• spec/pgbus/configuration_spec.rb
• spec/pgbus/health_probe_spec.rb
• spec/pgbus/process/consumer_spec.rb
• spec/pgbus/process/readiness_snapshot_spec.rb
• spec/pgbus/process/supervisor_spec.rb
• spec/pgbus/process/worker_spec.rb
• spec/pgbus/web/health_app_spec.rb