Skip to content

#273 wall-clock timeout in entry.sh - #273

Open
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:273-timeout
Open

#273 wall-clock timeout in entry.sh#273
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:273-timeout

Conversation

@VasilevNStas

Copy link
Copy Markdown
Contributor

When baza.rb pushes a job to the server, entry.sh runs judges update to process the factbase. Previously there was no time limit — if judges hung (e.g. due to a bug, network issue, or unexpected factbase state), the container would run forever and baza.rb's polling of finished? would time out after 10 minutes with no clear indication of what went wrong.

This commit wraps the judges call in GNU timeout(1):

timeout ${JOB_TIMEOUT:-600} judges update --summary --max-cycles=3 --no-log ...

What this changes

Before After
No time limit — container hangs forever timeout enforces a hard ceiling
baza.rb sees only "job disappeared" after 10 min Exit code 124 + "timed out" in stdout
Ops have to manually kill stray containers timeout sends SIGTERM → SIGKILL automatically

Key details

  • Default timeout: 600 seconds (10 minutes), matching baza.rb's polling ceiling
  • Configuration: override via JOB_TIMEOUT environment variable
  • Exit code: 124 (GNU timeout standard) propagates to the container, giving baza.rb a deterministic status instead of ambiguous "still running?"
  • Diagnostics: "timeout: the command timed out" appears in stdout

Why not use the existing --lifetime flag?

The judges gem has a --lifetime option, but it is an internal Ruby timeout. An OS-level timeout is more robust — it catches any hang, including Ruby VM freezes, native extension deadlocks, and blocked I/O.

Compatibility

timeout(1) is part of GNU coreutils and is available in every Linux-based Docker image (all our CI images, yegor256/rultor-image, etc.). macOS users need brew install coreutils (gtimeout) for local testing, but production always runs on Linux.

Checklist

  • bundle exec rubocop — 0 offenses
  • bundle exec rake — all tasks pass
  • HoC ≤ 133

@yegor256 please review

@VasilevNStas
VasilevNStas requested a review from yegor256 as a code owner June 29, 2026 16:35
VasilevNStas added a commit to VasilevNStas/swarm-template that referenced this pull request Jun 29, 2026
When baza.rb pushes a job to the server, entry.sh runs judges update to
process the factbase. Previously there was no time limit — if judges
hung (e.g. due to a bug, network issue, or unexpected factbase state),
the container would run forever and baza.rb's polling of finished?
would time out after 10 minutes with no clear indication of what went
wrong.

This commit wraps the judges call in GNU timeout(1):

  timeout ${JOB_TIMEOUT:-600} judges update ...

- Default timeout: 600 seconds (10 minutes), matching baza.rb's
  polling ceiling. Configurable via JOB_TIMEOUT env variable.
- On timeout: judges receives SIGTERM, then SIGKILL after default
  grace period. Exit code 124 (GNU timeout standard) propagates
  to the container, so baza.rb receives a deterministic "timed out"
  status instead of waiting indefinitely for finished? to become true.
- The timeout command is part of GNU coreutils and is available in
  all Linux-based Docker images.
@VasilevNStas

Copy link
Copy Markdown
Contributor Author

@yegor256 — this PR is part of a 3-PR series that addresses a critical reliability gap in the baza.rb ↔ entry.sh pipeline.

The problem: When baza.rb pushes a job, entry.sh runs judges update with no time limit, no convergence detection, and no signal handling. If judges hang or fail to converge, the container runs forever and baza.rb's polling of finished? times out after 10 minutes with no diagnostic information — the user sees a silent failure.

When baza.rb pushes a job to the server, entry.sh runs judges update to
process the factbase. Previously there was no time limit — if judges
hung (e.g. due to a bug, network issue, or unexpected factbase state),
the container would run forever and baza.rb's polling of finished?
would time out after 10 minutes with no clear indication of what went
wrong.

This commit wraps the judges call in GNU timeout(1):

  timeout ${JOB_TIMEOUT:-600} judges update ...

- Default timeout: 600 seconds (10 minutes), matching baza.rb's
  polling ceiling. Configurable via JOB_TIMEOUT env variable.
- On timeout: judges receives SIGTERM, then SIGKILL after default
  grace period. Exit code 124 (GNU timeout standard) propagates
  to the container, so baza.rb receives a deterministic "timed out"
  status instead of waiting indefinitely for finished? to become true.
- The timeout command is part of GNU coreutils and is available in
  all Linux-based Docker images.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant