#273 wall-clock timeout in entry.sh - #273
Open
VasilevNStas wants to merge 1 commit into
Open
Conversation
VasilevNStas
added a commit
to VasilevNStas/swarm-template
that referenced
this pull request
Jun 29, 2026
When baza.rb pushes a job to the server, entry.sh runs judges update to
process the factbase. Previously there was no time limit — if judges
hung (e.g. due to a bug, network issue, or unexpected factbase state),
the container would run forever and baza.rb's polling of finished?
would time out after 10 minutes with no clear indication of what went
wrong.
This commit wraps the judges call in GNU timeout(1):
timeout ${JOB_TIMEOUT:-600} judges update ...
- Default timeout: 600 seconds (10 minutes), matching baza.rb's
polling ceiling. Configurable via JOB_TIMEOUT env variable.
- On timeout: judges receives SIGTERM, then SIGKILL after default
grace period. Exit code 124 (GNU timeout standard) propagates
to the container, so baza.rb receives a deterministic "timed out"
status instead of waiting indefinitely for finished? to become true.
- The timeout command is part of GNU coreutils and is available in
all Linux-based Docker images.
VasilevNStas
force-pushed
the
273-timeout
branch
from
June 29, 2026 16:41
b3f81e3 to
90d7158
Compare
Contributor
Author
|
@yegor256 — this PR is part of a 3-PR series that addresses a critical reliability gap in the baza.rb ↔ entry.sh pipeline. The problem: When baza.rb pushes a job, entry.sh runs judges update with no time limit, no convergence detection, and no signal handling. If judges hang or fail to converge, the container runs forever and baza.rb's polling of finished? times out after 10 minutes with no diagnostic information — the user sees a silent failure. |
When baza.rb pushes a job to the server, entry.sh runs judges update to
process the factbase. Previously there was no time limit — if judges
hung (e.g. due to a bug, network issue, or unexpected factbase state),
the container would run forever and baza.rb's polling of finished?
would time out after 10 minutes with no clear indication of what went
wrong.
This commit wraps the judges call in GNU timeout(1):
timeout ${JOB_TIMEOUT:-600} judges update ...
- Default timeout: 600 seconds (10 minutes), matching baza.rb's
polling ceiling. Configurable via JOB_TIMEOUT env variable.
- On timeout: judges receives SIGTERM, then SIGKILL after default
grace period. Exit code 124 (GNU timeout standard) propagates
to the container, so baza.rb receives a deterministic "timed out"
status instead of waiting indefinitely for finished? to become true.
- The timeout command is part of GNU coreutils and is available in
all Linux-based Docker images.
VasilevNStas
force-pushed
the
273-timeout
branch
from
June 29, 2026 16:54
90d7158 to
e1641ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
baza.rbpushes a job to the server,entry.shrunsjudges updateto process the factbase. Previously there was no time limit — if judges hung (e.g. due to a bug, network issue, or unexpected factbase state), the container would run forever andbaza.rb's polling offinished?would time out after 10 minutes with no clear indication of what went wrong.This commit wraps the
judgescall in GNUtimeout(1):timeout ${JOB_TIMEOUT:-600} judges update --summary --max-cycles=3 --no-log ...What this changes
timeoutenforces a hard ceilingbaza.rbsees only "job disappeared" after 10 min"timed out"in stdouttimeoutsends SIGTERM → SIGKILL automaticallyKey details
baza.rb's polling ceilingJOB_TIMEOUTenvironment variablebaza.rba deterministic status instead of ambiguous "still running?""timeout: the command timed out"appears in stdoutWhy not use the existing
--lifetimeflag?The
judgesgem has a--lifetimeoption, but it is an internal Ruby timeout. An OS-leveltimeoutis more robust — it catches any hang, including Ruby VM freezes, native extension deadlocks, and blocked I/O.Compatibility
timeout(1)is part of GNU coreutils and is available in every Linux-based Docker image (all our CI images,yegor256/rultor-image, etc.). macOS users needbrew install coreutils(gtimeout) for local testing, but production always runs on Linux.Checklist
bundle exec rubocop— 0 offensesbundle exec rake— all tasks pass@yegor256 please review