Skip to content

#275 handle SIGTERM with graceful shutdown in entry.sh - #275

Open
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:275-sigterm-handler
Open

#275 handle SIGTERM with graceful shutdown in entry.sh#275
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:275-sigterm-handler

Conversation

@VasilevNStas

Copy link
Copy Markdown
Contributor

When the orchestrator (Docker, container scheduler, or manual cancel) sends SIGTERM to the container, entry.sh currently has no signal handler. The process is killed immediately — stdout is lost, and baza.rb cannot distinguish between "job was cancelled" and "job disappeared due to infrastructure failure".

The problem

# Before: SIGTERM arrives → silent death, no output
kill -TERM <entry.sh-pid>
# → nothing in stdout, baza.rb sees mystery disappearance

What this PR adds

trap cleanup SIGTERM

cleanup() {
  elapsed=$(($(date +%s) - start_time))
  echo "Job ${id} terminated after ${elapsed}s"
  exit 143
}
Scenario Before After
SIGTERM during job Silent death, no output "Job X terminated after Ys" + exit 143
Normal completion exit 0 exit 0 (unchanged)
baza.rb polling Sees "job disappeared" Sees finished? == yes, exit_code == 143

Why exit 143?

Exit code 143 = 128 + 15 (SIGTERM). This is the POSIX shell convention: when a process is killed by signal N, the shell reports exit code 128+N. Using 143 allows baza.rb (or any caller) to distinguish:

  • 143: terminated by SIGTERM (intentional cancel)
  • 1-127: normal error (judge logic failure, missing files, etc.)
  • 0: success

Signal flow

  1. Orchestrator sends SIGTERMentry.sh trap fires
  2. cleanup() prints diagnostic and calls exit 143
  3. Script terminates, child processes inherit signal handling cleanup

Docker, Kubernetes, and most container runtimes send SIGTERM first, then SIGKILL after a grace period. This trap handles the graceful phase.

Checklist

  • bundle exec rubocop — 0 offenses
  • bundle exec rake — all tasks pass
  • HoC ≤ 133

@yegor256 please review

@VasilevNStas
VasilevNStas requested a review from yegor256 as a code owner June 29, 2026 16:37
VasilevNStas added a commit to VasilevNStas/swarm-template that referenced this pull request Jun 29, 2026
When the orchestrator (Docker, container scheduler, or manual cancel)
sends SIGTERM to the container, entry.sh previously had no signal
handler. The process was killed immediately — stdout was lost, and
baza.rb could not distinguish between "job was cancelled" and "job
disappeared due to infrastructure failure".

This commit adds a trap on SIGTERM that:
- Captures the elapsed wall time since entry.sh started
- Prints "Job <id> terminated after <elapsed>s" to stdout
- Exits with code 143 (128 + 15 = SIGTERM), the standard convention
  for signal-induced termination

The orchestrator must send SIGTERM (not SIGKILL) for this to work,
which is the default behaviour of Docker, Kubernetes, and most
container runtimes.
@VasilevNStas
VasilevNStas force-pushed the 275-sigterm-handler branch from 81380cb to 903b21d Compare June 29, 2026 16:44
@VasilevNStas

Copy link
Copy Markdown
Contributor Author

@yegor256 — this PR is part of a 3-PR series that addresses a critical reliability gap in the baza.rb ↔ entry.sh pipeline.

The problem: When baza.rb pushes a job, entry.sh runs judges update with no time limit, no convergence detection, and no signal handling. If judges hang or fail to converge, the container runs forever and baza.rb's polling of finished? times out after 10 minutes with no diagnostic information — the user sees a silent failure.

When the orchestrator (Docker, container scheduler, or manual cancel)
sends SIGTERM to the container, entry.sh previously had no signal
handler. The process was killed immediately — stdout was lost, and
baza.rb could not distinguish between "job was cancelled" and "job
disappeared due to infrastructure failure".

This commit adds a trap on SIGTERM that:
- Captures the elapsed wall time since entry.sh started
- Prints "Job <id> terminated after <elapsed>s" to stdout
- Exits with code 143 (128 + 15 = SIGTERM), the standard convention
  for signal-induced termination

The orchestrator must send SIGTERM (not SIGKILL) for this to work,
which is the default behaviour of Docker, Kubernetes, and most
container runtimes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant