[Bug]: the engine exits 1 on SIGTERM, so every clean systemctl stop leaves the unit failed

[ThefloorMiner]

Summary

A normal systemctl stop on a healthy engine leaves the unit in failed:

août 03 10:07:38 systemd[1]: Stopping colibri.service...
août 03 10:07:49 systemd[1]: colibri.service: Main process exited, code=exited, status=1/FAILURE
août 03 10:07:49 systemd[1]: colibri.service: Failed with result 'exit-code'.
août 03 10:07:49 systemd[1]: Stopped colibri.service.

Nothing went wrong. The engine had served for 2 h 50 min, systemd sent SIGTERM, and the process exited 1. systemd correctly records that as a failure.

Consequences

systemctl is-failed is useless as a health check. After any planned stop the unit reports failed, so the state cannot distinguish "was stopped on purpose" from "crashed". On a box where the engine is stopped and started routinely — benchmarking, model swaps, maintenance — the unit is in failed most of the time it is not running.

Every start needs reset-failed first. With StartLimitBurst set, a few stop/start cycles trip the rate limiter and the unit refuses to start at all until manually reset. Every automation script I wrote against this engine ends up carrying:

systemctl stop colibri
...
systemctl reset-failed colibri    # otherwise the next start may be refused
systemctl start colibri

which is a workaround for the exit code, not for anything real.

Two ways to fix it, in preference order

1. Exit 0 on SIGTERM. The engine already has a soft-stop path — colibri.c documents SIGINT closing the current turn "per la stessa via del tetto NGEN (stats, usage_save, KV append, sentinella END tutti normali)", i.e. stats flushed, .coli_usage saved, KV appended. SIGTERM should land in the same place and return 0. That also makes the shutdown correct, not just quiet: a SIGTERM today does not obviously go through usage_save, which on a long-lived serve process is the learned routing profile.

2. Failing that, document SuccessExitStatus=1 in the systemd example. This is a smaller fix and it is honest — it tells systemd that 1 is an expected shutdown code. But it also makes a genuine exit-1 failure invisible, so it is second best.

I would rather send the first as a PR than the second, but SIGTERM handling in a serve loop touches shutdown ordering and I would want to know whether you consider the SIGINT path safe to share before I go near it.

Reproduce

systemctl start colibri     # let it load and serve one request
systemctl stop colibri
systemctl is-failed colibri # -> failed

---

Observed on dev @ 7fb1159, Ubuntu 24.04, CUDA build sm_86, 4× RTX A6000, coli web under systemd. Not GLM-specific as far as I can tell — the exit path is in the serve loop, not the model code.

[JustVugg]

Confirmed in the source, and the cause is narrower than "the engine exits 1".

Nothing handles SIGTERM at all. colibri.c:6031 installs a handler for exactly one signal:

static void intr_install(void){
    struct sigaction sa; memset(&sa,0,sizeof(sa));
    sa.sa_handler=intr_sig; sigemptyset(&sa.sa_mask);
    sa.sa_flags=SA_RESTART;
    sigaction(SIGINT,&sa,NULL);      /* <- and only SIGINT */
}

The comment above it is explicit that the soft stop is for SIGINT. SIGTERM — the one systemd actually sends — was never in scope.

And the serve path's own exit is clean: colibri.c:9322

if(getenv("SERVE_BATCH") && ...) run_serve_mux(&m,snap);
else run_serve(&m,snap);
if(stats) stats_dump(&m,stats); return 0;

So a serve loop that ends normally returns 0. The status=1/FAILURE you are seeing is the process being torn down on a path that never reaches that line — which is why systemctl stop produces it every single time and a :quit from a client does not.

Your preferred fix is the right one

Handle SIGTERM the way SIGINT is handled — set the same g_intr flag — so a stop ends the current turn through the ordinary path: stats written, .coli_usage saved, KV appended, END sentinel emitted, then return 0. That is one line in intr_install plus making sure the flag actually breaks the serve read loop rather than only the generation loop.

The reason to prefer it over trapping and exiting immediately is the one you gave without stating it: the learned cache lives in that path. An engine killed before usage_save loses the session's expert history. On a box where you stop and start routinely — benchmarking, model swaps — that is the profile being discarded every time, which is a slow, invisible cost on top of the noisy unit state.

Worth noting intr_install is POSIX-only (#else static void intr_install(void){}), so this is a Linux/macOS/BSD fix and Windows is unaffected either way.

The consequence you documented is the strongest part of the report

systemctl is-failed being useless as a health check is worse than a cosmetic wrong exit code. A unit that is failed most of the time it is not running cannot be monitored, and the reset-failed you have to carry in every script is a workaround for a state that is not real. That is the argument that makes this a bug rather than a papercut, and it is why I would not accept SuccessExitStatus=1 in a unit file as the answer — it would hide a genuine crash exiting 1 as well.

Labelled bug. This is a small fix with a clear test (systemctl stop leaves the unit inactive, not failed) — happy for you to take it, and happy to take it myself if you would rather move on to the other four.