Fix FD-leak self-shutdown becoming a half-dead zombie

[rephapeng]

Summary

Three fixes for the recurring failure where evonic.service dies on its own — the docker-backend watchdog SIGTERMs the process at fd > 400, but the process kept surviving as a half-dead zombie (runtime drained, web server still serving), so systemd never restarted it.

Background

The FD-watchdog (backend/tools/lib/backends/docker_backend.py) sends SIGTERM to the process when the open-FD count climbs past 400, to prevent a cascade. The original FD source was an SFTP permission-denied retry loop (fixed in #3 below). After that, two further problems remained:

Fixes

1. Close the rate-limiter's thread-local SQLite connections on teardown (app.py)

The dominant remaining FD leak. Flask's dev server runs threaded=True (one thread per request). The before_request rate-limit check opens a thread-local SQLite connection — 3 FDs each in WAL mode (db + -wal + -shm). teardown_request closed the main db connection but not the api_rate_limit / rate_limit ones, so they accumulated until GC — ~180 FDs on api_rate_limit.db alone, tripping the watchdog.

Fix: teardown_request now also calls api_rate_limit.close() and rate_limit.close(), mirroring the existing db.close() pattern.

Verified: open handles on api_rate_limit.db stayed flat at 8 across 70+ requests (previously grew unbounded toward 180+).

2. Guarantee the process actually exits on SIGTERM (backend/agent_runtime/runtime.py)

_signal_handler runs graceful_shutdown() then sys.exit(0). But sys.exit() only raises SystemExit in the main thread, and when SIGTERM arrives while the main thread is blocked in the threaded WSGI accept loop, the server swallows that SystemExit. The result: queue workers + background executor stop, but the process keeps serving requests. systemd still sees it as active, so Restart=always never fires.

Fix: arm a daemon threading.Timer(WORKER_JOIN_TIMEOUT_SECONDS + 3, os._exit(0)) backstop before sys.exit(0). The clean exit (running atexit Docker cleanup) is still attempted first; the backstop guarantees the process dies — and thus restarts — if the clean exit is swallowed.

3. Resolve the remote evonic dir against the REMOTE $HOME (backend/tools/lib/backends/ssh_backend.py)

Root cause of the original FD leak. _ensure_evonic_on_remote used os.path.expanduser('~/.evonic/evonic'), which expands ~ against the local process HOME (/root, since evonic runs as root). On the deploy VPS we log in as ubuntu, who can't write under /root, producing an endless SFTP [Errno 13] Permission denied retry loop (~every 10–20s) that leaked sockets until fd > 400.

Fix: _resolve_remote_evonic_dir() resolves the leading ~ via the remote shell (printf %s "$HOME"), cached per connection.

Testing

• ast.parse(...) on both changed Python modules — parse OK
• Imports models.api_rate_limit.close / models.rate_limit.close resolve
• Live: restarted the service, hammered 70+ /api/* requests, confirmed FD count stays bounded and queue workers/scheduler come back up