feat(server): GitHub-driven self-update for thetadatadx-server

[userFRM]

What

thetadatadx-server updates itself from GitHub. Push a tagged release and running servers pick it up, verify it, swap themselves, and keep serving. No manual upgrade step, and no registry release for the server. Includes a remote restart lever.

Why

Today, shipping a server fix means cutting a release and every operator re-downloading. We want to push to GitHub and have running terminals update on their own: fast for urgent fixes, deferred to restart for routine ones.

How

Release side (cargo-dist on tag):

• Build every platform, sign each binary and a small update-manifest.json with minisign (private key in an Actions secret), publish binaries as Release assets, and write the manifest to gh-pages (CDN cached).
• The whole release step becomes: tag and push.

Manifest fields: version, priority (immediate or on-restart), rollout (percent or cohort), min_version (force-upgrade floor), restart_nonce, per-target {url, sha256}, signature.

Server side:

• Poll the manifest every ~10s, jittered, off the CDN so there is no rate limit. Effectively instant since CI build time dominates.
• Verify the minisign signature against a set of public keys baked into the binary (a set, not one, so keys rotate without bricking older builds). Reject anything unsigned or tampered.
• Newer version, priority immediate, and this server is in the rollout cohort: download the platform binary, verify sha256 and signature, atomic swap (self-replace), drain in-flight work, re-exec. If the new build fails its startup health check, roll back to the previous binary.
• Priority on-restart: stage it, apply on the next restart.
• restart_nonce changed: graceful restart (drain, re-exec). This is the remote reboot lever.
• Manifest points at an older version: controlled downgrade. This is how we un-ship a bad release fleet-wide.
• Server below min_version: force the upgrade.

Safety and fleet behavior:

• Signature verification before any swap, no exceptions. This is the line the vendor bootstrap crosses (fetch and exec, key in URL); we do not.
• Atomic swap plus local rollback on health-check failure, and remote rollback via the manifest.
• Fail-safe: any update error leaves the current version running. An update must never brick a working server.
• Staged rollout: rollout percent or cohort so a release ramps instead of hitting every server at once, plus a kill-switch to pause a rollout. A bad build must not be able to take down the whole fleet in one shot.
• Graceful drain on swap or restart: finish in-flight HTTP, let WS clients reconnect (the SDK already auto-reconnects).
• Opt-out: --no-auto-update and an env var for operators who pin a version.
• Observability: structured audit logs and metrics for every poll, update, restart, and rollback.

Scope:

• Only thetadatadx-server moves to GitHub self-update (drop its registry publish).
• The libraries (core crate, Python, TypeScript) keep their registry releases. A library cannot self-update and downstream developers install them normally.

Tooling

cargo-dist, axoupdater, self-replace, minisign.

Out of scope for v1

A full control-plane / fleet-management service. The manifest model leaves room to add an outbound telemetry and command channel later for live fleet visibility and targeted commands.

Test plan

• Signature accept and reject paths.
• Atomic swap and rollback on a health-check failure.
• Staged-rollout cohort selection (a server in vs out of the cohort).
• Remote restart via restart_nonce.
• Controlled downgrade and min_version force-upgrade.
• Fail-safe when the manifest is unreachable or the download fails.
• Opt-out flag.