Headless observability: no JSON log format, and stderr logging is lost when the TUI does not start

[vnovick]

Summary

Running itervox headless (systemd unit, container, cloud VM) is currently hard to
observe, for two independent reasons:

1. There is no structured (JSON) log format, so every cloud log-based metric or alert has
   to regex unstructured text instead of querying a field.
2. Headless runs lose the stderr sink entirely after startup, so journalctl -u itervox
   shows the banner and then goes quiet.

Neither is a correctness bug; together they make the daemon meaningfully harder to run in
production than it needs to be. Both look cheap to fix.

1. No JSON log format

The two sinks are wired in cmd/itervox/main.go:359:

• file → slog.NewTextHandler (logfmt: level=ERROR msg="...")
• stderr → charmlog human format with ANSI colors

There is no JSON option and no flag to select one. Consequences for cloud ingestion:

• GCP Cloud Logging / CloudWatch / Azure Monitor all parse JSON natively into queryable
  fields. With logfmt they receive one opaque textPayload string.
• Log-based metrics degrade to regexes like textPayload =~ "level=ERROR", which break
  silently whenever a message is reworded.
• The structured attrs the codebase already sets carefully ("issue", identifier,
  "host", host, "error", err) are invisible to the log backend — the effort of
  structured logging is spent and then discarded at the boundary.

Suggested fix: a --log-format=text|json flag (env: ITERVOX_LOG_FORMAT) selecting
slog.NewJSONHandler for the file sink. slog gives this for free; the change is
roughly the handler construction plus flag plumbing. Defaulting to text keeps local UX
identical, and deployments opt in.

2. Headless runs lose stderr logging

cmd/itervox/main.go:740 redirects the default logger to the file sink only,
immediately before statusui.Run:

// Redirect slog to file-only before the TUI takes the alt-screen.
// Without this, concurrent slog writes to stderr corrupt the bubbletea display.
slog.SetDefault(slog.New(logging.NewRedactingHandler(
    slog.NewTextHandler(fileWriter, &slog.HandlerOptions{Level: logLevel}))))
...
tuiDone := statusui.Run(ctx, snap, logBuf, tuiCfg, tuiCancel)

The reason is sound — concurrent stderr writes corrupt the alt-screen. But the redirect is
unconditional, while the TUI itself is not: statusui.Run calls
checkForegroundTTYOwnershipWithRetry() (internal/statusui/tty_guard.go:40) and returns
early with a closed channel when there is no controlling terminal.

So in the headless case the daemon pays the cost of the redirect (stderr goes dark) while
getting none of the benefit (no TUI was started). Under systemd that means journald
captures the startup lines and nothing else, which is the opposite of what an operator
expects from a service.

Note this also affects the stderrOnly logger used for the one-time dashboard-URL-with-token
line — that one still reaches stderr, which is correct and shouldn't change.

Suggested fix: gate the redirect on whether the TUI actually started. statusui.Run
already knows; it could return that fact, or run() could check TTY ownership before
deciding. Headless then keeps the stderr+file fanout it had at startup and journald works
normally.

Evidence level: this is from reading the code path (main.go:740 → statusui.Run →
tty_guard.go:40), not from a runtime capture on a systemd box. Worth confirming
empirically before fixing, but the ordering is unambiguous in source.

Why both matter now

deploy/ (added alongside this issue) documents cloud VM deployment for GCP/AWS/Azure,
and both items required a workaround in the docs: point the logging agent at the rotating
file rather than journald, and match logfmt with regexes rather than field queries. Fixing
these would let that guidance collapse to "install the agent, done."

Related

• Discovered while writing the cloud deployment guide, alongside Reverse-tunnel/reverse-proxy deployments run with no auth: token auto-generation keys off bind address, not reachability #48.