fix(daemon): unbounded stderr rotated the tray diagnostics out of the log - #22
Merged
Conversation
… log Release logging was added so a tray failure leaves evidence. Two unbounded stderr sources defeated it: the GUI forwards this process's output into a log capped at 256 KB with KeepOne, so enough volume rotates the file and takes the startup and tray lines with it — exactly when a user finally reports the problem. The reconnect loop retried every 2s and logged two lines per cycle for as long as the failure lasted: 434 KB/hour against a held port, rotating the file every ~35 minutes. It now backs off 2s→30s and collapses an unchanged failure to the first three then one in ten, reporting the suppressed count when it next speaks. A changed error is always logged, because a new error is new information. Measured against a held port: 25 KB/hour, a 94% reduction. `ws_server` no longer logs the bind failure itself. It logged "ws bind failed" while the loop logged "ws_server exited" for the same event — two lines per cycle, and the throttle state could only live in the loop. The loop now owns the message and the error carries the address. The larger source was worse and only shows on a fresh install: the startup backfill logged one line per ingested turn, which on this machine was 16,252 lines and 876 KB in 60 seconds — three times over the cap, on the first launch, which is the launch a new user is most likely to need a log for. Per-turn lines are now scoped to live turns via the existing `broadcast_live` flag; seeing live turns arrive is the point of the tool and they are a few per minute. The backfill reports one total instead. Fresh-install output: 301 lines, 56 KB. `LUMEN_LOG=debug|trace` restores everything, using the same variable the GUI already takes its level from rather than inventing a second one. Eight tests cover the two decisions as pure functions, including that the delay cannot overflow or reach zero for any attempt count, and that an hour of one unchanged failure stays between 3 and 20 lines — bounded, but never silent. Closes #7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7.
Release logging exists so a tray failure leaves evidence. The GUI forwards the daemon's stderr into
its own log, which is capped at 256 KB with
KeepOne— so enough daemon output rotates the file andtakes the startup and tray lines with it, at exactly the moment someone finally reports a problem.
There were two unbounded sources. The issue named one; measuring the fix found the other, which
is 100× larger.
1. The reconnect loop — 434 KB/hour
Retried every 2s and logged two lines per cycle for as long as the failure lasted. Against a held
port that rotated the log every ~35 minutes.
asked more often, and the 30s cap bounds recovery (the GUI retries its own connection every 2s).
suppressed when it next speaks. A changed error always logs — a new error is new information.
ws_serverno longer logs the bind failure itself. It printedws bind failedwhile the loopprinted
ws_server exitedfor the same event: two lines per cycle, and the throttle state couldonly live in the loop. The loop owns the message now; the error carries the address.
2. The startup backfill — 876 KB in 60 seconds
Not in the issue, and much worse. The initial import logged one line per ingested turn: on this
machine 16,252 lines and 876 KB in a minute — over three times the entire log cap, on the first
launch, which is the launch a new user is most likely to need a log for.
Scoped to live turns using the
broadcast_liveflag that already distinguishes them. Seeing liveturns arrive is the point of the tool and they are a few per minute; the backfill now reports one
total line instead.
I included this rather than filing it separately because without it the PR would claim to fix #7
while its own acceptance criterion — diagnostics survive an hour — still failed on every fresh
install. Happy to split it if you'd rather review them apart.
Measured
The remaining 56 KB is one line per transcript file (not per turn), so it scales with file count
and is a one-time cost well under the 256 KB cap. I left those: they are what tells you which file
was slow or failed. Worth noting they contain transcript paths under
$HOME— the fault reporterredacts paths, the log does not. Say the word if you want that changed too.
Verifying it yourself
The end-to-end measurement, which is what actually convinced me:
LUMEN_LOG=debugrestores every line — the same variable the GUI already takes its level from,rather than a second one to learn.
What I deliberately did not do
logline!by design, because the GUI owns that pipe; adding a level filter would be a largerchange than the defect warrants.
faults::record_throttled. The issue suggested it. It keys on(kind, variant)with a wall-clock interval and is right for the fault spool, but the decisionhere is attempt-count-based and needs to reset on a changed error — a different shape, and forcing
one to serve both would make each worse.
note_faultalready throttles the spool at 60sindependently, and that path is unchanged.
Local gate
Green before pushing:
cargo fmt --all --check, both clippy invocations with-D warnings,cargo test --workspace, and the frontend suite and build under Node 26.