Skip to content

Make stdio logging fail-soft on write errors#231

Merged
TheJokr merged 1 commit into
cloudflare:mainfrom
dineshadhi:dineshadhi/logging-fail-soft
Jul 15, 2026
Merged

Make stdio logging fail-soft on write errors#231
TheJokr merged 1 commit into
cloudflare:mainfrom
dineshadhi:dineshadhi/logging-fail-soft

Conversation

@dineshadhi

Copy link
Copy Markdown
Contributor

Foundations currently wraps terminal/stderr logging drains with Fuse before passing them to slog_async. If the underlying stdout/stderr stream returns an I/O error such as BrokenPipe, Fuse turns that error into a panic in the async logging worker.

This is unsafe for services whose stdout/stderr is captured by systemd and journald: if the journald stream is interrupted, normal logging can crash the service.

Handle terminal and stderr output errors with ignore_res() instead, so stdio logging failures do not panic the async logging worker. File logging keeps the existing fused behavior.

Add a regression test proving a failing async stdio drain keeps processing subsequent records instead of terminating after the first write error.

@TheJokr TheJokr self-requested a review July 14, 2026 13:10
@TheJokr

TheJokr commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

The use of fuse() for our loggers is intentional - keeping a service running without log output is bad for observability. That being said, I understand there is a use case for this. I'm happy to merge a PR that makes this configurable:

  • Add a field ignore_io_errors: bool to LoggingSettings
  • Wrap the AsyncDrain construction in a new function that takes drain: impl Drain<...> and settings: &LoggingSettings
  • Apply fuse() or ignore_res() depending on the setting there
  • Use that function for the output types that are currently fused (terminal/stderr/file)

Logging drains for built-in terminal, stderr, and file outputs are fused by
default, preserving the existing behavior where output I/O errors are treated
as fatal logging failures.

Add `LoggingSettings::ignore_io_errors` to allow services to opt into
fail-soft handling for those built-in outputs. When enabled, the async logging
drain is configured with `ignore_res()` instead of `fuse()`.

Add regression coverage for both modes: the default fused behavior stops after
an output error, while `ignore_io_errors = true` keeps the async logging worker
alive for subsequent records.
@dineshadhi dineshadhi force-pushed the dineshadhi/logging-fail-soft branch from c8ccdb0 to 6eee7a1 Compare July 14, 2026 19:18
@TheJokr TheJokr merged commit 31f49d0 into cloudflare:main Jul 15, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants