fix(telemetry): drop permanently-rejected outbox batches instead of retrying forever

[HugoRCD]

Problem

The CLI telemetry outbox buffers events on disk and resends the entire pending batch together on every run. If the ingest endpoint rejects that batch with a non-2xx response for any reason, the old drain treated it as "keep retrying" — with no way to ever recover if the rejection was permanent (oversized batch, schema drift on old buffered events, unknown tool, etc.). That poisoned batch would then be resent first on every future run, blocking all telemetry for the tool indefinitely.

This was caught live: a local evlog-cli outbox had accumulated 89 events since testing began (including some missing a field added later), which combined to trip both maxBatchSize and payload validation on the deployed ingest endpoint — every subsequent command kept re-sending the same broken batch and getting a 400.

Fix

createHttpDrain now distinguishes permanent rejection from transient failure:

• 2xx → delivered, drop from outbox (unchanged)
• 4xx except 429 → permanently rejected, drop from outbox (new — was previously kept forever)
• 429 / 5xx / network error / timeout → transient, keep for retry (unchanged)

Testing

• packages/telemetry/test/drain.test.ts (new) — covers 2xx/400/429/5xx/network-error/empty-batch
• pnpm turbo run lint typecheck test --filter=@evlog/telemetry --filter=evlog-telemetry — all green
• Changeset included (patch, @evlog/telemetry)

Summary by CodeRabbit

• Bug Fixes

  • Prevented permanently rejected telemetry batches from blocking future telemetry.
  • Batches receiving non-retryable 4xx responses are now discarded, while rate limits, server errors, and network failures remain eligible for retry.

• Tests

  • Added coverage for successful delivery, permanent rejections, transient failures, network errors, and empty batches.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evlog-docs	Error		Jul 23, 2026 8:29pm
evlog-telemetry	Building	Preview, Comment	Jul 23, 2026 8:29pm
just-use-evlog	Error		Jul 23, 2026 8:29pm

[github-actions]

Thank you for following the naming conventions! 🙏

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1f587d44-0d0d-41f3-bb1e-f6176bb6f96f

📥 Commits

Reviewing files that changed from the base of the PR and between add9283 and 6a5e285.

📒 Files selected for processing (3)

• .changeset/drop-poisoned-telemetry-batches.md
• packages/telemetry/src/drain.ts
• packages/telemetry/test/drain.test.ts

---

📝 Walkthrough

Walkthrough

Telemetry HTTP drains now remove successfully delivered and permanently rejected batches from the outbox, while retaining rate-limited, server-error, and network-failed batches for retry. Tests cover these outcomes and a patch changeset documents the release.

Changes

Telemetry drain retry behavior

Layer / File(s)	Summary
HTTP drain classification and coverage packages/telemetry/src/drain.ts, packages/telemetry/test/drain.test.ts, .changeset/drop-poisoned-telemetry-batches.md	The drain returns true for successful responses and non-429 4xx rejections, returns false for transient failures, skips fetches for empty batches, and documents the patch release.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• HugoRCD/evlog#417: Changes the same createHttpDrain HTTP retry classification behavior.

Suggested labels: feature

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/telemetry-outbox-poisoning

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install timed out. The project may have too many dependencies for the sandbox.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}