Skip to content

[SVLS-9169] fix: lower log level to DEBUG for a trace failure#1240

Merged
lym953 merged 2 commits into
mainfrom
yiming.luo/fix-1232-handle-traces-body-read
May 29, 2026
Merged

[SVLS-9169] fix: lower log level to DEBUG for a trace failure#1240
lym953 merged 2 commits into
mainfrom
yiming.luo/fix-1232-handle-traces-body-read

Conversation

@lym953
Copy link
Copy Markdown
Contributor

@lym953 lym953 commented May 28, 2026

What

Log the handle_traces request-body-read failure at DEBUG instead of ERROR.

Fixes #1232.

Why

This failure happens probably because the execution environment is frozen when tracer sends traces to extension, causing timeout for TCP connections, which is not actionable on extension side. Lowering the log level to reduce noise.

Notes

If any customer sees other issues such as missing traces, let us know and we will treat it as a separate issue.

@lym953 lym953 changed the title fix(traces): downgrade handle_traces body-read failure from ERROR to WARN (#1232) fix(traces): log handle_traces body-read failure at DEBUG, not ERROR (#1232) May 28, 2026
…1232)

A body-read failure in handle_traces is almost always the tracer connection
closing mid-transfer when the Lambda sandbox freezes/terminates at the end of
an invocation -- an external lifecycle event, not an extension fault. Logging
it at ERROR misleads operators into thinking the extension is broken.

This is the same failure class fixed for handle_proxy in #1046 (downgraded to
WARN). It is a routine, per-invocation event under load, so it is too noisy
even at WARN -- DEBUG keeps it available for diagnosis without alarming
operators. Add a debug_response helper alongside the existing
error_response/warn_response trio and use it in handle_traces. The message
content and HTTP status are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lym953 lym953 force-pushed the yiming.luo/fix-1232-handle-traces-body-read branch from 0b88b72 to f7f5f8f Compare May 28, 2026 20:44
@lym953 lym953 changed the title fix(traces): log handle_traces body-read failure at DEBUG, not ERROR (#1232) [SVLS-9169] fix: lower log level to DEBUG for a trace failure May 28, 2026
@lym953 lym953 marked this pull request as ready for review May 28, 2026 20:51
@lym953 lym953 requested review from a team as code owners May 28, 2026 20:51
@lym953 lym953 requested review from Copilot and shreyamalpani May 28, 2026 20:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces log noise in the Lambda extension’s trace ingestion path by lowering the log level for a common, non-actionable handle_traces failure when the request body cannot be fully read (often due to the runtime/sandbox freezing and the client disconnecting mid-transfer).

Changes:

  • Switch handle_traces request-body-read failure logging from ERROR to DEBUG.
  • Introduce a debug_response helper analogous to existing error_response / warn_response helpers.

@datadog-prod-us1-3
Copy link
Copy Markdown

datadog-prod-us1-3 Bot commented May 28, 2026

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64, fips)   View in Datadog   GitLab

🔄 Retry job. This looks flaky and may succeed on retry. ThrottlingException: Rate exceeded during ListLayerVersions operation after maximum retries

DataDog/datadog-lambda-extension | e2e-test-status (amd64)   View in Datadog   GitLab

See error E2E tests failed. Command terminated with exit code 1.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fc4ab94 | Docs | Datadog PR Page | Give us feedback!

@lym953 lym953 merged commit b914019 into main May 29, 2026
53 of 55 checks passed
@lym953 lym953 deleted the yiming.luo/fix-1232-handle-traces-body-read branch May 29, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

handle_traces | Error extracting request body (v94)

3 participants