feat(ingest): HTTP OTLP backpressure parity (429 + Retry-After)#54
Merged
Conversation
The OTLP HTTP receiver now mirrors the gRPC RESOURCE_EXHAUSTED path
introduced in Phase 1: when the async ingest pipeline rejects a batch
because the queue is at capacity, the handler responds with HTTP 429
Too Many Requests, a Retry-After: 1 header, and an OTLP-shaped Status
protobuf body so spec-compliant clients back off.
`isQueueFull` recognizes both the local `ErrQueueFull` sentinel and any
gRPC RESOURCE_EXHAUSTED status returned by the underlying TraceServer /
LogsServer / MetricsServer Export methods, so a future change in error
plumbing inside ingest does not silently break the 429 mapping.
New metric `otelcontext_http_otlp_throttled_total{signal}` is wired in
main.go via SetThrottleCallback so operators see a unified backpressure
signal across both transports. Throttle counts are not incremented on
successful requests — verified by test.
Tests cover: 429 + Retry-After on traces and logs, callback invocation
with the right signal label, callback NOT firing on success, and
isQueueFull classification of both error shapes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
429 Too Many Requests+Retry-After: 1(mirroring the gRPCRESOURCE_EXHAUSTEDpath introduced in Phase 1) when the async ingest pipeline queue is at capacity. Body remains OTLP-shaped (Status protobuf), so spec-compliant clients back off.isQueueFullhelper recognizes bothErrQueueFulland gRPCcodes.ResourceExhaustedso the mapping survives future error-plumbing changes inside ingest.otelcontext_http_otlp_throttled_total{signal}wired viaSetThrottleCallbackfrom main.go gives operators a unified throttling signal across both transports.Why
Phase 4 of the 7-day-retention robustness initiative. Without this, healthy clients hitting the HTTP path during a queue spike would see a confusing 500 instead of the standard 429 + Retry-After flow that OTLP HTTP exporters know how to handle.
Test plan
go test ./... -race -count=1— full unit suite greenTestHTTPBackpressure_TracesReturns429WithRetryAfter— verifies status, header, and content typeTestHTTPBackpressure_LogsReturns429— same for logsTestHTTPBackpressure_ThrottleCallbackInvoked— verifies signal-labeled callback fires once per 429TestHTTPBackpressure_NotInvokedOnSuccess— verifies callback does NOT fire on 200TestIsQueueFull_ClassifiesCorrectly— verifies sentinel and gRPC status detectiongolangci-lint run --new-from-rev=origin/main— clean🤖 Generated with Claude Code