feat(otlp/export): offline OTLP trace/metric/log export

[khanayan123]

What

Adds a public otlp/export package: offline clients that POST already-built OTLP collector-proto requests as raw protobuf to Datadog's agentless OTLP intake or a caller-provided collector/Agent endpoint.

c, _ := export.NewTraceClient(export.Config{Site: "datadoghq.com", APIKey: key})
res, err := c.ExportTraces(ctx, []*tracepb.ExportTraceServiceRequest{req})
// also NewMetricClient/ExportMetrics and NewLogClient/ExportLogs

Stacked on #4936 (llmobs/export) — Stack 2/2, base #4936. Base branch is ayan.khan/llmobs-explicit-span-ids, so this diff shows only the OTLP additions plus the shared internal/exportutil package. This branch already contains all of #4936's commits — review/merge #4936 first.

Freshly rebased onto the current #4936 head (b4370d79d). As part of the rebase, internal/llmobs/exportutil is promoted back to a shared internal/exportutil (its original "shared by both offline export clients" intent): otlp/export needs the same bounded Snippet/Aggregate helpers as llmobs/export without importing internal/llmobs. The generic Retry engine is added there and llmobs/export's import is updated accordingly; the live llmobs path is unaffected. (#4936 scoped exportutil under internal/llmobs when llmobs/export was its only consumer — per @rarguelloF's review — and this second consumer promotes it to shared.)

The OTLP half of the RFC "Raw LLM Observability Export, Multi-Destination Routing, and Caller-Assigned IDs in dd-trace-go."

Design

• Accepts go.opentelemetry.io/proto/otlp/collector/{trace,metrics,logs}/v1 Export*ServiceRequest values (no new deps — proto/otlp + protobuf are already direct deps).
• Atomic per request: one *Export*ServiceRequest → one POST → one indexed RequestResult row (in ExportResult.Requests). The SDK never merges or splits requests. Exported sequentially; a Client is concurrency-safe and stateless, so throughput is caller-parallelized.
• Routes: Datadog (Site+APIKey → https://otlp.<site>/v1/<signal> with dd-api-key) or collector/Agent (Endpoint, no auth unless APIKey is also set). Metrics on the Datadog route add dd-otel-metric-config (metrics-only, and only when Endpoint is empty).
• Strict success contract: only HTTP 200 with a decodable OTLP Export*ServiceResponse is success. Any other 2xx (202/204/206) or a not-followed redirect is a failed export, and a 200 that reports rejected_* > 0 (partial success) is surfaced as a per-request error. A 200 whose body reads but does not proto-decode is a permanent per-request failure, whereas a 200 whose body cannot be read at all is surfaced as a retryable transport-class failure (status 0) so dropped records are not hidden behind a false success. Unknown protobuf fields are ignored (forward-compatible with newer collectors).
• No redirects: the default client — and any caller-provided client without its own policy — refuses redirects, so the POST body is never dropped and dd-api-key is never forwarded to a redirect target.
• OTLP/HTTP-spec retry (default 3 attempts): only 429/502/503/504 (+ network errors, status 0) retry; other 4xx/5xx are permanent; caller-cancelled context is not retriable. Honors Retry-After (delta-seconds parsed as int64, or HTTP-date; clamped to 60s).
• Configurable per-request timeout via Config.RequestTimeout; the 10s default applies only when the caller's context has no deadline, so a longer caller deadline (large export / slow collector) is not silently shortened.
• Readable errors: failed responses decode the google.rpc.Status message into RequestResult.ResponseSnippet (bounded/UTF-8-safe) instead of raw protobuf bytes.
• Caller trace IDs/flags/tracestate are preserved as-is. A raw-proto transport that coexists with the OTel-SDK-based exporters in ddtrace/opentelemetry; retry mechanics live in the shared internal/exportutil.Retry (pluggable classifier).

Performance

BenchmarkExportTraces (allocations/op + bytes/op); see the package Performance doc for the sequential/caller-parallelized model and per-request memory profile.

Ownership

CODEOWNERS: /otlp/export → @DataDog/trajectory @DataDog/ml-observability @DataDog/apm-go.

Tests

go test -race ./otlp/export/... ./internal/exportutil/... — endpoint derivation (Datadog vs collector, trailing-slash trim, schemeless/non-HTTP rejection, APIKey-or-Endpoint requirement), header injection, endpoint-override-with-APIKey (no metric config), protobuf round-trip, nil-request handling, per-request result rows, OTLP retry classification with exact attempt counts, context-cancel non-retriable, partial-success for all three signals, decoded-status snippet, strict-200 / non-200 failure, undecodable-body failure, read-error-on-2xx surfaced as a retryable failure, forward-compatible-response success, no-redirect (default + custom client), Retry-After parsing incl. overflow clamp, configurable/deadline-respecting per-request timeout, and the Retry engine. golangci-lint clean.

Review status

Squashed to a single commit; Codex-clean on the current HEAD. For #4936, the only outstanding note is a doc-placement item, intentionally handled in the package doc.go per maintainer review (not CONTRIBUTING.md). For #4994, no outstanding findings.

Contract testing: OTLP intake-shape has a cross-language guard in system-tests via Test_Otel_Tracing_OTLP (tracer OTLP path; Go passes). DataDog/system-tests#7314 (draft) additionally asserts the OTel gen_ai LLM-Obs-over-OTLP wire shape — the same schema an otlp/export caller emits for LLM Obs — but hermetically and via the tracer's OTLP producer, so it does not import or exercise this package. This package's own emission is covered by its unit tests (proto round-trip); the only thing not covered is validation against the real OTLP intake, which would reuse the existing otel_tracing_e2e real-backend pattern — a separate E2E effort, not a quick add.

Scope note: metrics coverage here is OTLP-protobuf only. Trajectory's native dd_metrics_v2 lane (JSON POST /api/v2/series + /api/v1/distribution_points) is intentionally out of scope — it has no path in dd-trace-go and is a deprecation-window fallback that OTLP replaces.

RFC Open Question #1 — resolved

Whether to reuse/generalize ddtrace/tracer's OTLP writer vs. a separate transport: this PR ships a deliberately separate raw-proto transport, and the owning teams are now recorded in CODEOWNERS (@DataDog/trajectory @DataDog/ml-observability @DataDog/apm-go) so the two OTLP paths have a clear owner keeping them coherent.

🤖 Generated with Claude Code

[datadog-official]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 85.09%
• Overall Coverage: 63.22% (+0.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5e29095 | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-07-24 18:50:27

Comparing candidate commit 5e29095 in PR branch ayan.khan/otlp-export with baseline commit b4370d7 in branch ayan.khan/llmobs-explicit-span-ids.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 326 metrics, 0 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

• scenario:BenchmarkOTLPTraceWriterFlush

[kakkoyun]

For RFC Open Question #1: ddtrace/tracer already has an OTLP writer and transport (otlp_writer.go, span_to_otlp.go). A separate transport here is a reasonable ownership choice; recording the owner in CODEOWNERS and closing OQ#1 with that decision would keep the repo from carrying parallel OTLP transports without a clear owner keeping them coherent.

[kakkoyun]

exportEach is sequential (one POST at a time) and holds each full request plus its marshaled bytes in memory, so a slow destination serializes the whole batch. Fine for v1 since Client is concurrency safe and the caller can fan out, but documenting that throughput is caller-parallelized, and offering a bounded-concurrency option if Trajectory needs it, would help. A benchmark for ExportTraces (allocs/op + bytes) fits the package-benchmark ask on #4936.

[linear-code]

TRA-148