Is your feature request related to a problem?
There is no dd-trace-go integration for goka, the Kafka-streams
library. goka builds its consumer group on top of IBM/sarama, but the existing
contrib/IBM/sarama integration cannot instrument a goka processor: goka's *Processor
is the sarama.ConsumerGroupHandler and passes itself directly to
consumerGroup.Consume(ctx, topics, g) (goka processor.go). Because there is no
user-supplied handler to wrap, the sarama contrib's handler-wrapping never attaches, and
consumed messages get no spans and no DSM checkpoints. Today users must hand-roll
instrumentation (we maintain two separate local patches — one for DSM, one for APM
propagation — that hook the same seam and duplicate a goka-header carrier).
Describe the solution you'd like
A new contrib/lovoo/goka integration (module
github.com/DataDog/dd-trace-go/contrib/lovoo/goka/v2, package goka) that provides both
APM distributed tracing and Data Streams Monitoring, built entirely on goka's existing public
extension points — no changes to goka are required:
- Consume side:
goka.WithContextWrapper (per-input-message hook) to extract the
propagated trace/DSM context from Context.Headers(), set the inbound DSM checkpoint
(direction:in) and TrackKafkaCommitOffset, and start a kafka.consume span.
- Produce side: a wrapping
goka.Context whose Emit/Loopback inject the active span
and a direction:out DSM checkpoint into outbound headers via goka.WithCtxEmitHeaders.
- Helpers for the standalone
goka.Emitter path and for extracting a span context, mirroring
the twmb/franz-go and IBM/sarama contribs.
Proposed surface:
tr := gokatrace.NewTracer(gokatrace.WithService("orders"), gokatrace.WithDataStreams())
p, _ := goka.NewProcessor(brokers, goka.DefineGroup(group,
goka.Input(topic, codec, tr.WrapCallback(handle)),
), goka.WithContextWrapper(tr.WrapContext))
The integration follows the established Kafka-family conventions: instrumentation.Load with
a new PackageLovooGoka, WithService/WithDataStreams options, the base64 DSM carrier,
kafka.consume/kafka.produce (v0) span names, the standard messaging tags
(span.kind=consumer, component=lovoo/goka, messaging.system=kafka,
messaging.destination.name, partition/offset), mocktracer + pathway-hash tests, and
replace => ../../.. in a dedicated submodule.
Notes
- Why a contrib and not a goka patch: goka carries no Datadog/OpenTelemetry dependency and its
maintainers keep it telemetry-free; the required seams are already public and stable
(WithContextWrapper since goka v1.1.8, WithCtxEmitHeaders since v1.0.6). Pinning target:
goka v1.1.8 (minimum version exposing WithContextWrapper).
- No
orchestrion.yml: unlike sarama/franz-go there is no single constructor call to rewrite
(a processor is assembled from a user-defined graph plus options), so the integration is an
explicit one-line opt-in rather than auto-instrumented.
Describe alternatives you've considered
- Patching goka to emit spans directly — rejected: adds a Datadog dependency the maintainers
won't accept, and couples goka's release cadence to dd-trace-go.
- Injecting an instrumented
sarama.Config/handler through goka's builder options
(WithConsumerGroupBuilder, etc.) — does not help on the consume side because goka owns the
handler; WithContextWrapper is the sanctioned hook.
- sarama KIP-42 interceptors (
Consumer.Interceptors / Producer.Interceptors) injected via
goka's config builders — this is the seam OpenTelemetry-style instrumentation uses, and a
ConsumerInterceptor does fire for goka (its consumer group routes through the
partitionConsumer responseFeeder, which applies interceptors before claim.Messages()).
Rejected as the primary seam because interceptors are strictly weaker for a stream processor:
(1) OnConsume fires before the callback with no "processing done" counterpart, so it cannot
scope a span around the goka callback or tag errors from ctx.Fail; (2) an interceptor sees
a raw *sarama.ConsumerMessage, not goka's Context, so it cannot propagate the consume
span into goka Emit/Loopback. It could still complement WithContextWrapper for
lower-level produce/consume spans if desired. Note dd-trace-go's IBM/sarama contrib
currently ships no interceptor implementation.
I have a working implementation with unit and integration tests and am happy to open the PR
once the approach is agreed.
Is your feature request related to a problem?
There is no dd-trace-go integration for goka, the Kafka-streams
library. goka builds its consumer group on top of
IBM/sarama, but the existingcontrib/IBM/saramaintegration cannot instrument a goka processor: goka's*Processoris the
sarama.ConsumerGroupHandlerand passes itself directly toconsumerGroup.Consume(ctx, topics, g)(gokaprocessor.go). Because there is nouser-supplied handler to wrap, the sarama contrib's handler-wrapping never attaches, and
consumed messages get no spans and no DSM checkpoints. Today users must hand-roll
instrumentation (we maintain two separate local patches — one for DSM, one for APM
propagation — that hook the same seam and duplicate a goka-header carrier).
Describe the solution you'd like
A new
contrib/lovoo/gokaintegration (modulegithub.com/DataDog/dd-trace-go/contrib/lovoo/goka/v2, packagegoka) that provides bothAPM distributed tracing and Data Streams Monitoring, built entirely on goka's existing public
extension points — no changes to goka are required:
goka.WithContextWrapper(per-input-message hook) to extract thepropagated trace/DSM context from
Context.Headers(), set the inbound DSM checkpoint(
direction:in) andTrackKafkaCommitOffset, and start akafka.consumespan.goka.ContextwhoseEmit/Loopbackinject the active spanand a
direction:outDSM checkpoint into outbound headers viagoka.WithCtxEmitHeaders.goka.Emitterpath and for extracting a span context, mirroringthe
twmb/franz-goandIBM/saramacontribs.Proposed surface:
The integration follows the established Kafka-family conventions:
instrumentation.Loadwitha new
PackageLovooGoka,WithService/WithDataStreamsoptions, the base64 DSM carrier,kafka.consume/kafka.produce(v0) span names, the standard messaging tags(
span.kind=consumer,component=lovoo/goka,messaging.system=kafka,messaging.destination.name, partition/offset),mocktracer+ pathway-hash tests, andreplace => ../../..in a dedicated submodule.Notes
maintainers keep it telemetry-free; the required seams are already public and stable
(
WithContextWrappersince goka v1.1.8,WithCtxEmitHeaderssince v1.0.6). Pinning target:goka
v1.1.8(minimum version exposingWithContextWrapper).orchestrion.yml: unlike sarama/franz-go there is no single constructor call to rewrite(a processor is assembled from a user-defined graph plus options), so the integration is an
explicit one-line opt-in rather than auto-instrumented.
Describe alternatives you've considered
won't accept, and couples goka's release cadence to dd-trace-go.
sarama.Config/handler through goka's builder options(
WithConsumerGroupBuilder, etc.) — does not help on the consume side because goka owns thehandler;
WithContextWrapperis the sanctioned hook.Consumer.Interceptors/Producer.Interceptors) injected viagoka's config builders — this is the seam OpenTelemetry-style instrumentation uses, and a
ConsumerInterceptordoes fire for goka (its consumer group routes through thepartitionConsumer
responseFeeder, which applies interceptors beforeclaim.Messages()).Rejected as the primary seam because interceptors are strictly weaker for a stream processor:
(1)
OnConsumefires before the callback with no "processing done" counterpart, so it cannotscope a span around the goka callback or tag errors from
ctx.Fail; (2) an interceptor seesa raw
*sarama.ConsumerMessage, not goka'sContext, so it cannot propagate the consumespan into goka
Emit/Loopback. It could still complementWithContextWrapperforlower-level produce/consume spans if desired. Note dd-trace-go's
IBM/saramacontribcurrently ships no interceptor implementation.
I have a working implementation with unit and integration tests and am happy to open the PR
once the approach is agreed.