feat(kafka): add messaging.kafka.cluster.id to producer and consumer spans

[shashank-reddy-nr]

Is your feature request related to a problem? Please describe.

When a service connects to multiple Kafka clusters, or when the same topic name exists across environments (dev/staging/prod), Kafka spans carry no information about which cluster a message came from. messaging.kafka.cluster.id is defined in the OTel messaging semantic conventions for Kafka but is not currently set by this instrumentation — making it impossible to distinguish spans by cluster from telemetry alone.

Describe the solution you'd like

Add messaging.kafka.cluster.id as a span attribute on both producer and consumer spans. The cluster ID is read from the client's own already-fetched Metadata object — no AdminClient, no extra broker connection, no additional security config. The attribute is opt-in (gated behind otel.instrumentation.kafka.experimental-span-attributes) and cached per client instance after the first successful read.

Implementation handles Kafka 3.7+ correctly: KafkaConsumer became a thin delegate wrapper in 3.7 (KIP-848) and no longer holds metadata directly — the fix walks KafkaConsumer.delegate → LegacyKafkaConsumer to find the metadata field.

Cross-language effort

This feature is being added consistently across all four OTel language instrumentations:

• Java (this repo): #18978 — reads cluster id from already-fetched Metadata; no extra broker connection
• JavaScript: open-telemetry/opentelemetry-js-contrib#3577 — fetches via admin().describeCluster() once per lifecycle
• Python: open-telemetry/opentelemetry-python-contrib#4727 — reads from aiokafka/kafka-python/confluent-kafka client metadata
• .NET: open-telemetry/opentelemetry-dotnet-instrumentation#5206 — one-time background AdminClient.DescribeClusterAsync() per bootstrap-servers string

Describe alternatives you've considered

• Extracting from consumer metrics() tags — Kafka does not expose the cluster ID as a standard metric tag, so this is not reliable across versions.
• Requiring users to configure the cluster ID manually — eliminates the zero-config goal.

Additional context

messaging.kafka.cluster.id is defined in the OTel semconv incubating attributes (KIP-78). The attribute is gated behind the existing otel.instrumentation.kafka.experimental-span-attributes flag. A working implementation with staging verification is ready.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[mznet]

@shashank-reddy-nr kafka cluster id was previously discussed in semconv and reached consensus, but it has not progressed further.

[shashank-reddy-nr]

Hey @mznet, I just raised a PR with the code changes. I'll fine-tune it and let you know once I've fixed the PR checks and Copilot comments. I'll also open a separate PR to update the Otel semantic convention spec for Kafka messaging.