feat(kafka): add messaging.kafka.cluster.id to producer/consumer spans#4727
Open
shashank-reddy-nr wants to merge 9 commits into
Open
feat(kafka): add messaging.kafka.cluster.id to producer/consumer spans#4727shashank-reddy-nr wants to merge 9 commits into
shashank-reddy-nr wants to merge 9 commits into
Conversation
|
|
shashank-reddy-nr
force-pushed
the
feature/kafka-cluster-id
branch
from
June 22, 2026 13:05
937e539 to
3760932
Compare
shashank-reddy-nr
marked this pull request as draft
June 22, 2026 20:24
6 tasks
shashank-reddy-nr
changed the title
shashank-reddy-nr
changed the title
shashank-reddy-nr
marked this pull request as ready for review
July 10, 2026 14:27
This was referenced Jul 13, 2026
shashank-reddy-nr
changed the title
shashank-reddy-nr
force-pushed
the
feature/kafka-cluster-id
branch
from
July 13, 2026 18:27
784ac81 to
185af6e
Compare
shashank-reddy-nr
changed the title
Author
|
Hi @xrmx Can you please review this PR, whenever you have time? |
…afka-python, confluent-kafka, and aiokafka Add always-on messaging.kafka.cluster.id span attribute to all three Python Kafka instrumentation libraries. Cluster ID is read lazily from the client instance (list_topics() for kafka-python/confluent-kafka, metadata() for aiokafka) with a 1-hour TTL cache per client. Removes the capture_experimental_span_attributes gate and promotes the attribute to default-on behavior matching the semconv stable promotion. Assisted-by: Claude Sonnet 4.6
…ducer/consumer spans aiokafka's ClusterMetadata receives cluster_id in every MetadataResponse but does not persist it as an attribute. Wrap cluster.update_metadata before start() to cache the cluster_id; read it from _extract_cluster_id_from_client at span creation time. Also refresh the attribute after send() in case the first metadata response arrived mid-send. Add _wrap_start_producer / _wrap_start_consumer wrappers and register them on AIOKafkaProducer.start / AIOKafkaConsumer.start in _instrument / _uninstrument. Tested E2E against PLAINTEXT, SASL/PLAIN, and SASL/SCRAM-SHA-256 listeners; messaging.kafka.cluster.id appears in all producer and consumer spans. Assisted-by: Claude Sonnet 4.6
…ght/pylint Move _start_producer_wrapper and _start_consumer_wrapper from utils.py into __init__.py where they are used, eliminating the unnecessary factory pattern (no captured variables) and resolving: - pyright reportUnusedFunction: functions were flagged as unused because pyright checks within-file usage for module-level private functions - pylint W0108/R6301: remove unnecessary lambda, add @staticmethod to test_patch_cluster_id_capture_ignores_none_cluster Assisted-by: Claude Sonnet 4.6
…it__ to satisfy pyright pyright reportUnusedFunction flags any module-level private function that is not accessed within the same file. _patch_cluster_id_capture was in utils.py but only called from __init__.py, triggering the error. Moving it to __init__.py where it is defined and called keeps all cross-file import graphs clean without requiring type: ignore annotations (prohibited by AGENTS.md). Update test_utils.py to import _patch_cluster_id_capture from __init__ instead. Assisted-by: Claude Sonnet 4.6
Assisted-by: Claude Sonnet 4.6
…data, not a separate admin client Mirror the aiokafka instrumentation: read messaging.kafka.cluster.id from the client's own already-resolved metadata instead of opening a separate KafkaAdminClient in a background thread. Removes the hand-maintained security-config allowlist (which also omitted ssl_ciphers) and opens no extra broker connection. The id is captured from the MetadataResponse via update_metadata, so it works on kafka-python 2.0.x (which does not persist cluster_id on ClusterMetadata) as well as 2.1+. Assisted-by: Claude Opus 4.8
…ee kafka packages aiokafka/confluent-kafka/kafka-python are coordinated packages, so their towncrier fragment belongs in the root .changelog/ directory (per CONTRIBUTING.md), not a package-level one. Move the fragment to .changelog/4727.added as a single entry with comma-separated package prefixes (matching the existing 4613.fixed fragment for the same packages), covering all three packages the PR touches. Remove the misplaced package-level .changelog/ directory and its self-referential .gitignore. Assisted-by: Claude Opus 4.8
…many-locals Adding the extract_cluster_id mock parameter pushed wrap_send_helper to 16 locals (pylint limit 15). Inline the single-use expected_span_name local to stay within the limit; no behavioral change. Assisted-by: Claude Opus 4.8
…ER_ID + add semconv TODO The private constant was _MESSAGING_CLUSTER_ID, which dropped the 'kafka' segment. Rename it to _MESSAGING_KAFKA_CLUSTER_ID across the aiokafka, confluent-kafka and kafka-python instrumentations so it matches the attribute key (messaging.kafka.cluster.id) and the eventual generated semconv constant (messaging_attributes.MESSAGING_KAFKA_CLUSTER_ID). Add a TODO to switch to that constant once it is generated in opentelemetry-semantic-conventions (semconv spec PR open-telemetry#3819). Assisted-by: Claude Opus 4.8
shashank-reddy-nr
force-pushed
the
feature/kafka-cluster-id
branch
from
July 23, 2026 05:42
d53b17e to
eefa7a5
Compare
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4809
Description
Adds
messaging.kafka.cluster.id(semconv) to Kafka producer and consumer spans across theaiokafka,kafka-python, andconfluent-kafkainstrumentations.It is a
Recommendedsemantic-convention attribute, so it is emitted by default (no opt-in flag) — consistent with the other Recommended Kafka attributes these instrumentations already produce. The id is read from each client's own already-resolved metadata, so no extra broker connection is opened:ClusterMetadata(cluster.cluster_id), captured from the brokerMetadataResponseviaupdate_metadata. This works onkafka-python2.0.x (which does not persistcluster_idonClusterMetadata) as well as 2.1+.list_topics()(falling back to anAdminClient) in a background daemon thread and cached.The attribute may be briefly absent on the very first span before metadata is resolved; it self-heals on subsequent spans.
Type of change
How Has This Been Tested?
Unit tests added covering:
Does This PR Require a Core Repo Change?
Checklist: