[otap-dataflow] updated kafka receiver metrics and logging

[c1ly]

Updated Kafka receiver to track the following metrics

• rebalances_total (counter)
• partitions_assigned (gauge)
• consumer_lag (gauge) — (offset delta between high watermark and committed offset per partition

Renamed:

• partitions_assigned (counter) -> partition_assignments
• partitions_revoked -> partitions_revocations

Added log events when:

• Partition assignment occurs, emits list of partition IDs assigned
• Partition revocation occurs emits list of partition IDs revoked
• Consumer joins or leaves consumer group

[opentelemetry-pr-dashboard]

Pull request dashboard status

Status last refreshed: 2026-07-26 17:02:58 UTC.

• Waiting on: Reviewers
• Next step: Review the latest changes.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected.

[Copilot]

Pull request overview

This PR enhances the Rust Kafka receiver’s consumer-group observability by expanding its metric set (rebalance counters, partition ownership gauge, and opt-in consumer lag gauge) and adding structured log events for rebalance/consumer-group lifecycle transitions.

Changes:

• Add new consumer-group metrics (rebalances_total, partitions_assigned gauge, partition_assignments, partition_revocations) and an opt-in consumer_lag gauge refreshed on a configurable interval.
• Emit structured otel_info! events for partition assignment/revocation and consumer-group join/leave transitions.
• Extend receiver configuration and docs with lag_refresh_interval_ms, plus tests and a changelog entry documenting the breaking metric semantic change.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
rust/otap-dataflow/crates/contrib-nodes/src/receivers/kafka_receiver/receiver.rs	Adds consumer-lag refresh logic/timer and folds new rebalance metrics, including partitions_assigned as a gauge.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/kafka_receiver/rebalance.rs	Tracks new rebalance metrics and emits structured logs for assignments/revocations and group join/leave events.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/kafka_receiver/metrics.rs	Introduces new counters/gauges and renames prior rebalance-related instruments.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/kafka_receiver/config.rs	Adds lag_refresh_interval_ms config plumbing, validation, and builder/getter support.
rust/otap-dataflow/crates/contrib-nodes/src/receivers/kafka_receiver/README.md	Documents the new config knob, metrics, and events.
rust/otap-dataflow/.chloggen/kafka-receiver-consumer-group-observability.yaml	Adds a release note entry describing new observability and breaking metric-contract changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 83.93285% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.90%. Comparing base (185447e) to head (fa227a6).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3535      +/-   ##
==========================================
+ Coverage   86.88%   86.90%   +0.01%     
==========================================
  Files         782      783       +1     
  Lines      319283   321634    +2351     
==========================================
+ Hits       277422   279525    +2103     
- Misses      41337    41585     +248     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	87.96% <83.93%> (+0.01%)	⬆️
query_engine	89.57% <ø> (ø)
otel-arrow-go	52.45% <ø> (ø)
quiver	92.20% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lquerel]

I believe there are few issues to fix before merging this PR. Thanks

[lquerel]

committable_snapshot() is the offset the local tracker would submit, not the consumer group offset acknowledged by Kafka. Normal commits use CommitMode::Async, so this snapshot advances before the commit callback succeeds and remains advanced when Kafka rejects the commit. For example, with high watermark 1000, local committable offset 1000, and broker-committed offset 100, this reports zero lag instead of 900. Please derive consumer_lag from broker-acknowledged committed offsets, either through a batched query or state updated by successful commit callbacks, or name and document this as local processing lag instead.