Skip to content

outputs: add Kafka output handler#131

Open
abhihashi wants to merge 1 commit into
mainfrom
feature/grove_kafka_output
Open

outputs: add Kafka output handler#131
abhihashi wants to merge 1 commit into
mainfrom
feature/grove_kafka_output

Conversation

@abhihashi

Copy link
Copy Markdown

Adds a new kafka output handler that publishes collected Grove logs to Apache Kafka or Confluent Platform.

  • Optional dep: pip install grove[kafka]
  • serialize() returns plain NDJSON; Kafka producer handles compression (lz4)
  • submit() chunks by max_records_per_message=500 and max_bytes_per_message=750000
  • descriptor overrides the default topic for per-operation routing
  • use_identity_as_key sets Kafka message key to the connector identity
  • Delivery errors and flush timeouts surface as AccessException

Files changed:

  • grove/outputs/kafka.py: new output handler
  • tests/test_outputs_kafka.py: 15 unit tests
  • pyproject.toml: kafka entrypoint and confluent-kafka optional dependency
  • docs/configuration.rst: Kafka output configuration reference

Adds a new kafka output handler that publishes collected Grove logs to Apache Kafka or Confluent Platform.

- Optional dep: pip install grove[kafka]
- serialize() returns plain NDJSON; Kafka producer handles compression (lz4)
- submit() chunks by max_records_per_message=500 and max_bytes_per_message=750000
- descriptor overrides the default topic for per-operation routing
- use_identity_as_key sets Kafka message key to the connector identity
- Delivery errors and flush timeouts surface as AccessException

Files changed:
- grove/outputs/kafka.py: new output handler
- tests/test_outputs_kafka.py: 15 unit tests
- pyproject.toml: kafka entrypoint and confluent-kafka optional dependency
- docs/configuration.rst: Kafka output configuration reference
@hashicorp-cla-app

Copy link
Copy Markdown

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

Have you signed the CLA already but the status is still pending? Recheck it.

1 similar comment
@hashicorp-cla-app

Copy link
Copy Markdown

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

Have you signed the CLA already but the status is still pending? Recheck it.

@abhihashi abhihashi marked this pull request as ready for review April 15, 2026 18:11
@abhihashi abhihashi requested a review from a team as a code owner April 15, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant