Use Case
In my use case, I would like to transmit verbose/complex header information for communication with Kafka. This includes metadata such as a unique ID, the source of the message (e.g., the service which created/processed the data), the physical origin, a timestamp and more key/value-pairs.
It would allow for clearer identification and processing of the data in downstream processes. The ability to transmit such complex headers is crucial for enhancing traceability in my system mostly for debugging purposes.
This functionality align with future efforts towards adopting standardisation like CloudEvents, which aims to describe event data in a common way (see: https://cloudevents.io/). Providing consistency, accessibility, and portability is becoming increasingly important.
Expected behavior
It should be possible to transmit header information as a complex data structure (e.g., JSON or array). This would enable flexible and application-specific transfer of metadata that can be used in Kafka messages.
The header could be structured in JSON format as follows:
{
"id": 12345,
"source": "Modbus TCP",
"origin": "SensorXYZ",
"time": "${metric_timestamp}",
"key_5": "value_5",
}
But also different structures with similar functionalities would be fine.
Actual behavior
Currently, it is only possible to add a single header information:
Add metric name as specified Kafka header if not empty
metric_name_header = ""
A complex header structure (e.g., JSON or array) cannot be used. This significantly limits the flexibility of the header information, and I cannot fully transmit the required metadata.
Additional info
If a workaround exists I cannot see to enable more complex header information, such as using another format or configuration, please provide guidance.
Use Case
In my use case, I would like to transmit verbose/complex header information for communication with Kafka. This includes metadata such as a unique ID, the source of the message (e.g., the service which created/processed the data), the physical origin, a timestamp and more key/value-pairs.
It would allow for clearer identification and processing of the data in downstream processes. The ability to transmit such complex headers is crucial for enhancing traceability in my system mostly for debugging purposes.
This functionality align with future efforts towards adopting standardisation like CloudEvents, which aims to describe event data in a common way (see: https://cloudevents.io/). Providing consistency, accessibility, and portability is becoming increasingly important.
Expected behavior
It should be possible to transmit header information as a complex data structure (e.g., JSON or array). This would enable flexible and application-specific transfer of metadata that can be used in Kafka messages.
The header could be structured in JSON format as follows:
But also different structures with similar functionalities would be fine.
Actual behavior
Currently, it is only possible to add a single header information:
A complex header structure (e.g., JSON or array) cannot be used. This significantly limits the flexibility of the header information, and I cannot fully transmit the required metadata.
Additional info
If a workaround exists I cannot see to enable more complex header information, such as using another format or configuration, please provide guidance.