Adding custom headers in a Kafka post

Question

I am sending the file as a message, converting it to an array of bytes using the kafka manufacturer.

I also need to add some headers for the message, such as file name, timestamps, etc., so in the end I can process the message based on the file name and other headers.

I am currently creating an object and wrapping raw messages and headers in it and sending the object to the byte array as a message.

I would like to know if there is a way by which I can add custom headers when posting a post?

Answer 1

Kafka does not depend on the content of the message and does not provide any special means for enriching it, so you need to do something yourself. A common way to deal with these things is to use a structured format such as json, avro or similar, where you can define the necessary fields and you can easily add metadata to your message and send it to Kafka brokers.

Answer 2

Kafka v0.11.0.0 adds support for custom headers.

You can add them when creating ProducerRecord as follows:

new ProducerRecord (key, value, headers, ...), where the headers are of type Iterable <Header>

For more details see:

https://issues.apache.org/jira/browse/KAFKA-4208

https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers

Answer 3

I had problems with the projects I was working on, so I created this simple library to help solve this problem: https://github.com/leandronunes85/messaging.Avro-based implementation currently exists, but it can be extended to use any other serialization structure of your choice.

You just need to create a serializer (de) for the objects you want to use in the stream (based on Avro or not), and let AvroMessageSerializer work on its magic.

It is still a very young library, but I feel that it can save many people a lot of time!