Geek Answers Handbook

How to get a kafka consumer to read from the last offset consumed, but not from the beginning

I am new to kafka and trying to figure out if there is a way to read messages from the last offset consumed, but not from the beginning.

I am writing an example so that my intention will not be diverted.

Eg: 1) I produced 5 messages at 7:00 PM and console consumer consumed those. 2) I stopped consumer at 7:10 PM 3) I produced 10 message at 7:20 PM. No consumer had read those messages. 4) Now, i have started console consumer at 7:30 PM, without from-beginning. 5) Now, it Will read the messages produced after it has started. Not the earlier ones, which were produced at 7.20 PM

Is there a way to get messages received from the last offset consumed.?

+8
source share
3 answers

I am new to kafka and trying to figure out if there is a way to read messages from the last offset consumed, but not from the beginning.

Yes, you can use the console consumer to read from the last offset consumed. You must add the consumer.config flag when calling kafka-console-consumer.

Example: -

 [ root@sandbox bin]# ./kafka-console-consumer.sh --topic test1 --zookeeper localhost:2181 --consumer.config /home/mrnakumar/consumer.properties

Here /home/mrnakumar/consumer.properties is the file containing group.id . This is what /home/mrnakumar/consumer.properties looks like: -

group.id = consoleGroup

Withoug, using user.config, can be read either from the beginning [using --from-begin] , or just to end the log. The end of the log means all messages published after the consumer started.

+6
source

You must set the auto.offset.reset parameter in your consumer’s settings to largest so that he reads all messages after the last recorded offset.

+3
source

Setting auto.offset.reset = the earliest, And a fixed group.id = something in the consumer configuration will start the consumer at the last fixed offset. In your case, it should start to consume at the first message at 7:20. If you want him to start reading messages sent AFTER the launch, then auto.offset.reset = latest will ignore the 10 messages sent at 7:20 and read everything that appears after it starts.

If you want it to start from the very beginning, you must either call seekToBeginning after the first consumer.poll (), or change the identifier of the consumer group to something unique.

0
source

More articles:


All Articles