How can I get the LAST Kafka theme offset?

Question

I am writing a kafka consumer using Java. I want to keep the real time of the message, so if there are too many messages waiting to be consumed, for example, 1000 or more, I must reject the misunderstood messages and start consuming from the last offset.

For this problem, I am trying to compare the last recorded offset and the last topic offset (only 1 section), if the difference between these two offsets is greater than a certain amount, I will set the last offset as the next offset so that I can discard these redundant messages.

Now my problem is how to get the latest topic bias, some people say that I can use the old consumer, but it's too complicated, does the new user have this feature?

Answer 1

The new consumer is also getting complicated.

//assign the topic consumer.assign();

//seek to end of the topic consumer.seekToEnd();

//the position is the latest offset consumer.position();

Answer 2

For Kafka version: 0.10.1.1

 // Get the diff of current position and latest offset Set<TopicPartition> partitions = new HashSet<TopicPartition>(); TopicPartition actualTopicPartition = new TopicPartition(record.topic(), record.partition()); partitions.add(actualTopicPartition); Long actualEndOffset = this.consumer.endOffsets(partitions).get(actualTopicPartition); long actualPosition = consumer.position(actualTopicPartition); System.out.println(String.format("diff: %s (actualEndOffset:%s; actualPosition=%s)", actualEndOffset -actualPosition ,actualEndOffset, actualPosition)); 

Answer 3

 KafkaConsumer<String, String> consumer = ... consumer.subscribe(Collections.singletonList(topic)); TopicPartition topicPartition = new TopicPartition(topic, partition); consumer.poll(0); consumer.seekToEnd(Collections.singletonList(topicPartition)); long currentOffset = consumer.position(topicPartition) -1; 

Above the fragment returns the current corrected offset message for the given section and section number.