Why does a Kafka consumer take a long time to start consuming?

Question

We start the Kafka consumer by listening to a topic that cannot yet be created (although automatic theme creation is enabled).

Soon after, the producer publishes posts on this topic.

However, the consumer needs some time to notice this: 5 minutes, to be precise. At this point, the consumer cancels his sections and rejoins the consumer group. Kafka stabilizes the group again. If you look at the time stamps of the magazines "consumer versus kafka", then this process is initiated on the consumer side.

I assume this is the expected behavior, but I would like to understand this. Is this actually rebalancing (from 0 to 1 section)? If we created the themes in advance, would this not happen?

2017-02-01 08:36:45.692 INFO 7 --- [afka-consumer-1] oakccinternals.ConsumerCoordinator : Revoking previously assigned partitions [] for group tps-kafka-partitioning 2017-02-01 08:36:45.692 INFO 7 --- [afka-consumer-1] osklKafkaMessageListenerContainer : partitions revoked:[] 2017-02-01 08:36:45.693 INFO 7 --- [afka-consumer-1] oakccinternals.AbstractCoordinator : (Re-)joining group tps-kafka-partitioning 2017-02-01 08:36:45.738 INFO 7 --- [afka-consumer-1] oakccinternals.AbstractCoordinator : Successfully joined group tps-kafka-partitioning with generation 1 2017-02-01 08:36:45.747 INFO 7 --- [afka-consumer-1] oakccinternals.ConsumerCoordinator : Setting newly assigned partitions [] for group tps-kafka-partitioning 2017-02-01 08:36:45.749 INFO 7 --- [afka-consumer-1] osklKafkaMessageListenerContainer : partitions assigned:[] 2017-02-01 08:41:45.540 INFO 7 --- [afka-consumer-1] oakccinternals.ConsumerCoordinator : Revoking previously assigned partitions [] for group tps-kafka-partitioning 2017-02-01 08:41:45.544 INFO 7 --- [afka-consumer-1] osklKafkaMessageListenerContainer : partitions revoked:[] 2017-02-01 08:41:45.544 INFO 7 --- [afka-consumer-1] oakccinternals.AbstractCoordinator : (Re-)joining group tps-kafka-partitioning 

Kafka logs

 [2017-02-01 08:41:45,546] INFO [GroupCoordinator 1001]: Preparing to restabilize group tps-kafka-partitioning with old generation 1 (kafka.coordinator.GroupCoordinator) [2017-02-01 08:41:45,546] INFO [GroupCoordinator 1001]: Stabilized group tps-kafka-partitioning generation 2 (kafka.coordinator.GroupCoordinator) [2017-02-01 08:41:45,551] INFO [GroupCoordinator 1001]: Assignment received from leader for group tps-kafka-partitioning for generation 2 (kafka.coordinator.GroupCoordinator) [2017-02-01 08:42:14,636] INFO [GroupCoordinator 1001]: Preparing to restabilize group tps-kafka-group-id with old generation 1 (kafka.coordinator.GroupCoordinator) [2017-02-01 08:42:14,636] INFO [GroupCoordinator 1001]: Stabilized group tps-kafka-group-id generation 2 (kafka.coordinator.GroupCoordinator) 

Answer 1

This is likely due to the default metadata.max.age.ms parameter, which controls how often the user forces metadata to be updated for the topic.

What happens when a consumer starts with a non-existent topic is that brokers automatically create this topic, but it takes a little time with the leader’s choices, etc., so when your consumer requests metadata for this topic, he gets LEADER_NOT_AVAILABLE and cannot receive messages. After reaching the above timeout, the consumer updates the metadata, successfully this time and starts reading messages. It does not depend on the manufacturer who writes messages on this topic, this is a purely consumer thing.

If you start your consumer, for example, 1000 ms, you should see a much shorter delay until messages are used.

In addition, if you create topics ahead or run the producer in front of the consumer, this behavior should not occur at all.