Geek Answers Handbook

Kafka Java Consumer Group Does Not Consume Multiple Messages

One problem is that the Kafka consumer group (implemented in java) skips several messages from the broker sequentially. As the first line of debugging, through the kafka console consumer, I can see these messages available in the broker.

Kafka broker version: 0.10.1.0

Client version of Kafka:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.11</artifactId>
  <version>0.9.0.1</version>
</dependency>

Kafka consumer configuration:

Properties props = new Properties();
props.put("bootstrap.servers","broker1,broker2,broker3");
props.put("group.id", "myGroupIdForDemo");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
props.put("heartbeat.interval.ms", "25000"); 
props.put("session.timeout.ms", "30000"); 
props.put("max.poll.interval.ms", "300000");
props.put("max.poll.records", "1");
props.put("zookeeper.session.timeout.ms", "120000");
props.put("zookeeper.sync.time.ms", "10000");
props.put("auto.commit.enable", "false");
props.put("auto.commit.interval.ms", "60000");
props.put("auto.offset.reset", "earliest");
props.put("consumer.timeout.ms", "-1");
props.put("rebalance.max.retries", "20");
props.put("rebalance.backoff.ms", "6000");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

EDIT - Add More Information

Additional information I would like to add: A total of 6 sections. However, the total number of users is 40 for a topic that has the same consumer group identifier. I understand that 34 consumers sit idle and do nothing.

, , , , , , ? , , . , / .

. .

+6
2

a) Kafka - , , kafka.

b) , 1 2- , , 2- kafka: .

3) kafka, , , , , , , ? , .

4) "" , , .

5) ? , ?

6) , / , ?

7) , , - ? - -Xmx. ( )

+4

@thomas ? .

0

More articles:


All Articles