Time-out

Question

We have an application in which the consumer reads the message, and the thread performs a number of actions, including access to the databases, before the message is released in another topic. The time between consuming and receiving a message in a stream can take several minutes. When a message is created in a new topic, a commit is performed to indicate that we have finished work on the message about the user's queue. For this reason, auto-lock is disabled.

I use a high-level consumer and I notice that the zookeeper and kafka session timeout because it is too long before we do anything in the consumer queue, so kafka finishes rebalancing every time the stream returns to read more from consumers ’queues, and it begins to take a long time before the consumer reads a new message after a while.

I can set the zookeeper session timeout very high so as not to make this problem, but then I need to adjust the rebalance parameters accordingly and kafka will not gain a new consumer for some time among other side effects.

What are my solutions to this problem? Is there a way to beat my heart with kafka and the zoo to be happy? Do I still have the same problems if I have to use a simple user?

Answer 1

It seems like your problems come down to relying on a high-level consumer to control the bias of the last read. Using a simple consumer will solve this problem, since you control the persistence of this bias. Note that all high consumption transactions are saving the last read offset in zookeeper. No other action was taken, and the message you just read is still in the section and is being read by other users.

With a simple kafka consumer, you have much more control over when and how this offset storage happens. You can even save this offset somewhere other than Zookeeper (like a database).

The bad news is that while a simple consumer in itself is simpler than a high-level consumer, much more work needs to be done with a code key to make it work. You will also have to write code to access several sections - something that a high-level consumer does for you is quite enjoyable.

Answer 2

I think the problem is the consumer polling method that initiates the user request. And when you increase session.timeout. Consumer heartbeat will not reach the coordinator. Due to this pulse, the missed coordinator marks the consumer dead. And also the return of consumers is very slow, especially in the case of a single consumer.

I ran into a similar problem and decided that I need to change the following parameter in the consumer configuration settings

session.timeout.ms = request.timeout.ms = more than session timeout

Also you need to add the following property to server.properties in kafka broker node. group.max.session.timeout.ms =

You can see the following link for more details. http://grokbase.com/t/kafka/users/16324waa50/session-timeout-ms-limit