Deleting a kafka consumer group in zookeeper

Question

I am using kafka_2.9.2-0.8.1.1 with zookeeper 3.4.6.

Is there a utility that can automatically delete a consumer group from zookeeper? Or can I just delete everything in / consumer / [group_id] in zookeeper? If the latter, is there anything else I am missing & can this be done using a live system?

Update: Starting from version 2.3.0, there is a new utility:

> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group

Related document: http://kafka.apache.org/documentation/#basic_ops_consumer_lag

See below for further discussion.

Answer 1

Currently, as far as I know, the only way to remove Kafka's consumer group is to manually delete the Zookeeper /consumers/[group_id] path.

If you just want to delete a consumer group, you don’t need to worry about deleting the Zookeeper path manually, but if you do this to rewind offsets, the following will help you.

First of all, you must stop all consumers belonging to the consumer group before deleting the Zookeeper path. If you do not, these consumers will not use the newly created messages and will soon close their connections to the Zookeeper cluster.

When restarting consumers, if you want consumers to start from the very beginning, auto.offset.reset property auto.offset.reset smallest (or earliest in new releases of Kafka). The default value for this property is largest (or latest in newer releases of Kafka), which causes restarted consumers to read after the largest offset, which in turn only consumes newly created messages. For more information about ownership, see Consumer Config in the Kafka documentation.

For your information, the question arises: How can I rewind the offset at the consumer? in the Kafka FAQ, but that didn't help me much.

Answer 2

Starting with v0.9.0, Kafka comes with a set of tools in /bin , one of which is the kafka-consumer-groups.sh tool. This will delete the user group. ./kafka-consumer-groups.sh --zookeeper <zookeeper_url> --delete --group <group-name>

Answer 3

For new consumers (who use the kafka theme to manage offsets instead of zookeeper), you cannot delete group information using the kafka built-in tools.

Here is an example attempt to delete group information for a new style user using the kafka-consumer-groups.sh script:

 bin/kafka-consumer-groups.sh --bootstrap-server "kafka:9092" --delete --group "indexer" --topic "cleaned-logs" Option '[delete]' is only valid with '[zookeeper]'. Note that there no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires. 

Here's the important part of this answer:

Note that there is no need to delete group metadata for a new user, as the group is deleted when the last committed offset for this group expires.

This is disgusting in terms of monitoring (especially when tracking offsets via something like burrow ), because it means that if you change the names of consumer groups in your code, you will see that the old groups lag behind their offsets before these offsets expire.

Hypothetically, you could write a headstone on this topic manually (what happens during the bias expiration ), but I did not find any tools that make this easy.

Answer 4

You can remove a group from kafa using the CLI

 kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group group_name 

Answer 5

I had a problem with a consumer group and I wanted to delete the group. The following approach worked for me:

• Set offsets.retention.minutes=1 in the Kafka server.config file and restart the Kafka server.
• Shutting down / exiting processes that use a consumer group
• Set the offsets of this topic / sections to 0 (something like kafka-consumer-groups.sh --bootstrap-server yourserver:9092 --group yourgroup --reset-offsets --to-latest --all-topics --execute

Remember to set the parameters again and restart Kafka.