How to subscribe to a list of several kafka pattern templates using kafka-python?

Question

I subscribe to Kafka using a template with a template as shown below. A wildcard is a dynamic client identifier.

consumer.subscribe(pattern='customer.*.validations') 

This works well because I can snatch the client id from the subject line. But now I need to expand the functionality to listen to a similar theme for a slightly different purpose. Let me call him customer.*.additional-validations . The code should live in the same project, because so many functions are divided, but I need to be able to use a different path based on the type of the queue.

In the Kafka documentation, I see that you can subscribe to many topics. However, these are hard-coded strings. Not templates that provide flexibility.

 >>> # Deserialize msgpack-encoded values >>> consumer = KafkaConsumer(value_deserializer=msgpack.loads) >>> consumer.subscribe(['msgpackfoo']) >>> for msg in consumer: ... assert isinstance(msg.value, dict) 

So I'm wondering, is it possible to somehow make a combination of the two? A view like this (non-working):

 consumer.subscribe(pattern=['customer.*.validations', 'customer.*.additional-validations']) 

Answer 1

In KafkaConsumer code, it maintains a list of themes or a template,

https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717

  def subscribe(self, topics=(), pattern=None, listener=None): """Subscribe to a list of topics, or a topic regex pattern Partitions will be dynamically assigned via a group coordinator. Topic subscriptions are not incremental: this list will replace the current assignment (if there is one). 

This way you can create a regular expression with an OR condition with | , which should work as a subscription to several dynamic regex partitions, since it internally uses the re module for matching.

(customer.*.validations)|(customer.*.additional-validations)