Is there a way to maintain ordering messages between sections of a kafka topic with a single consumer?

Question

We are developing a kafka-based flow system in which a manufacturer will produce several sections within his theme, and one consumer will consume from this theme. I know that kafka maintains the order of messages within sections, but can we maintain a global order of messages between sections within a topic?

Answer 1

Short answer: no, Kafka does not provide any guarantees of orders between sections.

Long answer: I do not quite understand your problem. If you say that you have only one consumer consuming your topic, why will you have more than one section in this topic and reinvent the wheel, trying to maintain order between sections? If you want to leave some space for future growth, for example. adding another consumer to consume part of the sections, you will have to rethink the idea of ​​a "global message order."

Do you really need ALL messages processed in order? Or maybe you can split the client / application / whatever and maintain order on the partition? In most cases, you really do not need this global message order, but you just need to properly break up your data.

Maintaining order between multiple consumers is really a difficult problem to solve, and even if it is solved correctly, you simply neglect all the benefits of Kafka.

Answer 2

You cannot benefit from kafka if you want a global order in more than one section. Kafka only supports message order in only one section. In our company, we only need the same messages that are sent to the same partition, which can easily be partitioned using partitionId.

Answer 3

The purpose of the sections in Kafka is to create a partial order of messages in a wider topic, where messages follow a strict general order in any given section. Thus, the answer is no, if we introduce any concept of the order of the partition, the purpose of the sections would be violated.

Instead, I would suggest focusing on how messages (records, in the Kafka language) are key, which effectively determines how they are displayed in the section. Which section does not matter if the mapping is deterministic and reproducible - all you need is that records with the same key will always appear in the same section and, therefore, will not be assigned to several consumers at the same time (in that same consumer group).

If you publish updates to persistent entities, an entity primary key is usually a good starting point for a Kafka record key. If a linked entity graph requires some sort of update order, then getting the root identifier of the graph and turning it into a key is likely to satisfy your ordering needs.