Geek Answers Handbook

Cluster Kafka and Akka

Below is my usage example

  • A bunch of applications puts messages in Kafka on different topics.
  • The consumer of each topic distributes the work to the employee in the cluster. Work can be classified as long-running, intense memory, simple, etc., and the worker is selected accordingly.

This allows me to explore the Akka cluster for work distribution, routing, and scaling. I can use Akka Supervisor as a Kafka consumer and assign inbound work to the appropriate employee based on their classification.

But what I'm still trying to understand is the right way to implement a sustainable way of communication between the leader and employees of the Akka cluster. Because as soon as the supervisor consumes a message from Kafka, the Kafka offset is committed. If some error occurs while processing the offset, is the next acceptable way to recover and start from where it was last?

Make your supervisor a regular actor using a reliable mailbox supported by Kafka. The supervisor completes the work in Kafka, and the employee receives the work from Kafka and performs its shift only after completion of the work.

+4
source share
1 answer

, , .

, Akka Streams Kafka, .

, (. ).

:

Consumer.committableSource Kafka. , , .

, Akka application.conf, akka.kafka.consumer:

akka.kafka.consumer {

  # Properties defined by org.apache.kafka.clients.consumer.ConsumerConfig
  # can be defined in this configuration section.

  kafka-clients {
    # Disable auto-commit by default
    enable.auto.commit = false
  }

}

akka-stream-kafka_2.11 ( 0.16) Akka 2.5.x, akka-stream_2.11 Akka. Akka 2.5.3, .

, , :)

+1

More articles:


All Articles