Kafka in a distributed system

Question

I am new to kafka, now I am running kafka on the same machine. I want to run kafka in a distributed environment on multiple machines. There is no relevant documentation for this. Any documentation or suggestion on this subject would be really helpful.

Answer 1

Adding to previous answer by user2720864

Suppose you need a Kafka system with a lower configuration.

7 knots Kafka

3 Zoo Keepers

In order to achieve this installation, 7 Kafka instances, in 7 different / vm servers (instances), and in each of these instances set a different broker identifier, this will allow zookeeper to identify the different kafka nodes for accounting and maintenance. broker.id = X (/config/server.properties)

To start zookeepers, you can use 3 of the previous kafka instances, or you can use new servers to start zookeepers. Once the servers running zookeepers are resolved, change /config/server.properties to specify zookeepers.

zookeeper.connect = hostname1: port1, hostname2: port2

In a distributed environment, it's nice to have 3 zoo keepers. While there is only one zookeeper that acts like a true master, the other 2 zookeepers act like a failure. When the master fails, one of the two ZK takes over as the leader.

I found this link very useful, it helped me clarify a lot about kafka architecture.

This is a good reference for all configurations in properties files in kafka.

Hope this helps!

Answer 2

Basically, you need to do the following:
1) Set up kafka on all machines
2) Configure the config/server1.properties property file to specify a unique id for each machine. This can be done by setting broker.id properties in the configuration file. e.g. broker.id=1 , broker.id=2 . For each broker, this identifier must be unique. This is how each node is identified in a kafka cluster.
3) Start kafka in all nodes

You can refer to Step 6: Configuring a cluster with several brokers from your official quick launch page.

Also here is a good article worth noting.