Kafka producer in a multi-broker, multi-server cluster cannot write to a newly created topic

Question

I have a Kafka cluster with 2 brokers, each on its own server (AWS) (I created the cluster using the instructions here ). I use SASL (but not encryption). On server 2, on which broker 2 is running, I created a theme:

KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
    bin/kafka-topics.sh --create \
    --zookeeper zookeeper-server-01.eigenroute.com:2181,zookeeper-server-02.eigenroute.com:2181,zookeeper-server-03.eigenroute.com:2181/apps/kafka-cluster-demo \
    --replication-factor 2   --partitions 9   --topic another-test-topic

it would seem, with success, because the description of the topics shows that it was at least created:

KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
    bin/kafka-topics.sh --describe \
    --zookeeper zookeeper-server-01.eigenroute.com:2181,zookeeper-server-02.eigenroute.com:2181,zookeeper-server-03.eigenroute.com:2181/apps/kafka-cluster-demo
Topic:another-test-topic    PartitionCount:9    ReplicationFactor:2 Configs:    MarkedForDeletion:true
    Topic: another-test-topic   Partition: 0    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 1    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 2    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 3    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 4    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 5    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 6    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 7    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 8    Leader: none    Replicas: 2,1   Isr:

As you can see, this topic is not assigned a leader and has no built-in replicas. I assigned write permissions to the manufacturer:

KAFKA_HEAP_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Xmx256M -Xms128M" \
    bin/kafka-acls.sh --authorizer-properties \
    zookeeper.connect=zookeeper-server-01.eigenroute.com:2181,zookeeper-server-02.eigenroute.com:2181,zookeeper-server-03.eigenroute.com:2181/apps/kafka-cluster-demo \
   --add --allow-principal User:producer1 --producer --topic another-test-topic
...
Current ACLs for resource `Topic:another-test-topic`:
    User:producer1 has Allow permission for operations: Describe from hosts: *
    User:producer1 has Allow permission for operations: Write from hosts: *

My producer, however, cannot write about this topic:

KAFA_HEAP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.krb5.debug=true" \
    bin/kafka-console-producer.sh \
    --broker-list server-01.eigenroute.com:9092,server-02.eigenroute.com:9092 \
    --topic another-test-topic --producer.config config/sasl-producer.properties
>this is a test message
[2018-01-07 21:16:02,650] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {another-test-topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

The ACL in the ZooKeeper node for this section:

[zk: zookeeper-server-03.eigenroute.com:2181(CONNECTED) 8] getAcl /apps/kafka-cluster-demo/brokers/topics/another-test-topic
'world,'anyone
: r
'sasl,'kafka/[e-mail]
: cdrwa

... kafka/[e-mail] ( Kerberos 1) , kafka/[e-mail] ( Kerberos 2)?

- , , ?

: @Vladimir Nabokov:

• node:

  kafka @server-02:/var/log/kafka $ls -alhtr 124K -rw-r-r-- 1 kafka kafka 0 . 7 23:27.lock -rw-r-r-- 1 kafka kafka 0 . 7 23:27 clean-offset-checkpoint -rw-r - r-- 1 kafka kafka 54 7 23:27 meta.properties drwxr-xr-x 7 root root 4.0K 9 06:25.. drwxr-xr-x 2 kafka kafka 4.0K 18 05:30 __consumer_offsets-29 ... drwxr-xr-x 2 kafka kafka 4.0K 18 05:30 __consumer_offsets-1 -rw-r-r-- 1 kafka kafka 600 18 05:56 -- -rw-r - r-- 1 kafka kafka 600 18 . 05:56 -- -rw-r - r-- 1 kafka kafka 4 . 18 05:56 log-start-offset-checkpoint drwxr-xr-x 27 kafka kafka 4.0K 18 05:56.

kafka@server-01:/var/log/kafka$ ls -alhtr
total 124K
-rw-r--r--  1 kafka kafka    0 Jan  7 23:26 .lock
-rw-r--r--  1 kafka kafka    0 Jan  7 23:26 cleaner-offset-checkpoint
-rw-r--r--  1 kafka kafka   54 Jan  7 23:26 meta.properties
drwxr-xr-x  7 root  root  4.0K Jan 17 06:25 ..
drwxr-xr-x  2 kafka kafka 4.0K Jan 18 05:30 __consumer_offsets-0
...
drwxr-xr-x  2 kafka kafka 4.0K Jan 18 05:30 __consumer_offsets-32
-rw-r--r--  1 kafka kafka  600 Jan 18 05:58 recovery-point-offset-checkpoint
-rw-r--r--  1 kafka kafka    4 Jan 18 05:58 log-start-offset-checkpoint
-rw-r--r--  1 kafka kafka  600 Jan 18 05:59 replication-offset-checkpoint
drwxr-xr-x 27 kafka kafka 4.0K Jan 18 05:59 .

2. kafka, , kafka, /var/log/kafka/ :

   kafka @server-02: ~/kafka_2.11-1.0.0/confkafka@server-01:/var/log $ll/var/log | grep kafka drwxr-xr-x 27 kafka kafka 4096 18 05:49 kafka ig $ll/var/log | grep kafka

3. , telnet , :

   sjamal- > telnet server-01.eigenroute.com 9092 54.175.56.39... server-01.eigenroute.com. Escape "^]". ^ CConnection . [~//microservices/-/kafka_2.11-1.0.0] sjamal- > telnet server-02.eigenroute.com 9092 18.221.32.34... server-02.eigenroute.com. Escape "^]". ^ CConnection .

4. - :

   kafka @server-02: ~/kafka_2.11-1.0.0/config $host server-01.eigenroute.com server-01.eigenroute.com 54.175.56.39 kafka @server-01:/var/log $host server-02.eigenroute.com server-02.eigenroute.com 18.221.32.34

5. . :

   [2018-01-18 00: 45: 31,931] WARN [ Id = -1, groupId = console-consumer-95024] 7022: {another-test-topic = UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2018-01-18 00: 45: 32,063] WARN [Consumer clientId = -1, groupId = console-consumer-95024] 7024: {another-test-topic = UNKNOWN_TOPIC_OR_PARTITION} (org. apache.kafka.clients.NetworkClient) [2018-01-18 00: 45: 32,194] WARN [Consumer clientId = -1, groupId = console-consumer-95024] 7025: {another-test-topic = UNKNOWN_TOPIC_OR_PARTITION} (. apache.kafka.clients.NetworkClient) [2018-01-18 00: 45: 32,327] WARN [ Id = -1, groupId = console-consumer-95024] 7026: {another-test-topic = UNKNOWN_TOPIC_OR_PARTITION} (org. apache.kafka.clients.NetworkClient)

drwxr-xr-x 27 kafka kafka 4096 18 05 05:44 kafka

- "", , .

# 2: , :

// sasl-producer.properties
bootstrap.servers=server-01.eigenroute.com:9092
compression.type=none
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true  \
        keyTab="/path/to/producer1.whatever.keytab" \
        principal="producer1/[e-mail]";

// sasl-consumer.properties
bootstrap.servers=server-01.eigenroute.com:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true  \
        keyTab="/path/to/consumer1.whatever.keytab" \
        principal="consumer1/[e-mail]";

# producer command
KAFA_HEAP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.krb5.debug=true"     bin/kafka-console-producer.sh     --broker-list server-01.eigenroute.com:9092,server-02.eigenroute.com:9092     --topic another-test-topic --producer.config config/sasl-producer.properties

# consumer command
KAFA_HEAP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.krb5.debug=true"     bin/kafka-console-consumer.sh     --bootstrap-server server-01.eigenroute.com:9092,server-02.eigenroute.com:9092     --topic another-test-topic --consumer.config config/sasl-consumer.properties --from-beginning

Answer 1

, , , . ( "a-test-topic2" ) Broker 1 ( ), Broker 2 ( ).

, Broker 1 off line, Broker 2 ( "a-test-topic3" ), , , , (?) 1.

1 ( "a-test-topic4" ) Broker 2 2 ... ! Isr 2,1 1,2, - 1 2.

... Broker 1 ( "a-test-topic5" ). ? , . Isr s. - Broker 1 Broker 2, Broker 2, voila - 1 2 Isrs 1,2 2,1.

, , , ?

Answer 2

:

WARN [ clientId = -] 1: { - = UNKNOWN_TOPIC_OR_PARTITION} (Org.apache.kafka.clients.NetworkClient)

, .

, "".

1) , dir , , cd data_dir/topic/partition, , . ( )

2) , , data_dir kafka?

3) "telnet kafka_host kafka_port", , kafka

4) -, kafka, /etc/hosts IP- . , IP- ( DNS )

5) .