Connecting Kafka-Python with a Kerberos Cluster

Question

I am trying to connect to Kafka with Kafka-Python, the Kafka cluster has Kerberos, we need to collect some commands in order to take a few steps.

I created one topic in the cluster and I ran a test with ./kafka-console-producer.sh and ./kafka-console-consumer.sh and it works very well.

But when I try to connect to Kafka-Python, I have a problem. See my code below:

 def produce(): print ('Producer') k_producer = KafkaProducer(bootstrap_servers='hostname:6667', security_protocol='SASL_PLAINTEXT', sasl_plain_username='machine_usr', sasl_plain_password='machine_pwd', sasl_mechanism='PLAIN') for i in range(10): print ('Before send') k_producer.send('myTopic', 'Testing My Topic ' + str(i)) print ('After send') 

Well, by running this stuff, I got this erro message after 30 seconds:

 File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\producer\kafka.py", line 328, in __init__ File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 202, in __init__ File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 791, in check_version kafka.errors.NoBrokersAvailable: NoBrokersAvailable 

I run it on a remote machine. And bootstrap_server I used the name and port of Zookeeper, but it didn’t work.

I found a few things about this, and in git Kafka-Python I found that they are implemented .

Is there any other way to connect?

Answer 1

Guys,

I found a problem.

The problem is that Kerberos is not supported for the Kafka manufacturer in Python using the key tab.

To use the key tab, we need to set the Java environment variable.

According to Hortonworks, we need to set client_jaas_client to connect.

The solution used Py4j to invoke Kafka Producer in the JVM.

See an example here .

Answer 2

If the task is to solve this problem in python, another alternative could be to use the confluent-kafka-python library, which internally uses librdkafka , which is written in C, and supports SASL and the use of the keytab file. This does not require a separate Java process to communicate with Kafa over SASL.

See also the librdkafka library documentation for instructions :

https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - general introduction https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - properties that can be transferred to constructor confluent_kafka.Producer and confluent_kafka.Consumer in python

Answer 3

Please, could you tell me how you used Py4j to solve the problem? I have the same error!

Thanks in advance !