How to check if Amazon Kinesis Python client is running

Question

I am trying to create an Amazon Kinesis Python consumer using the KCL library for Python ( https://github.com/awslabs/amazon-kinesis-client-python ). I started by checking the sample code. I was able to run part of the producer and consumer code in the sample code, but I can’t check if the data from my kinesia stream (with one shard) is transferred to the Python sample consumer script, sample_kclpy_app.py .

I used amazon_kclpy_helper.py to generate a Java command that would sample.properties Python script through the sample.properties file. I launched the Java command, and on the output of the terminal I see that the data from the Kinesis stream is being read. I added the statement print in function process_record the Python script user to verify whether the data has been transferred to it. But it does not appear on the terminal output.

I also tried using logging to generate STDOUT messages as well as to write to a file. I also added the assert 0 line in Python code to force the script to be disabled and see that the exception appears in the output log file. Then I intentionally added a syntax error to the Python code. However, it seems that all of them were not discovered by Java MultiLangDaemon, which simply continued to collect and publish INFO log messages.

What could be the problem? And is there a better way to check if data is actually being sent to the Python script script consumer process_record function?

Answer 1

After numerous trial and error, I realized that the Python script assigned to executableName should be visible from the Java KCL process that calls it.

A quick fix is ​​to set the PATH environment variable to the script directory and make the script an executable file (e.g. chmod +x )

Answer 2

I had the same problem when my code just didn't run. Only when I looked at the Node version did I realize that executableName in sample.properties is the command that runs, not just the python file name. I changed executableName = sample_kclpy_app.py to executableName = python sample_kclpy_app.py and started working as expected.