How to get application id from feed id or driver id programmatically

Question

I am sending Spark Job in cluster deployment mode. I get the Submit ID in my code. To use Spark rest Api, we need applicationId. So, how can we programmatically get the application identifier through the send identifier

Answer 1

To answer this question, it is assumed that it is known that the certificate identifier can be obtained using the function:

scala> spark.sparkContext.applicationId 

If you want to receive, this is a little different. Otherwise, the API can send commands. For example, as in the following useful tutorial about the Apache REST API: http://arturmkrtchyan.com/apache-spark-hidden-rest-api Another solution is to program it yourself. You can send a Spark message in the following style:

 $SPARK_HOME/bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://my-cluster:7077 \ --deploy-mode cluster \ /path/to/examples.jar 1000 

Here, in the background of the terminal, you will see a log file. There is an application identifier (App_ID) in the log file. $SPARK_HOME here is the environment variable that leads to the folder in which the spark is located.

Using the Python example, you can get the App_ID in the code described below. First, we create a list to send a command with Python with our subprocess module. The subprocess module has the ability to make PIPE from which you can extract the log information, instead of using Spark in its standard version to send it to the terminal. Be sure to use communicate() after Popen to prevent the OS from waiting. Then divide it on the line and clear it to find the App_ID . An example can be found below:

 submitSparkList=['$SPARK_HOME/bin/spark-submit','--class','org.apache.spark.examples.SparkPi','--master spark://my-cluster:7077','--deploy-mode cluster','/path/to/examples.py','1000'] sparkCommand=subprocess.Popen(submitSparkList,stdout=subprocess.PIPE,stderr=subprocess.PIPE) stdout, stderr = sparkCommand.communicate() stderr=stderr.splitlines() for line in stderr: if "Connected to Spark cluster" in line: #this is the first line from the rest API that contains the ID. Scrape through the logs to find it. app_ID_index=line.find('app-') app_ID=line[app_ID_index:] #this gives the app_ID print('The app ID is ' + app_ID) 

Following the Python manual, it contains a warning when the connect () function is not used: https://docs.python.org/2/library/subprocess.html

A warning. This will be inhibited when using stdout = PIPE and / or stderr = PIPE, and the child process generates sufficient output to the channel so that it blocks waiting for the OS buffer to receive more data. Use message () to avoid this.

Answer 2

I have a problem too, I think

web UI ( http: // spark_hostIp: port / 8080 ) bypass ~

get ApplicationID

one of the methods