Geek Answers Handbook

Identify a living spark master during spark-submit

I have a 5 node spark cluster where a 2 node host is running. in the HA script (according to Zookeeper), any of them will be selected by the master.

at the time of submitting application using command

/bin/spark-submit  --class SparkAggregator.java --deploy-mode cluster --supervise --master spark://host1:7077

getting error 
Can only accept driver submissions in ALIVE state. Current state: STANDBY.

spark-submit doe not allow multiple master name in --master.

Question:
How to identify the elected master at the time of spark-submit.

Thanks
Pankaj
+4
source share
2 answers

A wizard can accept multiple spark masters, so if you have multiple lists with a comma in between. eg.

/ bin / spark-submit --class SparkAggregator.java - deployment cluster -supervise --master spark: // host1: 7077, host2: 7077, host3: 7077

If you try to connect to all of them, the first one that answers will allow you to use several masters in a cluster where only one is active and the rest are in standby mode.

+3
source

Spark API, Spark Cluster

API http://SPARK_MASTER_IP:8080/json/

-

    {
  "url" : "spark://10.204.216.233:7077",
  "workers" : [ {
    "id" : "worker-20170606104140-10.204.217.96-40047",
    "host" : "10.204.217.96",
    "port" : 40047,
    "webuiaddress" : "http://10.204.217.96:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760671542
  }, {
    "id" : "worker-20170606104144-10.204.219.15-42749",
    "host" : "10.204.219.15",
    "port" : 42749,
    "webuiaddress" : "http://10.204.219.15:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760675649
  }, {
    "id" : "worker-20170606104151-10.204.217.249-35869",
    "host" : "10.204.217.249",
    "port" : 35869,
    "webuiaddress" : "http://10.204.217.249:8081",
    "cores" : 4,
    "coresused" : 0,
    "coresfree" : 4,
    "memory" : 29713,
    "memoryused" : 0,
    "memoryfree" : 29713,
    "state" : "ALIVE",
    "lastheartbeat" : 1496760682270
  } ],
  "cores" : 12,
  "coresused" : 0,
  "memory" : 89139,
  "memoryused" : 0,
  "activeapps" : [ ],
  "completedapps" : [ ],
  "activedrivers" : [ ],
  "status" : "ALIVE"
}
+1

More articles:


All Articles