Geek Answers Handbook

Setting elasticsearch properties in spark-submit

I am trying to run Spark jobs that use Elastic Search, enter through the command line using spark-submit, as described in http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/spark.html

I set the properties in the file, but when spark-submit starts, it gives the following warnings:

~/spark-1.0.1-bin-hadoop1/bin/spark-submit --class Main --properties-file spark.conf SparkES.jar

Warning: Ignoring non-spark config property: es.resource=myresource
Warning: Ignoring non-spark config property: es.nodes=mynode
Warning: Ignoring non-spark config property: es.query=myquery
...
Exception in thread "main" org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed

My configuration file looks (with the correct values):

es.nodes      nodeip:port
es.resource   index/type
es.query      query

Setting properties in an object The configuration in code works, but I need to avoid this workaround.

Is there a way to set these properties through the command line?

+4
source share
2 answers

, ( , ?), :

import org.elasticsearch.spark.rdd.EsSpark

EsSpark.saveToEs(rdd, "spark/docs", Map("es.nodes" -> "10.0.5.151"))

Bye

+4

spark-submit, , "spark".

,

spark.es.nodes <es-ip>


val conf = new SparkConf()
conf.set("es.nodes", conf.get("spark.es.nodes"))
0

More articles:


All Articles