Spark on Windows, Yarn on Linux

Question

Spark on Windows, Yarn on Linux

I have yarn running on top of hadoop 2.0 on a Linux environment and I have Spark on my Windows machine. I can run Spark locally on my windows. But I'm trying to use my direct line for Windows Yarn on Hadoop (Linux) to use yarn containers. How to set HADOOP_CONF_DIR on the window side so that it can point and see / etc / hadoop / conf on the Linux side. thank you

+5

hadoop hdfs yarn apache-spark pyspark

user2562618 Jul 15 '15 at 18:07

source share

No one has answered this question yet.

See related questions:

26

Sparks concept yarn concept

4

Exception: java.lang.Exception: when working with the main "thread" in the environment, HADOOP_CONF_DIR or YARN_CONF_DIR must be set. into sparks

3

spark-submit do not use YARN