Upgrading Apache Spark from 1.5.2 to 1.6.0 using homebrew, resulting in a permission denied error on execution

Question

I just updated Spark from 1.5.2 to 1.6.0 using Homebrew and reset my SPARK_HOME environment variable to /usr/local/Cellar/apache-spark/1.6.0 . Now, executing pyspark, it gives the resolved error.

If I go into the earlier installation directory 1.5.2 and run pyspark from there, it works fine. But starting pyspark from the installation directory 1.6.0 fails if this permission is denied.

/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: Permission denied

/usr/local/Cellar/apache-spark/1.6.0/bin/load-spark-env.sh: line 2: exec: /usr/local/Cellar/apache-spark/1.6.0/libexec/bin/load-spark-env.sh: cannot execute: Undefined error: 0

What could be the reason for this?

Answer 1

unset SPARK_HOME && pyspark (credit for noli below)

Getting the SparkContext running in Ipython turned out to be simple.

unset SPARK_HOME IPYTHON=1 pyspark

Answer 2

I ran into the same problem and the easiest solution is to install $SPARK_HOME in /usr/local/Cellar/apache-spark/<your_spark_version>/libexec/ .

You can also build from source code, and you can find instructions here .

Basically just

 git clone https://github.com/apache/spark/` cd spark git checkout origin/branch-XY build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package 

You need to install $SPARK_HOME in the top level directory of the spark source code.

Answer 3

The ticket on the project page helped:

 unset SPARK_HOME && spark-submit 

https://github.com/Homebrew/homebrew/issues/48898#issuecomment-180633967

Answer 4

I also encountered the same error as the permission to run spark-shell . I changed the file permissions. It worked for me. Go to the parent directory spark-1.6.x and do the following:

 chmod 777 spark-1.6.x -R 

This will change the resolution of the file recursively.