SparkR Error in sparkR.init (master = "local") in RStudio

Question

I installed the SparkR package from the Spark distribution into the R library. I can call the following command and it seems to work correctly: library(SparkR)

However, when I try to get the Spark context using the following code,

 sc <- sparkR.init(master="local") 

After some time, it does not work with the following message:

 Error in sparkR.init(master = "local") : JVM is not ready after 10 seconds 

I installed JAVA_HOME and I have a working RStudio where I can access other packages like ggplot2. I do not know why it does not work, and I do not even know where to investigate it.

Answer 1

I had the same problem and my spark-submit.cmd also did not execute from the command line. The following steps worked for me

Go to the environment variables and in the system variables select the PATH variable name. Along with other values, add c:/Windows/System32/ , separated by semicolons. This made my spark-submit.cmd work from the command line and ultimately from Rstudio.

I realized that we get the above problem only if all the required path values ​​are not specified. Ensure that all path values ​​(R, Rtools) are specified in the environment variables. For example, my Rtools path was c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin

Hope this helps.

Answer 2

Try granting permissions to execute C: /sparkpath/bin/spark-submit.cmd. It worked for me.

Answer 3

I got the same error message. My case is a little different since I successfully ran SparkR in RStudio. But after a few days it no longer works.

Having looked at the conversion between Shivaram Venkataraman and Prakash Ponshankarhinnhusamy, I realized that this may have something to do with a work permit.

https://issues.apache.org/jira/browse/SPARK-8603

So what I did, which ultimately works, is that I will unzip tar.gz again into my C: / folder (it was previously stored in D: / drive) AND SparkR WORKS !!!

RStudio Desktop Screenshot

Answer 4

After installing Hadoop, Spark follows.

spark_path <- strsplit(system("brew info apache-spark",intern=T)[4],' ')[[1]][1] # Get your spark path .libPaths(c(file.path(spark_path,"libexec", "R", "lib"), .libPaths())) library(SparkR