Geek Answers Handbook

How to use Jupyter + SparkR and configure R install

I use a Dockerized image and a Jupyter block along with the SparkR core. When I create a SparkR laptop, it uses the Microsoft R installation (3.3.2) instead of the vanilla CRAN R install (3.2.3).

The Docker image that I use installs some custom R libraries and Python pacakages, but I do not explicitly install Microsoft R. Regardless of whether I can remove Microsoft R or use it next to each other, how can I force my kernel SparkR use custom installation R ?

Thank you in advance

0
source share
2 answers

, , Jupyter kernel.json, ( ), jupyter kernelspec list; , (Linux):

$ jupyter kernelspec list
Available kernels:
  python2       /usr/lib/python2.7/site-packages/ipykernel/resources
  caffe         /usr/local/share/jupyter/kernels/caffe
  ir            /usr/local/share/jupyter/kernels/ir
  pyspark       /usr/local/share/jupyter/kernels/pyspark
  pyspark2      /usr/local/share/jupyter/kernels/pyspark2
  tensorflow    /usr/local/share/jupyter/kernels/tensorflow

, , kernel.json R (ir)

{
  "argv": ["/usr/lib64/R/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
  "display_name": "R 3.3.2",
  "language": "R"
}

pyspark2:

{
 "display_name": "PySpark (Spark 2.0)",
 "language": "python",
 "argv": [
  "/opt/intel/intelpython27/bin/python2",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "env": {
  "SPARK_HOME": "/home/ctsats/spark-2.0.0-bin-hadoop2.6",
  "PYTHONPATH": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python:/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/lib/py4j-0.10.1-src.zip",
  "PYTHONSTARTUP": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/pyspark/shell.py",
  "PYSPARK_PYTHON": "/opt/intel/intelpython27/bin/python2"
 }
}

, argv - GNU R ir Intel Python 2.7 pyspark2. , GNU R, .

0

R, , Spark :

    "spark.r.command": "/custom/path/bin/R",
    "spark.r.driver.command": "/custom/path/bin/Rscript",
    "spark.r.shell.command" : "/custom/path/bin/R"

: https://spark.apache.org/docs/latest/configuration.html#sparkr

0

More articles:


All Articles