Geek Answers Handbook

Shell Sparks: Invalid Initial Heap Size

I run spark-1.6.1 on Linux, and when I install SPARK_DRIVER_MEMORY in spark-env.sh on 4g and run the spark shell, I get the following:

Invalid initial heap size: -Xms4g
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

How to increase the maximum displayed size? and if I set SPARK_DRIVER_MEMORY = 3g (or below 3g), everything will be fine.
PS My computer has more than 4 g of memory, and my jdk has 1.6 and 64 bits.

+4
source share
1 answer

If you use Spark on Yarn, then Spark.driver.memory should have less memory for the yarn container. Since spark applications will work inside the yarn container

Spark.driver.memory(spark-default.conf) < yarn.scheduler.maximum--MB ( site.xml)

+1

More articles:


All Articles