How to change SparkContext properties in an Interactive PySpark session

Question

How can I change spark.driver.maxResultSize in the pyspark interactive shell? I used the following code

from pyspark import SparkConf, SparkContext conf = (SparkConf() .set("spark.driver.maxResultSize", "10g")) sc.stop() sc=SparkContext(conf) 

but it gives me an error

 AttributeError: 'SparkConf' object has no attribute '_get_object_id' 

Answer 1

So your vision is that SparkConf not a Java object, this is because it is trying to use SparkConf as the first parameter, if you do sc=SparkContext(conf=conf) instead, it should use your configuration. You might be better off starting a regular python program rather than stopping the intrinsic safety context by default and restarting it, but you will need to use the named parameter method to pass in the conf object anyway.

Answer 2

The correct way to embed intrinsic safety parameters for a given Spark context requires that the context be closed. For example:

 from pyspark import SparkContext SparkContext.setSystemProperty('spark.driver.maxResultSize', '10g') sc = SparkContext("local", "App Name") 

source: https://spark.apache.org/docs/0.8.1/python-programming-guide.html

ps if you need to close SparkContext just use:

 SparkContext.stop(sc) 

and double check the current settings that have been set that you can use:

 sc._conf.getAll()