How to limit processing to a specified number of cores in spark offline mode

Question

We tried to use various combinations of settings, but mpstat shows that all or most processors are always used (in the same 8th system system)

Attempt:

install wizard for:

local[2]

send

conf.set("spark.cores.max","2")

in spark configuration

Also using

--total-executor-cores 2

and

--executor-cores 2

In all cases

mpstat -A

indicates that the entire processor is in use, not just the master.

So, I am now at a loss. We need to limit the use to a specified number of processors.

Answer 1

The spark seems to standaloneignore the setting spark.cores.max. This setting works in yarn.

Answer 2

, , . , , , :

from pyspark import SparkConf, SparkContext

# In Jupyter you have to stop the current context first
sc.stop()

# Create new config
conf = (SparkConf().set("spark.cores.max", "2"))

# Create new context
sc = SparkContext(conf=conf)

, . , , , , :)