How to enable a bright scheduler?

Question

I would like to understand the insides of the Spark FAIR planning mode. The fact is that this does not look as fair as might be expected according to the official Spark documentation

It seems that jobs are not handled the same and are actually managed in fifo order.

For more information on the topic:

I use Spark on YARN. I am using the Java Spark API. To enable honesty mode, code:

SparkConf conf = new SparkConf(); conf.set("spark.scheduler.mode", "FAIR"); conf.setMaster("yarn-client").setAppName("MySparkApp"); JavaSparkContext sc = new JavaSparkContext(conf); 

Did I miss something?

Answer 1

It seems that you did not configure the pools, and all your tasks fell into the same default pool, as described in Setting the pool properties :

The properties of certain pools can also be changed through the configuration file.

and later

A complete example is also available in the conf / fairscheduler.xml.template file. Please note: all pools that are not configured in the XML file will simply receive the default values ​​for all settings (FIFO scheduling mode, weight 1 and minShare 0).

It may also be that you have not configured a local property to configure the pool to use for tasks (tasks), as described in Fair Pool Schedulers :

Without any intervention, newly submitted jobs fall into the default pool, but job pools can be set by adding the "local spark.scheduler.pool property" in SparkContext to the stream that sends them.

Finally, this may mean that you are using a single FIFO default pool, so a single pool in FIFO mode does not change anything compared to FIFO without pools.

Only you know the real answer :)