How to limit dynamic self-allocation of resources in a Hadoop cluster under a yarn?

Question

In our Hadoop cluster, which runs under the yarn, we are faced with the problem that some smart people can consume significantly larger chunks of resources by setting up Spark jobs on pySpark Jupyter laptops, for example:

conf = (SparkConf() .setAppName("name") .setMaster("yarn-client") .set("spark.executor.instances", "1000") .set("spark.executor.memory", "64g") ) sc = SparkContext(conf=conf) 

This leads to a situation where these people literally crowd out others less “smarter”.

Is there a way to prevent users from independently allocating resources and leaving the allocation of resources exclusively for yarn?

Answer 1

YARNs have very good support for capacity planning in a cluster with several tenants in turn, YARN ResourceManager uses CapacityScheduler by default.

Here we take the queue name as alpha in the spark clause for a demo purpose.

 $ ./bin/spark-submit --class path/to/class/file \ --master yarn-cluster \ --queue alpha \ jar/location \ args 

Setting up queues:

CapacityScheduler has a predefined queue called root. All queues in the system are children of the root queue. In capacity-scheduler.xml the yarn.scheduler.capacity.root.queues parameter yarn.scheduler.capacity.root.queues used to define child queues;

for example, to create 3 queues, specify the name of the queues in a comma-separated list.

 <property> <name>yarn.scheduler.capacity.root.queues</name> <value>alpha,beta,default</value> <description>The queues at the this level (root is the root queue).</description> </property> 

These are several important properties for capacity planning.

 <property> <name>yarn.scheduler.capacity.root.alpha.capacity</name> <value>50</value> <description>Queue capacity in percentage (%) as a float (eg 12.5). The sum of capacities for all queues, at each level, must be equal to 100. Applications in the queue may consume more resources than the queue's capacity if there are free resources, providing elasticity.</description> </property> <property> <name>yarn.scheduler.capacity.root.alpha.maximum-capacity</name> <value>80</value> <description>Maximum queue capacity in percentage (%) as a float. This limits the elasticity for applications in the queue. Defaults to -1 which disables it.</description> </property> <property> <name>yarn.scheduler.capacity.root.alpha.minimum-capacity</name> <value>10</value> <description>Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For eg, suppose the value of this property is 25. If two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. If a third user submits an application, no single user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed. The default is 100. Value is specified as a integer.</description> </property> 

References: YARN Property Queue CapacityScheduler