Geek Answers Handbook

Get yarn configuration from command line

In EMR, is there a way to get a specific configuration value based on a configuration key using the yarn ?

For example, I would like to do something like this

 yarn get-config yarn.scheduler.maximum-allocation-mb
+7
elastic-map-reduce hadoop emr yarn hadoop2
source share
1 answer

This is a bit unintuitive, but it turns out that the hdfs getconf is able to check the configuration properties for YARN and MapReduce, not just HDFS.

 > hdfs getconf -confKey fs.defaultFS hdfs://localhost:19000 > hdfs getconf -confKey dfs.namenode.name.dir file:///Users/chris/hadoop-deploy-trunk/data/dfs/name > hdfs getconf -confKey yarn.resourcemanager.address 0.0.0.0:8032 > hdfs getconf -confKey mapreduce.framework.name yarn

The advantage of using this is that you will see the actual final results of any configuration properties, since they are actually used by Hadoop. This will take into account some more complex configuration patterns, such as using XInclude in XML files or property substitutions, for example:

  <property> <description>The address of the applications manager interface in the RM.</description> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property>

Any scripting approach that tries to parse XML files directly is unlikely to exactly match the implementation, as is done inside Hadoop, so it’s best to ask Hadoop itself.

You may be wondering why the hdfs command can get configuration properties for YARN and MapReduce. Great question! This is somewhat a coincidence of an implementation requiring you to insert an instance of MapReduce JobConf into some objects created using reflection. The corresponding code can be seen here:

https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java#L82- L114

This code runs as part of the hdfs getconf . By launching a link to JobConf , it forces the class loading and static initialization of the corresponding MapReduce and YARN classes, which add yarn-default.xml, yarn-site.xml, mapred-default.xml and mapred-site.xml to the set of configuration files.

Since this is a coincidence with the implementation, it is possible that some of these actions will be changed in future versions, but it will be a change incompatible with the reverse change, so we would definitely not change this behavior inside the current Hadoop 2.x line. The Apache Hadoop Compatibility policy requires backward compatibility within the main version, so you can trust that this will continue for at least version 2.x.

+14
source share

More articles:


All Articles