YARN Application Wizard cannot connect to resource manager

Question

I have a 4 node cluster (1 Namenode / Resource Manager 3 datanodes / node)

I am trying to run a simple tez orderedWordCount example

hadoop jar C:\HDP\tez-0.4.0.2.1.1.0-1621\tez-mapreduce-examples-0.4.0.2.1.1.0-1621.jar orderedwordcount sample/test.txt /sample/out 

The work is accepted, the application wizard and container are installed, but on nodemanager I see these logs

2014-09-10 17: 53: 31 982 INFO [ServiceThread: org.apache.tez.dag.app.rm.TaskSchedulerEventHandler] org.apache.hadoop.yarn.client.RMProxy: connection to ResourceManager at / 0.0.0.0: 8030

2014-09-10 17: 53: 34 060 INFO [ServiceThread: org.apache.tez.dag.app.rm.TaskSchedulerEventHandler] org.apache.hadoop.ipc.Client: retrying to connect to the server: 0.0.0.0/0.0 .0.0: 8030. Already tried 0 times; Repeat Policy - RetryUpToMaximumCountWithFixedSleep (maxRetries = 10, sleepTime = 1000 Milliseconds)

After configurable timeout, job failure

I searched for this problem and always pointed to yarn.resourcemanager.scheduler.address configuration. In all node and node resource managers, I correctly defined this configuration, but for some reason I am not getting it

 <property> <name>yarn.resourcemanager.hostname</name> <value>10.234.225.69</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> </property> 

Answer 1

Your ResourceManager may be listening on the IPv6 port, while your work nodes (e.g. NodeManagers) can use IPv4 to connect to the ResourceManager

To quickly check if this is true, do

 netstat -aln | grep 8030 

If you get something like :::8030 , then your ResourceManager is really listening on the IPv6 port. If its port is IPv4, you should see something similar to 0.0.0.0:8030

To fix this, you may need to disable IPv6 on all your computers and try again.

Answer 2

This is because your resource manager is unavailable. Try pinging the resource manager from other nodes and see if it works. Maintain these configurations in a cluster.

Answer 3

In Hadoop2 code, there is a problem with setting up the yarn.resourcemanager.scheduler.address file, for example:

 <property> <name>yarn.resourcemanager.scheduler.address</name> <value>qadoop-nn001.apsalar.com:8030</value> </property> 

Currently, it is incorrectly placed in the "conf" configuration in Hadoop-2.7.0 / SRC / Hadoop-yarn-project / Hadoop-yarn / Hadoop-yarn common / SRC / Main / Java / org / Apache / Hadoop / thread / client /RMProxy.java

To prove this problem, we fixed this file to directly enter our scheduler address. The patch below is hacking. The main reason is related to the "conf" object, which should load the "yarn.resourcemanager.scheduler.address" property.

  @Private protected static <T> T createRMProxy(final Configuration configuration, final Class<T> protocol, RMProxy instance) throws IOException { YarnConfiguration conf = (configuration instanceof YarnConfiguration) ? (YarnConfiguration) configuration : new YarnConfiguration(configuration); LOG.info("LEE: changing the conf to include yarn.resourcemanager.scheduler.address at 10.1.26.1"); conf.set("yarn.resourcemanager.scheduler.address", "10.1.26.1"); RetryPolicy retryPolicy = createRetryPolicy(conf); if (HAUtil.isHAEnabled(conf)) { RMFailoverProxyProvider<T> provider = instance.createRMFailoverProxyProvider(conf, protocol); return (T) RetryProxy.create(protocol, provider, retryPolicy); } else { InetSocketAddress rmAddress = instance.getRMAddress(conf, protocol); LOG.info("LEE: Connecting to ResourceManager at " + rmAddress); T proxy = RMProxy.<T>getProxy(conf, protocol, rmAddress); return (T) RetryProxy.create(protocol, proxy, retryPolicy); } } 

EDIT: We solved this problem by adding yarn-site.xml to CLASSPATH. no need to modify RMProxy.java