Moving thread nodes not communicating with node master?

Question

I can not see my nodes when I do yarn node -list , although I configured /etc/hadoop/conf/yarn-site.xml with the correct properties (it seems to me, at least for that matter, Slave nodes are not in the Yarn ResourceManager )

Here is what I have done so far:

• installed resourcemanager on the main
• installed nodemanager on slaves

• marked yarn-site.xml for this on ALL sites:

  <property> <name>yarn.resourcemanager.hostname</name> <value>master-node</value> </property>

• after changing the configuration file, restart the resourcemanager and nodemanager on the master and slave devices, respectively.

But still, when I do yarn node -list , I only see

 Total Nodes: 0 Node-Id Node-state Node-Http-Address Number-of-Running-Containers 

In my nodes, I looked at the .out files in /var/log/hadoop-yarn/ , and I see this in them:

 ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 244592 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

EDIT: when I look at the .log files, I see the following, but I'm not sure how to fix it:

  INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address') Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address') 

How to connect slave nodes to my node wizard?

Answer 1

The value set for yarn.resourcemanager.hostname acts as the base value for all ResourceManager properties. The yarn.resourcemanager.resource-tracker.address defaults to ${yarn.resourcemanager.hostname}:8031 . See yarn-default.xml for a complete list of default YARN configurations.

And from nodemanager ERROR magazine,

 Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address') 

It seems that the yarn.resourcemanager.hostname property yarn.resourcemanager.hostname incorrectly configured as <master node ip>:8020 instead of <master node ip> on the sub nodes.

Modify yarn-site.xml on all sites to have

 <property> <name>yarn.resourcemanager.hostname</name> <value>master_node</value> <!-- IP address or Hostname of the node where Resource Manager is started, Omit the port number --> </property> 

Finally, restart the YARN services.

Answer 2

 please set all this properties and try <property> <name>yarn.resourcemanager.address</name> <value>master_node:8032</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master_node:8033</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master_node:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master_node:8031</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master_node:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value>master_node:8090</value> </property> 

Answer 3

You need to set ip for yarn.resourcemanager.hostname property. if you want to use the hostname, your machine must know which ip this hostname points to. Therefore, you need to add the host entry to the / etc / hosts file .

To do this,

• Open terminal

• Type vim / etc / hosts and press enter

• Add this line to the end of the file (use the i key to enable insertion)

  <your resourcemanager ip><space><your hostname>

   example: `192.168.1.23 master-node` 

• Save the file by typing <Esc> +: wq

• Restart nodemanager

I recommend using the ambari management tool to work with such products. This makes it easy to change the configuration at any time for the hadoop environment. Since manual work is always more likely to make a mistake.