Hadoop yarn container does not allocate enough space

Question

I am running a Hadoop job, and in the yarn-site.xml file I have the following configuration:

<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>2048</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>4096</value> </property> 

However, I still sometimes get the following error:

 Container [pid=63375,containerID=container_1388158490598_0001_01_000003] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used. Killing container. 

I found that by increasing the value of yarn.scheduler.minimum-allocation-mb, the physical memory allocated for the container is increasing. However, I do not always want 4GB to be allocated for my container, and thought that by explicitly specifying the maximum size, I could solve this problem. I understand that Hadoop cannot understand how much memory it needs to allocate for the container before running mapper, since I should allocate more memory for the container only if it needs extra memory?

Answer 1

You must also properly configure memory allocations for MapReduce. From this HortonWorks tutorial :

[...]

For our example cluster, we have the minimum RAM for the container (yarn.scheduler.minimum-allocation-mb) = 2 GB. Therefore, assign 4 GB for the map task containers and 8 GB to reduce the task containers.

In mapred-site.xml:

mapreduce.map.memory.mb : 4096

mapreduce.reduce.memory.mb : 8192

In each container, a JVM will be launched for the Map and Reduce tasks. The JVM heap size should be set below the card and reduce the memory defined above so that they are within the memory container allocated by YARN.

In mapred-site.xml:

mapreduce.map.java.opts : -Xmx3072m

mapreduce.reduce.java.opts : -Xmx6144m

The above settings, configure the upper limit of the physical memory that the "Card" and "Reduce" will use .

Finally, someone from this thread on the Hadoop mailing list had the same problem, and in their case it turned out that they had a memory leak in their code.