The container runs out of memory

Question

In Hadoop v1, I assigned every 7 slots for a 1 GB gimbal and gearbox, my cartographers and gearboxes work fine. My machine has 8G memory, 8 processors. Now with YARN, when you run the same application on the same machine, I got a container error. By default, I have the following settings:

<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>8192</value> </property> 

This gave me an error:

 Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container. 

Then I tried to set a memory limit in mapred-site.xml:

  <property> <name>mapreduce.map.memory.mb</name> <value>4096</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>4096</value> </property> 

But still an error occurs:

 Container [pid=26783,containerID=container_1389136889967_0009_01_000002] is running beyond physical memory limits. Current usage: 4.2 GB of 4 GB physical memory used; 5.2 GB of 8.4 GB virtual memory used. Killing container. 

I am confused why the map task needs so much memory. In my opinion, 1 GB of memory is enough for my map / reduce task. Why, since I assign more memory to the container, does the task use more? Is it because every task gets more splits? I feel that it is more efficient to reduce the size of the container a bit and create more containers so that more tasks are performed in parallel. The problem is, how can I make sure that no more partitions are assigned to each container than it can handle?

Answer 1

You must also properly configure maximum memory allocations for MapReduce. From this HortonWorks tutorial :

[...]

Each machine in our cluster has 48 GB of RAM. Part of this RAM must be reserved for use by the operating system. On each node, allocate 40 GB of RAM to use> YARN and leave 8 GB for the operating system

For our example cluster, we have the minimum RAM for the container (yarn.scheduler.minimum-allocation-mb) = 2 GB. Well, at the same time, assign 4 GB for the card task Containers and 8 GB for the Reduction task Containers.

In mapred-site.xml:

mapreduce.map.memory.mb : 4096

mapreduce.reduce.memory.mb : 8192

Each container will run a JVM for Map and Reduce tasks. JVM The heap size should be set lower than Map and Reduce memory are defined above, so that they are within the container memory allocated YARN.

In mapred-site.xml:

mapreduce.map.java.opts : -Xmx3072m

mapreduce.reduce.java.opts : -Xmx6144m

The above settings adjust the upper limit of physical memory that Map and Reduce Tasks will use .

Summarizing:

1. In YARN, you should use mapreduce configs, not mapred . UPDATE: This comment is no longer applicable when you edited your question.
2. In fact, you are configuring how much you want to request, and not what maximum can be allocated.
3. Maximum limits are configured using the java.opts settings listed above.

Finally, you can check out this other SO question describing a similar problem (and solution).

Answer 2

At the yarn level, a check is made of the ratio of virtual and physical memory use. The problem is not only that the virtual machine does not have enough physical memory. But this is because the use of virtual memory is greater than expected for this physical memory.

Note This happens in Centos / RHEL 6 due to aggressive allocation of virtual memory.

This can be solved in one of the following ways:

1. Disable virtual memory usage checking by setting yarn.nodemanager.vmem-check-enabled to false ;

2. Increase the VM: PM ratio by setting a higher value for yarn.nodemanager.vmem-pmem-ratio .

References :

https://issues.apache.org/jira/browse/HADOOP-11364

http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/

Add the following property to yarn-site.xml

  <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> <description>Whether virtual memory limits will be enforced for containers</description> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description> </property> 

Answer 3

I had a very similar problem with using HIVE in EMR. None of the existing solutions worked for me - i.e. None of the mapreduce configurations worked for me; and did not set yarn.nodemanager.vmem-check-enabled to false.

However, the installation of tez.am.resource.memory.mb worked in the end, for example:

 hive -hiveconf tez.am.resource.memory.mb=4096 

Another setting to configure is yarn.app.mapreduce.am.resource.mb

Answer 4

I can not comment on the accepted answer due to the low reputation. However, I would like to add that this is by design. NodeManager kills your container. It looks like you are trying to use the streams of chaops that are executed as a child of the map reduction task. NodeManager controls the entire process tree of the task, and if it consumes more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb, respectively, we expect Nodemanager to kill the task, otherwise your task is to steal memory belonging to other containers that you do not need.

Answer 5

While working with sparks in EMR, I had the same problem, and setting maximizeResourceAllocation=true did the trick; hope this helps someone. You must install it when creating the cluster. From EMR Docs:

 aws emr create-cluster --release-label emr-5.4.0 --applications Name=Spark \ --instance-type m3.xlarge --instance-count 2 --service-role EMR_DefaultRole --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole --configurations https://s3.amazonaws.com/mybucket/myfolder/myConfig.json 

Where myConfig.json should say:

 [ { "Classification": "spark", "Properties": { "maximizeResourceAllocation": "true" } } ] 

Answer 6

We also encountered this problem recently. If the problem is with the memory of the cartographer, I would like to offer a few things to check.

• Check if the combiner is on or not ? If yes, then this means that the reduction logic should be executed for all records (mapper output). This happens in memory. Depending on your application, you need to check if enabling the combiner helps or not. A trade-off between transmission bytes over the network and the time / memory / CPU spent for the logic to reduce the number of X records.
  • If you think the combiner is not of great value, just turn it off.
  • If you need a combiner, and 'X' is a huge number (say, millions of records), then consider changing the split logic (for input formats, use a smaller block size by default, usually 1 block size = 1 split) to display fewer records into one cartographer.
• The number of processed records in one mapper. Remember that all of these entries must be sorted in memory (mapper output is sorted). Try setting mapreduce.task.io.sort.mb (200 MB by default) to a higher value if necessary. mapred-configs.xml
• If any of the above does not help, try running the display logic as a separate application and profile the application using Profiler (for example, JProfiler) and see where the memory is used. This can give you a very good understanding.