Difference between 3 memory options in Hadoop 2?

Question

I use Hadoop 2.0.5 (Alpha) to do relatively large jobs, and I came across these errors:

The container [pid = 15023, containerID = container_1378641992707_0002_01_000029] going beyond the limits of virtual memory. Current usage: 492.4 MB 1 GB used physical memory; 3.3 GB virtual memory used 2.1 GB . Killing container.

Then I found out about these two parameters:

property yarn.nodemanager.vmem-pmem-ratio , which is set to 2.1 by default.

yarn.app.mapreduce.am.command-opts , which by default is set to -Xmx1024mb (= 1 GB).

This explains the limits noted above.

Setting these parameters to a larger value helped, but then I found this parameter: yarn.app.mapreduce.am.resource.mb , which is set to 1536 by default.

And I can’t tell the difference between 3 of the description given in the Hadoop XML standards, and how I should install them correctly for optimization purposes.

An explanation or a good recommendation would be very helpful.

Answer 1

since we know that yarn is a new architecture for managing resources in the haop ecosystem.

property yarn.nodemanager.vmem-pmem-ratio: Defines the relation of virtual memory to available memory physics. Here 2.1 means that virtual memory will be twice as much physical memory.

yarn.app.mapreduce.am.command-opts: Hide and seek ApplicationMaster (AM) is responsible for providing the necessary resources. Thus, this property determines how much memory is required to run AM. Do not confuse this with nodemanager where the work will be done.

yarn.app.mapreduce.am.resource.mb: This property sets the criteria for selecting a resource for a particular job. Here, 1536 funds are given. Any person with equal or greater available memory will be selected to complete the task.

Answer 2

@Twid's answer is mixed. According to the white paper here: http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

"yarn.app.mapreduce.am.resource.mb" indicates "The amount of memory that MR AppMaster requires", in other words, it determines how much memory the container that is used to run the needs of the application wizard, this is not related to the containers that are used to start the converters / gearboxes.