Yarn handles resource management and also services batch workloads that MapReduce and real-time workloads can use.
There are memory settings that can be set at the level of the yarn container, as well as at the level of the cartographer and gearbox. Memory is requested as the size of the yarn container increases. The crankcase and gearbox tasks are performed inside the container.
mapreduce.map.memory.mb and mapreduce.reduce.memory.mb
the above parameters describe the upper memory limit for the map reduction task, and if the memory signed by this task exceeds this limit, the corresponding container will be killed.
These parameters determine the maximum amount of memory that can be assigned to the cartographer and, accordingly, reduce tasks. Let's look at an example: Mapper is bound by an upper limit for memory, which is defined in the mapreduce.map.memory.mb configuration parameter.
However, if the value of yarn.scheduler.minimum-allocation-mb is greater than this value of mapreduce.map.memory.mb , then yarn.scheduler.minimum-allocation-mb , and containers of this size are returned .
This parameter must be set carefully, and if it is not set correctly, this can lead to errors or OutOfMemory errors.
mapreduce.reduce.java.opts and mapreduce.map.java.opts
This property value must be less than the upper bound of the map / reduce task, as defined in mapreduce.map.memory.mb / mapreduce.reduce.memory.mb , since it must correspond to the memory allocation for the map / reduce task.
Gaurav mishra
source share