Geek Answers Handbook

How to install vcore in hasoop mapreduce / yarn?

Below is my configuration:

**mapred-site.xml**
map-mb : 4096 opts:-Xmx3072m
reduce-mb : 8192 opts:-Xmx6144m

**yarn-site.xml**
resource memory-mb : 40GB
min allocation-mb : 1GB

Vcores in the hadoop cluster displays 8 GB, but I do not know how to calculate or where to configure it.

hope someone can help me.

+4
source share
3 answers

Short answer

This, most likely, doesn’t matter if you just started hasoop out of the box on your single node cluster or even a small personal distributed cluster. You just need to worry about memory.

Long answer

vCores , . YARN , CPU. vCores Hadoop!

nodemanager vcores 1. ! - 2 4.. :

yarn.nodemanager.resource.memory-mb/mapreduce.[map|reduce].memory.mb

, vCores :

yarn.nodemanager.resource.cpu-vcores/mapreduce.[map|reduce].cpu.vcores

. capacity-scheduler.xml DefaultResourceCalculator DominantResourceCalculator.

vCores vCores, node? , LinuxContainerExecutor DefaultContainerExecutor, Linux, . , .

+6

yarn.nodemanager.resource.cpu-vcores - , .

mapreduce.map.cpu.vcores - ,

mapreduce.reduce.cpu.vcores -

+4

, , , .

. / hasoop , ResourceManager . ResourceManager node NodeManager. ( ). . this.

. NodeManager . : yarn.nodemanager.resource.memory-mb yarn.nodemanager.resource.cpu-vcores $HADOOP_CONF_DIR/yarn-site.xml. , .

. , /. ( ).

conf.set("mapreduce.map.cpu.vcores", "4");
conf.set("mapreduce.map.memory.mb", "2048");

4 2048 .

, , yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.am.resource.cpu-vcores.

$HADOOP_CONF_DIR/mapred-default.xml.

For more options and default values, I would recommend that you take a look at this and this

+3
source

More articles:


All Articles