Geek Answers Handbook

Yarn: how to use the full resources of the cluster?

So I have a cloudera cluster with 7 work nodes.

  • RAM 30 GB
  • 4 vCPUs

Here are some of my configurations that I found important (from Google) in tuning my cluster performance. I am running with:

  • yarn.nodemanager.resource.cpu-vcores => 4
  • yarn.nodemanager.resource.memory-mb => 17 GB (reserved for OS and other processes)
  • mapreduce.map.memory.mb => 2GB
  • mapreduce.reduce.memory.mb => 2GB
  • Startup nproc=> 4 (Number of Processing Units Available)

Now I'm worried, when I look at mine ResourceManager, I see the available memory as 119 GB, which is fine. But when I run a heavy task sqoopand my cluster is at its peak, it uses only ~59 GBmemory, leaving the memory ~60 GBunused.

One of the ways that I see can fix this problem with unused memory, increases map|reduce.memoryto 4 GB, so we can use up to 16 GB on node.

Another way is to increase the number of containers, and I'm not sure how to do it.

  • 4 cores x 7 nodes = 28 possible containers. 3, which are used by other processes, only 5 are currently available for sqoop.

What should be the correct configuration to improve cluster performance in this case. Can I increase the number of containers, say, 2 containers per core. And is it recommended?

Any help or suggestions on cluster configuration would be greatly appreciated. Thank.

+3
source share
1 answer

26 , YARN 26 .

7 2 - 26 , :

  • Node1: 4 mappers = > 8
  • Node2: 4 mappers = > 8
  • Node3: 4 mappers = > 8
  • Node4: 4 mappers = > 8
  • Node5: 4 mappers = > 8
  • Node6: 3 mappers = > 6
  • Node7: 3 mappers = > 6
  • : 26 mappers = > 52

, , , , mappers , 26x2 = 52 . , (-) ApplicationMaster, 59 - , .

, , 26 , . 60 , , , , . 60 , . .

:

, , , , vcores. Apache Hadoop, :

yarn.scheduler.capacity.resource-calculator: ResourceCalculator, . , , org.apache.hadoop.yarn.util.resource.DefaultResourseCalculator , DominantResourceCalculator Dominant-resource , , CPU .. Java ResourceCalculator.

yarn.nodemanager.resource.cpu-vcores 4, 1 vcore, 4 node .

yarn.nodemanager.resource.cpu-vcores 8. .

+1

More articles:


All Articles