Aggregate resource allocation for work at YARN

Question

I am new to Hadoop. When I start the task, I see the cumulative resource allocation for this task as 251248654 MB-s., 24462 seconds of vcore. However, when I find information about the cluster, it shows that the total amount of memory is 888 and the total amount of memory is 15.90 TB. Can someone tell me how this is related? which means MB-second and Vcore-seconds for the job.

Are there any materials on the Internet to find out? I tried surfing, give the correct answer

Answer 1

VCores-Total: Indicates the total number of VCores available in the cluster Memory-Total: Indicates the total memory available in the cluster. 

For example, I have one cluster node where I set the memory requirements for each container: 1228 MB (defined by configuration: yarn.scheduler.minimum-allocation-mb ) and vCores for each container up to 1 vCore (defined by configuration: yarn.scheduler.minimum-allocation-vcores ).

I installed: from yarn.nodemanager.resource.memory-mb to 9830 MB. Thus, on node (9830/1228 = 8) there can be only 8 containers.

So for my cluster:

 VCores-Total = 1 (node) * 8 (containers) * 1 (vCore per container) = 8 Memory-Total = 1 (node) * 8 (containers) * 1228 MB (memory per container) = 9824 MB = 9.59375 GB = 9.6 GB 

The figure below shows the indicators of the cluster:

Now let's see "MB-seconds" and "vcore-seconds" . As described in the code (ApplicationResourceUsageReport.java):

MB sec . The total amount of memory (in megabytes) in the application distributed the time in seconds during which the application was run.

vcore-seconds : the aggregated number of vcores that the application assigned once in seconds during which the application was launched.

The description is self-evident (remember the keyword: Aggregated).

Let me explain this with an example. I completed the DistCp task (which spawned 25 containers), for which I received the following:

 Aggregate Resource Allocation : 10361661 MB-seconds, 8424 vcore-seconds 

Now let's do a rough calculation of how long each container takes:

 For memory: 10361661 MB-seconds = 10361661 / 25 (containers) / 1228 MB (memory per container) = 337.51 seconds = 5.62 minutes For CPU 8424 vcore-seconds = 8424 / 25 (containers) / 1 (vCore per container) = 336.96 seconds = 5.616 minutes 

This means that, on average, each container took 5.62 minutes.

Hope this makes it clear. You can complete the task and confirm it yourself.