Geek Answers Handbook

Hadoop and card reduction on multi-core machines

I read a lot about Hadoop and Map-Reduce running on machine clusters. Does anyone know if the Apache distribution can be run on SMP with multiple cores. In particular, you can run multiple Map-Reduce processes on the same machine. The scheduler will take care of spreading them across several cores. Thank you - KG

+8
multicore hadoop
source share
3 answers

Yes. You have several cards and slot reduction on each machine, which are determined by RAM and CPU (each JVM instance needs 1 GB by default, so an 8-gigabyte machine with 16 cores should have 7 task slots)

from hadoop wiki

Use the configuration knob: mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum to control the number of maps / vaults generated simultaneously on TaskTracker. By default, it is set to 2, so you can see a maximum of 2 displays and 2 is reduced to this instance on TaskTracker.

You can configure those that are based on each task to accurately reflect your equipment (i.e. set them to higher numbers on a clearer task controller, etc.).

+8
source share

For Apache Hadoop 2.7.3, my experience is that enabling YARN will also support multi-core support. Here is a simple guide to enable YARN on a single node:

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_a_Single_Node

The default configuration works very well. If you want to configure basic usage, then maybe look at the settings for "yarn.scheduler.minimum-allocation-vcores" and "yarn.scheduler.maximum-allocation-vcores" in the yarn-site.xml file ( https: // hadoop .apache.org / docs / r2.7.1 / hadoop-yarn / hadoop-yarn-common / yarn-default.xml )

Also see instructions for setting up a simple Hadoop sandbox with multi-core support: https://bitbucket.org/aperezrathke/hadoop-aee

0
source share

You can use these lightweight MapReduce frameworks for multi-core computers.

for example

LeoTask: A Lightweight, Productive, and Reliable Map Creation Framework for Multi-Core Computers

https://github.com/mleoking/LeoTask

-one
source share

More articles:


All Articles