Geek Answers Handbook

Launch a cluster of haops on docker containers

I want to run a multi-user cluster node hasoop with each node inside a docker container on a different host. This image - https://github.com/sequenceiq/hadoop-docker works well to run hadoop in pseudo-distributed mode, what is the easiest way to change this so that each node in a different container on a separate ec2 node?

+7
docker amazon-ec2 networking cluster-computing hadoop
source share
2 answers

I did this with two containers with master and slave nodes on two different ubuntu nodes. I created a network between containers using weaving. I added container images to the docker hub account div4. I installed hadoop in the same way as on other hosts. I added two images with coomands to run haddop on them here:

https://registry.hub.docker.com/u/div4/hadoop_master/ https://registry.hub.docker.com/u/div4/hadoop_slave/ .

+4
source share

The people at sequenceiq have created a new project called cloud-break, which is designed to work with various cloud computing providers and easily creates hasoop clusters. You just have to enter your credentials, and then it works the same for all providers, as far as I can see.

So, for ec2, this will probably be the easiest solution (especially because of the nice graphical interface):

https://github.com/sequenceiq/cloudbreak-deployer

+2
source share

More articles:


All Articles