How to run Spark 2.0 on EC2

Question

With the release of Spark 2.0 today, they removed the built-in support for launching the Spark EC2 cluster on AWS:

https://spark.apache.org/releases/spark-release-2-0-0.html#removals-behavior-changes-and-deprecations

Spark EC2 script completely moved to UC Berkeley AMPLab external repository

On the AMPLab GitHub page, it contains the following instructions:

https://github.com/amplab/spark-ec2/tree/branch-2.0#launching-a-cluster

Change to the ec2 directory in the Apache Spark release that you downloaded.

The problem is that in boot 2.0 there is no ec2 folder. Does anyone know how I can run a Spark 2.0 cluster in EC2?

Thanks in advance.

Answer 1

LAST EDIT

For those who have this problem, the answer is simpler: here .

EDIT 2

I realized after the first editing that it is a bit confusing, so here is a new change for those who may find it useful in the future.

The problem is that Spark no longer provides the ec2 directory as part of the official distribution. If you are used to deploying your autonomous clusters this way, this is a problem.

The solution is simple:

• Download the official ec2 catalog as described in the Spark 2.0.0 documentation.
• If you simply copy the directory to Spark 2.0.0 and run the spark-ec2 executable to simulate how everything works in Spark 1. *, you can deploy your cluster as usual. But when you enter this, you will realize that not one of the binaries exists anymore.
• So, as soon as you deploy your cluster (as usual with spark-ec2 , which you downloaded in step 1), you need rsync to create a local Spark 2.0.0 directory as the master of your newly created cluster. Once this is done, you can spark-submit to complete the tasks as usual.

Actually simple, but it seems to me that the Spark docs can be understood by all of us as normal.

---

EDIT: That was the right decision. For those who have the same question: download the ec2 directory from AMPLab, as Spark suggests, put this folder in the local Spark-2.0.0 and usual startup scripts. Apparently, they only turned off the directory for maintenance purposes, but the logic is the same. It would be nice to have a few words about this in the Spark docs.

---

I tried the following: cloned the spark-ec2-branch-1.6 directory from the AMPLab link into the spark-2.0.0 directory and tried to start the cluster using the regular ./ec2/spark-ec2 . Maybe this is what they want from us?

I am running a small 16 node cluster. I see this on the AWS dashboard, but the terminal got stuck printing a regular SSH error in the past ... almost two hours.

Warning: SSH connection error. (This could be temporary.) Host: ec2-54-165-25-18.compute-1.amazonaws.com SSH return code: 255 SSH output: ssh: connect to host ec2-54-165-25-18.compute-1.amazonaws.com port 22: Connection refused

Will be updated if I find anything useful.

Answer 2

You need to download all sources from here . Please note that I provided the 2.0 branch of github. Launch instructions can be found in the previous link and here is a blog post I wrote about a subject that can simplify your life.

I hope I managed to help! :)

Answer 3

Amplab Spark-ec2 is no longer supported by its github repo page. You can create a spark cluster using flintrock instead of AWS EC2 .