Cluster hangs in ssh-ready state using Spark 1.2.0 EC2 launch script

Question

I am trying to start a stand-alone Spark cluster using its pre-packaged EC2 scripts, but it just hangs endlessly in ssh-ready state:

ubuntu@machine :~/spark-1.2.0-bin-hadoop2.4$ ./ec2/spark-ec2 -k <key-pair> -i <identity-file>.pem -r us-west-2 -s 3 launch test Setting up security groups... Searching for existing cluster test... Spark AMI: ami-ae6e0d9e Launching instances... Launched 3 slaves in us-west-2c, regid = r-b_______6 Launched master in us-west-2c, regid = r-0______0 Waiting for all instances in cluster to enter 'ssh-ready' state.......... 

However, I can use SSH in these cases without complaint:

 ubuntu@machine :~$ ssh -i <identity-file>.pem root@master-ip Last login: Day MMM DD HH:mm:ss 20YY from c-AA-BBB-CCCC-DDD.eee1.ff.provider.net __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2013.03-release-notes/ There are 59 security update(s) out of 257 total update(s) available Run "sudo yum update" to apply all updates. Amazon Linux version 2014.09 is available. root@ip-internal ~]$ 

I am trying to figure out if this is a problem in AWS or Spark scripts. I have never had this problem before.

Answer 1

Spark 1.3.0 +

This issue has been fixed in Spark 1.3.0 .

Spark 1.2.0

Your problem is because SSH silently stops due to conflicting entries in your known_hosts SSH file .

To solve the problem, add -o UserKnownHostsFile=/dev/null to the spark_ec2.py script like this .

If necessary, in order to clean up and avoid problems connecting to the cluster with SSH later, I recommend you:

• Remove all lines from ~/.ssh/known_hosts that include EC2 hosts, for example:

ec2-54-154-27-180.eu-west-1.compute.amazonaws.com,54.154.27.180 ssh-rsa (...)

2. Use this solution to stop checking and saving fingerprints of the temporary IP-code of your EC2 instances in general

Answer 2

I had the same problem and followed all the steps mentioned in the stream (basically adding -o UserKnownHostsFile = / dev / null to your spark_ec2.py script), but still it hung saying

 Waiting for all instances in cluster to enter 'ssh-ready' state 

Short answer:

Change the permission of the private key file and restart the spark-ec2 script

 [ spar@673d356d ]/tmp/spark-1.2.1-bin-hadoop2.4/ec2% chmod 0400 /tmp/mykey.pem 

Long answer:

To fix the problems, I modified spark_ec2.py and registered the used ssh command and tried to execute it on the command line, it was a bad key permission:

 [ spar@673d356d ]/tmp/spark-1.2.1-bin-hadoop2.4/ec2% ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /tmp/mykey.pem -o ConnectTimeout=3 uroot@52.1.208.72 Warning: Permanently added '52.1.208.72' (RSA) to the list of known hosts. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0644 for '/tmp/mykey.pem' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /tmp/mykey.pem Permission denied (publickey). 

Answer 3

I faced the same specific situation. I entered the python script in def is_ssh_available() and threw the return code and cmd.

 except subprocess.CalledProcessError, e: print "CalledProcessError " print e.returncode print e.cmd 

I had the location of the key file as ~/.pzkeys/mykey.pem - as an experiment, I changed it to fully qualified, i.e. /home/pete.zybrick/.pzkeys/mykey.pem and it worked fine.

Right after that, I encountered another error - I tried using --user=ec2-user (I try to avoid using root), then I got a permission error on rsync, deleted --user-ec2-user , so it will use root for by default, made another attempt with --resume , ran to a successful conclusion.

Answer 4

I used the absolute (non-relative) path to my identification file (inspired by Peter Zybrik) and did everything that Grzegorz Dubicki suggested. Thanks.