Geek Answers Handbook

Completing the spark step in aws

I want to set up a series of spark steps on an EMP spark cluster and stop the current step if it takes too long. However, when I ssh in the master node and run the hasoop -list work tasks, the master node seems to think there are no jobs. I do not want to break the cluster, because it will force me to buy a whole new hour of any cluster in which I work. Can someone help me stop the spark phase in EMR without completing the entire cluster?

+7
amazon-web-services hadoop emr apache-spark
source share
2 answers

This is easy:

yarn application -kill [application id]

you can list your running applications with

 yarn application -list
+16
source share

You can kill the application from the resource manager (in the links at the top right under the cluster status). In the resource manager, click the application you want to kill, and on the application page there is a small “kill” label (top left) that you can click to kill the application.

Obviously, you can also use SSH, but in this way, I think it’s faster and easier for some users.

+3
source share

More articles:


All Articles