Geek Answers Handbook

Kill one spark task

I have a very long Spark job, of which a small number of tasks are currently stalled. Is there any way to kill these running tasks from the node driver?

For permission reasons, I can log in, but not kill the jobs on the subordinate nodes, so I'm looking for a way to do this only from the node driver. Please note that I do not want to kill all the work of Spark - only one or two tasks with a stop.

If this helps, I use Mesos and have access to the web interface, but there is no way to kill the task.

+7
distributed-computing apache-spark mesos
source share
1 answer

No, not at all.

You cannot kill an individual spark task manually, however you can use spark speculation to automatically determine which tasks take too long and restart them proactively.

If you want to do this, set spark.speculation to true and [if you dare] change the configuration settings for spark.speculation.interval , spark.speculation.multiplier , spark.speculation.quantile .

Related documents: http://spark.apache.org/docs/latest/configuration.html#viewing-spark-properties

Associated SO: How to work with too long tasks (compared to others in the task) in the yarn-client?

+5
source share

More articles:


All Articles