How to stop using Spark Streaming app? Gracefully?

Question

How to stop sparking? My spark flow work continuously. I want to stop gracefully.

I saw below the option to disable the streaming application.

sparkConf.set("spark.streaming.stopGracefullyOnShutdown","true") 

Spark Configuration: Available Properties

But how can I update this setting in a running application?

Answer 1

Check out this blog post . This is a “wonderful” way to gracefully terminate the streaming job I encountered.

How to transmit a shutdown signal:

Now we know how to provide a graceful shutdown in a spark stream. But how can we transmit a shutdown signal for sparking. One naive option is to use the CTRL + C command on the terminal screen, where we run the driver program, but obviously this is not a good option. One of the solutions I use is the grep process of driving a spark stream and sending a SIGTERM signal. When the driver receives this signal, it initiates a graceful shutdown of the application. We can write a command, as shown below, in some shell script and run the script to send a shutdown signal:

ps -ef | greek spark | grep | awk '{print $ 2}' | xargs kill -SIGTERM

eg. ps -ef | greek spark | grep DataPipelineStreamDriver | awk '{print $ 2}' | xargs kill -SIGTERM