End multithreaded python program

Question

How to respond to a multi-threaded response in Python to a key event of Ctrl + C?

Edit: The code is as follows:

import threading current = 0 class MyThread(threading.Thread): def __init__(self, total): threading.Thread.__init__(self) self.total = total def stop(self): self._Thread__stop() def run(self): global current while current<self.total: lock = threading.Lock() lock.acquire() current+=1 lock.release() print current if __name__=='__main__': threads = [] thread_count = 10 total = 10000 for i in range(0, thread_count): t = MyThread(total) t.setDaemon(True) threads.append(t) for i in range(0, thread_count): threads[i].start() 

I tried removing join () for all threads, but it still doesn't work. Is this because the lock segment inside each thread execution routine ()?

Edit: The above code should work, but it always breaks when the current variable is in the range of 5000-6000 and due to errors below

 Exception in thread Thread-4 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner File "test.py", line 20, in run <type 'exceptions.TypeError'>: unsupported operand type(s) for +=: 'NoneType' and 'int' Exception in thread Thread-2 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner File "test.py", line 22, in run 

Answer 1

Make each thread, except the main thread, a daemon ( t.daemon = True in 2.6 or better, t.setDaemon(True) in 2.6 or less, for each stream object t , before you start it). Thus, when the main thread receives KeyboardInterrupt, if it does not catch it or does not catch it, but decides to stop it anyway, the whole process will end. See documents .

edit : just seeing the OP code (not originally published) and the statement that “this does not work”, it looks like I should add ...:

Of course, if you want your main thread to remain responsive (for example, to control C), do not do this in blocking calls, such as join another thread, especially not completely useless blocking calls, such as join ing daemon threads. For example, just change the final loop in the main thread from the current one (without loss and damage):

 for i in range(0, thread_count): threads[i].join() 

to something more reasonable, for example:

 while threading.active_count() > 0: time.sleep(0.1) 

if your main task has nothing better than for all threads that should complete independently, or to gain control of C (or another signal).

Of course, there are many other applicable patterns, if you prefer that your threads do not interrupt suddenly (as there may be demonic threads) - if they also do not get bogged down permanently in unconditionally blocking calls, deadlocks, etc .; -.)

Answer 2

There are two main ways: one is clean and simple.

The clean way is to catch KeyboardInterrupt in your main thread and set a flag that your background threads can check to let them know what to exit; here's a simple / slightly dirty version using global:

 exitapp = False if __name__ == '__main__': try: main() except KeyboardInterrupt: exitapp = True raise def threadCode(...): while not exitapp: # do work here, watch for exitapp to be True 

A minor but easy way is to catch KeyboardInterrupt and call os._exit (), which will immediately terminate all threads.

Answer 3

A worker may be useful to you:

 #!/usr/bin/env python import sys, time from threading import * from collections import deque class Worker(object): def __init__(self, concurrent=1): self.concurrent = concurrent self.queue = deque([]) self.threads = [] self.keep_interrupt = False def _retain_threads(self): while len(self.threads) < self.concurrent: t = Thread(target=self._run, args=[self]) t.setDaemon(True) t.start() self.threads.append(t) def _run(self, *args): while self.queue and not self.keep_interrupt: func, args, kargs = self.queue.popleft() func(*args, **kargs) def add_task(self, func, *args, **kargs): self.queue.append((func, args, kargs)) def start(self, block=False): self._retain_threads() if block: try: while self.threads: self.threads = [t.join(1) or t for t in self.threads if t.isAlive()] if self.queue: self._retain_threads() except KeyboardInterrupt: self.keep_interrupt = True print "alive threads: %d; outstanding tasks: %d" % (len(self.threads), len(self.queue)) print "terminating..." # example print "starting..." worker = Worker(concurrent=50) def do_work(): print "item %d done." % len(items) time.sleep(3) def main(): for i in xrange(1000): worker.add_task(do_work) worker.start(True) main() print "done." # to keep shell alive sys.stdin.readlines() 

Answer 4

I would rather go with the code suggested in this blog post :

 def main(args): threads = [] for i in range(10): t = Worker() threads.append(t) t.start() while len(threads) > 0: try: # Join all threads using a timeout so it doesn't block # Filter out threads which have been joined or are None threads = [t.join(1000) for t in threads if t is not None and t.isAlive()] except KeyboardInterrupt: print "Ctrl-c received! Sending kill to threads..." for t in threads: t.kill_received = True 

What I changed is t.join from t.join (1) to t.join (1000) . The actual number of seconds does not matter if you do not specify a timeout number, the main thread will remain responsive to Ctrl + C. With the exception of KeyboardInterrupt , signal processing is more explicit.

Answer 5

If you create Thread the same way - myThread = Thread(target = function) - and then do myThread.start(); myThread.join() myThread.start(); myThread.join() . When CTRL-C is triggered, the main thread does not exit because it is waiting to block the call to myThread.join() . To fix this, simply set a timeout on the .join () call. The timeout can be as long as you want it. If you want it to wait indefinitely, just enter a very long timeout, for example 99999. It is also good to make myThread.daemon = True so that all threads exit when the main thread (non-daemon) exits.

Answer 6

You can always set your threads to daemon threads, for example:

 t.daemon = True t.start() 

And whenever the main thread dies, all threads will die with it.

http://www.regexprn.com/2010/05/killing-multithreaded-python-programs.html

Answer 7

 thread1 = threading.Thread(target=your_procedure, args = (arg_1, arg_2)) try: thread1.setDaemon(True) # very important thread1.start() except (KeyboardInterrupt, SystemExit): cleanup_stop_thread(); sys.exit() 

If you want to kill a thread, just use:

 thread1.join(0)