Python Protocol Early Termination

Question

While my script is running, it is possible that at some point an error may occur. In this case, all processes should be properly terminated, an error message should be returned, and the script should exit.

The code that I now seem to not yet meet these requirements. When an error occurs, it is sent to report_error(), and the script ends up hanging in the terminal, and Activity Monitor shows that many Python processes are still running.

Environment

• Mac OS X 10.8.5
• Python 3.3.3

What is the correct way to terminate all processes from anywhere in the script?

#!/usr/bin/env python3
# -*- coding: utf-8 -*-


import sys
from multiprocessing import Pool


# Global variables.

input_files = [
    'test_data_0.csv',
    'test_data_1.csv'
]


def report_error(error):

    # Reports errors then exits script.
    print("Error: {0}".format(error), file=sys.stderr)
    sys.exit(1)

    # What I really want is to report the error, properly terminate all processes,
    # and then exit the script.


def read_file(file):

    try:
        # Read file into list.
    except Exception as error:
        report_error(error)


def check_file(file):

    # Do some error checking on file.
    if error:
        report_error(error)


def job(file):

    # Executed on each item in input_files.

    check_file(file)
    read_file(file)


def main():

    # Sets up a process pool. Defaults to number of cores.
    # Each input gets passed to job and processed in a separate process.
    p = Pool()
    p.map(job, input_files)

    # Closing and joining a pool is important to ensure all resources are freed properly.
    p.close()
    p.join()


if __name__ == '__main__':
    main()

Answer 1

-, sys.exit() map. multiprocessing - , ( , , ).

, . , , , imap_unordered map, , , , imap_unordered:

def report_error(error):

    # Reports errors then exits script.
    print("Error: {0}".format(error), file=sys.stderr)
    raise error # Raise the exception

...

def main():
    p = Pool()
    try:
        list(p.imap_unordered(job, input_files))
    except Exception:
        print("a worker failed, aborting...")
        p.close()
        p.terminate()
    else:
        p.close()
        p.join()

if __name__ == '__main__':
    main()

imap_unordered , . , , . , .