Geek Answers Handbook

Python permutation threads

I created permutations using the itertools.permutations function in python. The problem is that the result is very large, and I would like to go through it in several threads, but I don’t know how to do it, here is what I still have:

perms = itertools.permutations('1234', r=4) #I would like to iterate through 'perms' with multiple threads for perm in perms: print perm
+6
source share
4 answers

If the work you want to do with elements from the permutation generator has processor intensity, you probably want to use processes, not threads. CPython Global Interpreter Lock (GIL) makes multithreading of limited value when working with the CPU.

Instead, use the multiprocessing module Pool , for example:

 import multiprocessing import itertools def do_stuff(perm): # whatever return list(reversed(perm)) if __name__ == "__main__": with multiprocessing.Pool() as pool: # default is optimal number of processes results = pool.map(do_stuff, itertools.permutations('1234', r=4)) # do stuff with results

Please note that if you iterate over the results (instead of doing something with it as a list), you can use imap instead of map to get an iterator that you can use to work with the results since they are produced from work processes. If it doesn't matter in which order the elements are returned, you can use imap_unordered to (I think) save some memory.

The if __name__ is "__main__" template if __name__ is "__main__" required on Windows, where the multiprocessing module must work with OS restrictions (no fork ).

+4
source

Assuming your processing function is f (x) you want to do

 from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': pool = Pool(processes=4) # start 4 worker processes perms = itertools.permutations('1234', r=4) for r in pool.map(f, perms): print (r)

In fact, using threads will not execute processes in parallel unless it is related to IO. If it is connected to the CPU, and you have a quad core, then this is the way to go. If you do not have multi-core processors, and this is related to the CPU, then I am afraid that a parallel connection will not improve your current situation.

+1
source

Divide the perm number index between threads, then use this function to generate a variable from its index in each thread, rather than generate all perms and split them between threads.

+1
source

The Python futures module makes it easy to split work between threads. In this example, 4 threads will be used, but you can change them to suit your needs.

 from concurrent import futures def thread_process(perm): #do something with futures.ThreadPoolExecutor(max_workers=4) as executor: for perm in perms: executor.submit(thread_process, perm)
0
source

More articles:


All Articles