Geek Answers Handbook

Multiprocessor and multithreaded processing

I asked a similar question before, but did not get a useful answer, so I will try to make everything more understandable.

I am looking to launch a multi-threaded or preferably multi-processor approach to a specific linux command. If someone is familiar with Picard , I want to run an earlier version in the bam file and simultaneously run a newer version in the same bam file. The idea is to check how much faster the new version will be, and if it gives the same result.

My main problem is that I have no idea how to implement multiprocessing in the Popen team. For instance.

cmd1 = ['nice', 'time', 'java', '-Xmx6G', '-jar', '/comparison/old_picard/MarkDuplicates.jar', 'I=/comparison/old.bam', 'O=/comparison/old_picard/markdups/old.dupsFlagged.bam', 'M=/comparison/old_picard/markdups/old.metrics.txt', 'TMP_DIR=/comparison', 'VALIDATION_STRINGENCY=LENIENT', 'ASSUME_SORTED=true']
cmd2 = ['nice', 'time', 'java', '-Xmx6G', '-jar', '/comparison/new_picard/MarkDuplicates.jar', 'I=/comparison/new.bam', 'O=/comparison/new_picard/markdups/new.dupsFlagged.bam', 'M=/comparison/new_picard/markdups/new.metrics.txt', 'TMP_DIR=/comparison', 'VALIDATION_STRINGENCY=LENIENT', 'ASSUME_SORTED=true']

c1 = subprocess.Popen(cmd1, stdout=subprocess.PIPE)
c2 = subprocess.Popen(cmd2, stdout=subprocess.PIPE)

And then I have a timer function:

def timeit(c):
    past = time.time()
    results = [c.communicate()]
    present = time.time()
    total = present - past
    results.append(total)
    return results

What I want to do:

p = Process(target=timeit, args=(c1,c2))
p.start()
p.join()

, "Popen object not itable". - , , ? , . , c1 c2 , , !

+5
2

.Popen( , ), :

import time
import subprocess
from multiprocessing import Process

cmd1 = ['nice', 'time', 'java', '-Xmx6G', '-jar', '/comparison/old_picard/MarkDuplicates.jar', 'I=/comparison/old.bam', 'O=/comparison/old_picard/markdups/old.dupsFlagged.bam', 'M=/comparison/old_picard/markdups/old.metrics.txt', 'TMP_DIR=/comparison', 'VALIDATION_STRINGENCY=LENIENT', 'ASSUME_SORTED=true']
cmd2 = ['nice', 'time', 'java', '-Xmx6G', '-jar', '/comparison/new_picard/MarkDuplicates.jar', 'I=/comparison/new.bam', 'O=/comparison/new_picard/markdups/new.dupsFlagged.bam', 'M=/comparison/new_picard/markdups/new.metrics.txt', 'TMP_DIR=/comparison', 'VALIDATION_STRINGENCY=LENIENT', 'ASSUME_SORTED=true']

def timeit(cmd):
    print cmd
    past = time.time()
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    results = [p.communicate()]
    present = time.time()
    total = present - past
    results.append(total)
    return results

p1 = Process(target=timeit, args=(cmd1,))
p2 = Process(target=timeit, args=(cmd2,))

for p in (p1, p2):
    p.start()
for p in (p1, p2):
    p.join()

ETA: , @Jordan , . ?

+4

, - :

p1 = Process(target=timeit, args=(c1,))
p2 = Process(target=timeit, args=(c2,))
p1.start()
p2.start()
p1.join()
p2.join()

, ( ?).

, , . , , , , , .

0

More articles:


All Articles