Geek Answers Handbook

What is the best way to perform tasks in parallel in Ksh and Perl?

I have this big C ++ project that I need to build on a platform that does not have parallel make (e.g. make -j on Linux). The server has 6 processors, and I want to do a parallel build manually.

I can create a list of tasks, for example, about 300 object files. I use the Makefile for dependency checking and incremental builds:

make -f makefile obj1.o

make -f makefile obj2.o

make -f Makefile obj3.o ...

How to perform these tasks in parallel with no more than 6 tasks performed simultaneously using Ksh and Perl? (Java or Python are not available :-()

+4
source share
5 answers

In Perl, you should look at Parallel :: ForkManager . You can do something like this:

my @make_obj = qw( obj1.o obj2.o obj3.o ... ); my $fm = $pm = new Parallel::ForkManager(6); foreach my $obj (@make_obj) { $fm->start and next; system("make -f Makefile $make_obj"); $fm->finish(); }
+5
source

An alternative to forking is to run each make in its thread.

 use threads; my $max_threads = 5; my @targets = qw(obj1.o obj2.o obj3.o ...); while(@targets) { my $num_threads = threads->list(threads::running); if( $num_threads < $max_threads ) { my $target = shift @targets; threads->create(sub { return system "make $target" }); } }

Unfortunately, I wave my hand around two bits. Firstly, making the loop wait until the stream ends. I believe this is achieved using threads :: shared cond_wait () and a semaphore variable.

The second is to get the return value from make, so you know that something failed, and stop the build. To do this, you will need to join () to each thread to get the result of system (), the process termination code.

Sorry for the hasty reply. We hope the community fills the rest.

+4
source

Does gnu on the HPUX make the -j flag?

http://hpux.connect.org.uk/hppd/hpux/Gnu/make-3.81/

+3
source

Using GNU Parallel, you can write:

 parallel -j6 make -f Makefile obj{}.o ::: {1..500}

10 second installation:

 (wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

More details: http://www.gnu.org/software/parallel/parallel_tutorial.html https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

+1
source

If the OS properly handles interprocess communication and scheduling, you should simply be able to throw all the components into background processes if there are no interdependencies.

If they are all independent, I would issue a series of commands like this:

 make -f Makefile obj1.o & make -f Makefile obj2.o & ... make -f Makefile objn.o &

With dependencies, write them with a double ampersand ( && ) so that the dependent element does not start until its parent element completes.

I understand that this is not exactly your solution, but how would I attack the problem :)

-1
source

More articles:


All Articles