Performing functions in parallel

Question

I have a function that should loop around 20K lines from an array and apply an external script to each. This is a slow process since PHP is waiting for the script to execute before continuing with the next line.

To speed up this process, I thought about running the function in different parts at the same time. So, for example, lines from 0 to 2000 as one function, from 2001 to 4000 to another, etc. How can I do it neatly? I could create different cron jobs, one for each function with different parameters: myFunction(0, 2000) , then another cron job with myFunction(2001, 4000) , etc., but that doesn't seem too clean. What a good way to do this?

Answer 1

If you want to do parallel tasks in PHP, I would consider using Gearman . Another approach would be to use pcntl_fork () , but I would prefer the actual workers when it was based on a job.

Answer 2

The only waiting time you suffer is receiving data and processing the data. Data processing is actually completely blocked (you just have to wait). You probably won’t get any benefits from increasing the number of processes to the number of cores that you have. Basically, I think this means that the number of processes is small, so scheduling 2-8 processes does not seem so disgusting. If you are concerned about the inability to process data while receiving data, you can theoretically get your data from the database in small blocks, and then distribute the processing load between several processes, one for each core.

I think that I am more attached to the approach of forking child processes to actually start processing threads. In the comments on the pcntl_fork document page, there is a vivid demo demonstrating the implementation of the job daemon class

http://php.net/manual/en/function.pcntl-fork.php

 <?php declare(ticks=1); //A very basic job daemon that you can extend to your needs. class JobDaemon{ public $maxProcesses = 25; protected $jobsStarted = 0; protected $currentJobs = array(); protected $signalQueue=array(); protected $parentPID; public function __construct(){ echo "constructed \n"; $this->parentPID = getmypid(); pcntl_signal(SIGCHLD, array($this, "childSignalHandler")); } /** * Run the Daemon */ public function run(){ echo "Running \n"; for($i=0; $i<10000; $i++){ $jobID = rand(0,10000000000000); while(count($this->currentJobs) >= $this->maxProcesses){ echo "Maximum children allowed, waiting...\n"; sleep(1); } $launched = $this->launchJob($jobID); } //Wait for child processes to finish before exiting here while(count($this->currentJobs)){ echo "Waiting for current jobs to finish... \n"; sleep(1); } } /** * Launch a job from the job queue */ protected function launchJob($jobID){ $pid = pcntl_fork(); if($pid == -1){ //Problem launching the job error_log('Could not launch new job, exiting'); return false; } else if ($pid){ // Parent process // Sometimes you can receive a signal to the childSignalHandler function before this code executes if // the child script executes quickly enough! // $this->currentJobs[$pid] = $jobID; // In the event that a signal for this pid was caught before we get here, it will be in our signalQueue array // So let go ahead and process it now as if we'd just received the signal if(isset($this->signalQueue[$pid])){ echo "found $pid in the signal queue, processing it now \n"; $this->childSignalHandler(SIGCHLD, $pid, $this->signalQueue[$pid]); unset($this->signalQueue[$pid]); } } else{ //Forked child, do your deeds.... $exitStatus = 0; //Error code if you need to or whatever echo "Doing something fun in pid ".getmypid()."\n"; exit($exitStatus); } return true; } public function childSignalHandler($signo, $pid=null, $status=null){ //If no pid is provided, that means we're getting the signal from the system. Let figure out //which child process ended if(!$pid){ $pid = pcntl_waitpid(-1, $status, WNOHANG); } //Make sure we get all of the exited children while($pid > 0){ if($pid && isset($this->currentJobs[$pid])){ $exitCode = pcntl_wexitstatus($status); if($exitCode != 0){ echo "$pid exited with status ".$exitCode."\n"; } unset($this->currentJobs[$pid]); } else if($pid){ //Oh no, our job has finished before this parent process could even note that it had been launched! //Let make note of it and handle it when the parent process is ready for it echo "..... Adding $pid to the signal queue ..... \n"; $this->signalQueue[$pid] = $status; } $pid = pcntl_waitpid(-1, $status, WNOHANG); } return true; } } 

Answer 3

you can use "pthreads"

very easy to install and works great on windows

download here → http://windows.php.net/downloads/pecl/releases/pthreads/2.0.4/

Extract the zip file and then

• move the file "php_pthreads.dll" to the php \ ext \ folder.

• move the pthreadVC2.dll file to the php \ directory.

then add this line to your "php.ini" file:

 extension=php_pthreads.dll 

save file.

you just did :-)

Now you can see an example of use:

 class ChildThread extends Thread { public $data; public function run() { /* Do some expensive work */ $this->data = 'result of expensive work'; } } $thread = new ChildThread(); if ($thread->start()) { /* * Do some expensive work, while already doing other * work in the child thread. */ // wait until thread is finished $thread->join(); // we can now even access $thread->data } 

for more information on PTHREADS read php docs here:

PHP DOCS PTHREADS

• If you use WAMP like me, then you should add 'pthreadVC2.dll' to \ WAMP \ Bin \ Apache \ ApacheX.XX \ Bin and also edit the file "php.ini" (same path) and add the same line, as before

  extension = php_pthreads.dll

LUCK!

Answer 4

Take a look at pcntl_fork . This allows you to create child processes that can then do the separate work that you need.

Answer 5

Not sure if the solution is for your situation, but you can redirect the output of system calls to a file, so PHP will not wait for the program to finish. Although this can lead to an overload of your server.

http://www.php.net/manual/en/function.exec.php - If the program starts with this function so that it continues to work in the background, the output of the program should be redirected to a file or other output stream. Otherwise, PHP will hang until the program terminates.