Creating many processes with PHP to process data.

Question

I have a queue (Amazon SQS) of data that needs to be processed, and I would like to do this using several processes (in PHP).

I want child workers to do something like this (pseduoish code):

while(true) { $array = $queue->fetchNItems(10); // get 10 items if(!count($array)) killProcess(); foreach($array as $item) { ... // process the item $queue->remove($item); } sleep(2); } 

I always need one child process, but if necessary, I want (fork?) To create a child process so that it can process the queue faster.

Can someone help me with the rude PHP skeleton of what I need, or point me in the right direction?

I think I need to take a look at http://php.net/manual/en/function.pcntl-fork.php , but I'm not sure how I can use this to control multiple processes.

Answer 1

When you fork the process. you duplicate this process. In other words, a fork contains all the source processes (including files)

So, how do you know if you are a parent or a branched process?

The example from the linked page shows this quite clearly.

 <?php $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); //Protect against Zombie children } else { // we are the child } ?> 

To expand it to the desired

 <?php $pid = pcntl_fork(); if ($pid == -1) { die('could not fork'); } else if ($pid) { // we are the parent pcntl_wait($status); //Protect against Zombie children } else { // we are the child while(true) { $array = $queue->fetchNItems(10); // get 10 items if(!count($array)) { exit(); } foreach($array as $item) { ... // process the item $queue->remove($item); } sleep(2); } } ?> 

This will create a forked process (waste in this case) uses a loop to create multiple processes. when the child has finished exit, he will kill the child process. and pcntl_wait () will return so the parent can continue. I'm not sure about php, but if the parent process dies or exits, it will kill the child process, even if the child is not finished yet. hence pcntl_wait. A more sophisticated system is required if you create multiple children.

maybe, not forking, should you look at the range of exec functions?

Warning.

The forking process can be caused by problems when the database descriptors are closed when they exit the child, etc. You can also kill a server with many processes if something goes wrong. spend a lot of time on games and testing and reading.

DC

Answer 2

I know this is an old thread, but it looks like it can use a more complete answer. This is how I usually run several processes in PHP.

A word of caution: PHP was supposed to die. Meaning, the language was designed for a few seconds, and then exit. Although garbage cleaning in PHP has come a long way, be careful. Control your processes for unexpected memory usage or other weird things. Watch everything like a hawk until you install it and forget about it, and even then, still check the processes once in a while or automatically let them know that something is wrong.

As I typed this, it seemed like a good idea to pat him on github too.

When you are ready to run the program, I recommend doing the -f tail in the log to see the result.

 <?php /* * date: 27-sep-2015 * auth: robert smith * info: run a php daemon process * lic : MIT License (see LICENSE.txt for details) */ $pwd = realpath(""); $daemon = array( "log" => $pwd."/service.log", "errorLog" => $pwd."/service.error.log", "pid_file" => $pwd."/", "pid" => "", "stdout" => NULL, "stderr" => NULL, "callback" => array("myProcessA", "myProcessB") ); /* * main (spawn new process) */ foreach ($daemon["callback"] as $k => &$v) { $pid = pcntl_fork(); if ($pid < 0) exit("fork failed: unable to fork\n"); if ($pid == 0) spawnChores($daemon, $v); } exit("fork succeeded, spawning process\n"); /* * end main */ /* * functions */ function spawnChores(&$daemon, &$callback) { // become own session $sid = posix_setsid(); if ($sid < 0) exit("fork failed: unable to become a session leader\n"); // set working directory as root (so files & dirs are not locked because of process) chdir("/"); // close open parent file descriptors system STDIN, STDOUT, STDERR fclose(STDIN); fclose(STDOUT); fclose(STDERR); // setup custom file descriptors $daemon["stdout"] = fopen($daemon["log"], "ab"); $daemon["stderr"] = fopen($daemon["errorLog"], "ab"); // publish pid $daemon["pid"] = sprintf("%d", getmypid()); file_put_contents($daemon["pid_file"].$callback.".pid", $daemon["pid"]."\n"); // publish start message to log fprintf($daemon["stdout"], "%s daemon %s started with pid %s\n", date("YMd H:i:s"), $callback, $daemon["pid"]); call_user_func($callback, $daemon); // publish finish message to log fprintf($daemon["stdout"], "%s daemon %s terminated with pid %s\n", date("YMd H:i:s"), $callback, $daemon["pid"]); exit(0); } function myProcessA(&$daemon) { $run_for_seconds = 30; for($i=0; $i<$run_for_seconds; $i++) { fprintf($daemon["stdout"], "Just being a process, %s, for %d more seconds\n", __FUNCTION__, $run_for_seconds - $i); sleep(1); } } function myProcessB(&$daemon) { $run_for_seconds = 30; for($i=0; $i<$run_for_seconds; $i++) { fprintf($daemon["stdout"], "Just being a process, %s, for %d / %d seconds\n", __FUNCTION__, $i, $run_for_seconds); sleep(1); } } ?>