Get an answer from several jobs in Gearman, but interrupt after a timeout

Question

In a nutshell . I want to have a common timeout for a call runTasks()in the Gearman client.

It seems to me that I cannot be the first to want this, but I cannot find an example of how to assemble it.

Here is what I want to achieve:

• In a PHP script, use the Gearman client to run multiple episodes in parallel.
• Each task will produce some search results that the PHP script will need to process.
• Some of the tasks may take some time, but I do not want to wait for the slowest. Instead, after N milliseconds, I want to process the results from all completed tasks and interrupt or ignore those that did not.

Requirements 1 and 2 are quite simple using PHP GearmanClient addTask () and runTasks () , but this is blocked until all provided tasks are completed, therefore it does not meet requirements 3.

Here are a few approaches I've tried so far:

• The timeout with setTimeout () measures the connection idle time, which is not of interest to me.
• Using background jobs or tasks is not possible to retrieve data returned to workers. There are several questions that already cover this: 1 2
• addTaskStatus() - , , , . " ", , .
• GEARMAN_CLIENT_NON_BLOCKING, , , runTasks() setTaskBackground() setTask().

, , , , , Gearman RabbitMQ.

Answer 1

, , .

, runTasks() - , :

• , , addTask().
• setCompleteCallback(), , .
• - - setTimeout(), .
• runTasks() , , , -. , " N , , X " ..

, - PHP, @.

:

// How long should we wait each time around the polling loop if nothing happens
define('LOOP_WAIT_MS', 100);
// How long in total should we wait for responses before giving up
define('TOTAL_TIMEOUT_MS', 5000);

$client= new GearmanClient();
$client->addServer();

// This will fire as each job completes.
// In real code, this would save the data for later processing,
// as well as tracking which jobs were completed, tracked here with a simple counter.
$client->setCompleteCallback(function(GearmanTask $task) use (&$pending) {
        $pending--;
        echo "Complete!\n";
        echo $task->data();
});

// This array can be used to track the tasks created. This example just counts them.
$tasks = [];
// Sample tasks; the workers sleep for specified number of seconds before returning some data.
$tasks[] = $client->addTask('wait', '2');
$tasks[] = $client->addTask('wait', '2');
$tasks[] = $client->addTask('wait', '2');
$tasks[] = $client->addTask('wait', '2');
$tasks[] = $client->addTask('wait', '2');
$tasks[] = $client->addTask('wait', '2');

$pending = count($tasks);

// This is the key polling loop; runTasks() here acts as "wait for a notification from the server"
$client->setTimeout(LOOP_WAIT_MS);
$start = microtime(true);
do {
        // This will abort with a PHP Warning if no data is received in LOOP_WAIT_MS milliseconds
        // We ignore the warning, and try again, unless we've reached our overall time limit
        @$client->runTasks();
} while (
        // Exit the loop once we run out of time
        microtime(true) - $start < TOTAL_TIMEOUT_MS / 1000
        // Additional loop exit if all tasks have been completed
        // This counter is decremented in the complete callback
        && $pending > 0
);

echo "Finished with $pending tasks unprocessed.\n";