Background processing of videos, what is the professional way in PHP?

Question

I am developing a website for downloading videos, and I have a dilemma: downloaded videos need to be converted to FLV format for display to the visitor, but if I execute the command in a script, the script will hang for 10-15 minutes while FFMPEG converts the video.

I had the idea of ​​inserting a record into the database indicating that the file needs to be processed, then using the cron job set every 5 minutes to select the records from the database that need to be processed, process them, and then update in the database where they are were processed. My concern with this is that there are too many processes running and the server crashing under stress, and does anyone have any solutions for this or a way to improve the process that I have in mind?

---

Ok, now this is what I mean, so the user uploads the video, and the row is added to the database, indicating that the video needs to be processed. The cron task, set every 5 minutes, checks what needs to be processed and what is being processed, say, I would do a maximum of five processes at a time, so the script will check whether any video needs to be processed and how much video is processed if it is less than five , it updates the record indicating that it is being processed, after processing the video, it updates the record indicating that it has been processed, and the cron job starts again, any thoughts?

Answer 1

If you expect a lot of traffic, you should seriously consider a dedicated server.

On a single server, you can use shell_exec along with the UNIX nohup command to get the process PID.

  function run_in_background($Command, $Priority = 0) { if($Priority) $PID = shell_exec("nohup nice -n $Priority $Command 2> /dev/null & echo $!"); else $PID = shell_exec("nohup $Command 2> /dev/null & echo $!"); return($PID); } function is_process_running($PID) { exec("ps $PID", $ProcessState); return(count($ProcessState) >= 2); } 

A full description of this technique is given here: http://nsaunders.wordpress.com/2007/01/12/running-a-background-process-in-php/

You could put the PID list in the MySQL table and then use your cron job every 5 minutes to determine when the video is complete and update the corresponding values ​​in the database.

Answer 2

Gearman is a good solution for this kind of problems, it allows you to instantly send tasks and have any number of employees (which may be on different servers) available for its implementation.

For starters, you can run several workers on the same server, but if you start running problems with the download, you can just start another server with several more workers, so it scales horizontally.

Answer 3

If you use PHP-FPM, you can use fastcgi_finish_request (), as described on PHP.net. FastCGI Process Manager (FPM)

fastcgi_finish_request () is a special function for completing a request and clearing all data while continuing to do something time-consuming (video conversion, processing statistics, etc.);

If you are not using PHP-FPM or want something more advanced, you might consider using a queue manager like Gearman , which is great for the script you are describing. The advantage of using Gearman over starting a process with shell_exec is that you can see how many jobs are running / how many are left and check their statuses. You also simplify scaling, as it is now trivial to add job servers:

 $worker->addServer("10.0.0.1"); 

Answer 4

I like this class (see specific comment) in the PHP manual: http://www.php.net/manual/en/function.exec.php#88704

Basically, it allows you to filter out the background process on * Nix systems. it returns a pid that you can save in the session. When you reload the page to check it, you simply recreate the ForkedProcess class with the saved pid , and you can check its status. If it is completed, the process must be completed.

This does not allow many errors to be verified, but it is incredibly easy.

Answer 5

You can call ffmpeg use system and send the result to /dev/null , this will immediately return the call, effectively processing it in the background.

Answer 6

Create a pair of workflows that will consume messages from the message queue, for example, beanstalkd . Thus, you can control the number of simultaneous tasks (conversions), and also do not pay prices for spawning processes (because the processes continue to work in the background).

I think it would be much faster if you used / encoded C and used Redis as a message queue. Redis has a very nice c client library called Hiredis . I do not think it would be insanely difficult to accomplish.