How to handle a video encoding queue during multiple videos?

Question

I am working on a video streaming site where users can upload videos to the site (multiple videos at once using the jQuery downloadable plugin).

Now I am faced with the issue of encoding videos in FLV for streaming them on the Internet.

When should the video encoding process take place? If this happens immediately after the download is complete (for example, redirecting the user to the successful download page, and then starting the encoding in the background using the exec command for ffmpeg?) However, using this approach, how to determine if the encoding has completed successfully? What if users upload corrupted video and ffmpeg cannot encode it? How can I handle this in PHP?

How do I order a video encoding, since several users can upload a video? Does FFMpeg have its own encoding queue?

I also read about transmission and message queue options, such as redis and AMQP, in another related SO thread. Are these one of the potential solutions?

I would really appreciate it if someone could answer my questions.

Answer 1

You should use software called gearman . This is a job server, and you can call functions using PHP. You can transfer the process to the background, and it automatically processes the queue. Many processes can also run in parallel.

I used it and it is very easy to install and use.

In your use case

Wrap the ffmpeg process in a php file with exec. Save the uploaded file to db and give it an id. Call the encode function after a user uploads file. As soon as the encode function starts, update the db to reflect that the file was "picked". First run the ffmpeg info on the file to see if it is fine. Then encode the file and after it is done, update db flag to "done". 

After the process is complete, you can call another function to send email, tweet, etc. to the user that the encoding is complete.

Answer 2

Since encoding may take some time, you may want to add input files to a folder and add an entry to the database. Then you have a script that runs either continuously or every x minutes that convert the pending videos from the database to the FLV format.

To queue them, you need to create a custom script that re-runs FFMpeg for each file.

Answer 3

You can use the cron job on the server or use something like beanstalkd (or gearman, as mentioned in other answers to this question).

Answer 4

FFMpeg is just a command line utility. It does not have a queue, and you will need to create your own queue system to perform asynchronous work.

For PHP queues, I had good experience with the beanstalkd server using the pheanstalk PHP client. Your script loading should then insert the elements into the encoding queue and return a response to the user that the video will be processed shortly. Your employees should receive items from the queue and run FFMpeg on them. You can read the FFMpeg status code to see if the encoding has completed successfully.