Reading a PHP file in a file that is growing in size

Question

Can I use the PHP read function in a remote file whose size is unknown and increasing in size? Here is the scenario:

I am developing a script that downloads video from a third-party site and transcodes the video to MP3 at the same time. This MP3 is then transmitted to the user through a read file.

The query used for the above process is as follows:

wget -q -O- "VideoURLHere" | ffmpeg -i - "Output.mp3" > /dev/null 2>&1 & 

Thus, the file is extracted and encoded at the same time. Now that the above process is running, I start sending mp3 output to the user via readfile. The problem is that the encoding process takes some time and, therefore, depending on the speed of reading the user download files, it reaches the estimated EoF before the entire file is encoded, resulting in the user receiving partial content / incomplete files.

My first attempt to fix this is to apply a speed limit for users to download, but this is not reliable, because the time and speed of encoding is load-dependent, and this still leads to partial downloads.

So, is there a way to implement this system in such a way that I can simultaneously upload files along with the encoding, and also guarantee sending the full file to the end user?

Any help is appreciated.

EDIT: In response to Peter, I use fread (read readfile_chunked):

  <?php function readfile_chunked($filename,$retbytes=true) { $chunksize = 1*(1024*1024); // how many bytes per chunk $totChunk = 0; $buffer = ''; $cnt =0; $handle = fopen($filename, 'rb'); if ($handle === false) { return false; } while (!feof($handle)) { //usleep(120000); //Used to impose an artificial speed limit $buffer = fread($handle, $chunksize); echo $buffer; ob_flush(); flush(); if ($retbytes) { $cnt += strlen($buffer); } } $status = fclose($handle); if ($retbytes && $status) { return $cnt; // return num. bytes delivered like readfile() does. } return $status; } readfile_chunked($linkToMp3); ?> 

This still does not guarantee a full load, because depending on the users' download speed and encoding speed, EOF () may be reached prematurely.

Also in response to Jeztah's comment, I am trying to achieve this without requiring the user to wait ... and this is not an option.

Answer 1

Since you are dealing with threads, you should probably use stream processing functions :). passthru appears, although this will only work if the download | transcode download | transcode .

If it is running externally, see stream_get_contents .

Answer 2

Libevent, as Evert mentioned, seems like a general solution in which you should use the file as a buffer. However, in your case, you can do all this in your script without using the file as a buffer:

 <?php header("Content-Type: audio/mpeg"); passthru("wget -q -O- http://localhost/test.avi | ffmpeg -i - -f mp3 -"); ?> 

Answer 3

I do not think there was a notice that there is new data, with the exception of something like inotify.

I suggest that if you press EOF, you will start polling the file modification time (using clearstatcache() between calls) every 200 ms or so. When you find that the file size is increased, you can open the file again, go to the last position and continue.

Answer 4

I can highly recommend using libevent for such applications.

It is great for such cases.

The PHP documentation for this is a bit sparse, but you can find more compelling examples on the Internet.