PHP memory differs from server to server

Question

I have a massive PHP script.

So much so that I had to do

ini_set('memory_limit', '3000M'); set_time_limit (0); 

It works fine on one server, but on the other I get: Not enough memory (allocated 1653342208) (tried to allocate 71 bytes) in / home / writeabo / public _html / propturk / feedgenerator / simple_html_dom.php on line 848

Both are in the same package from the same host, but from different servers.

Above problem solved new problem below for reward

Update. The script is so large that it loads site and parser data from 252 pages, including more than 60,000 images, which makes two copies. Since then I have broken it into pieces.

I now have one more problem. when I write an image from an external site to the server as follows:

 try { $imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs $h = fopen($writeTo,'w'); fwrite($h,$imgcont); fclose($h); } catch(Exception $e) { $error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />"; } 

Suddenly he goes to a page with an internal internal server error, and I need to do it again, after which it works, because the files are only copied if they do not already exist. Is there anyway I can get a 500 response code and send it back to the url so that it reappears? How should all this be an automated process?

Answer 1

Thanks for all your input. I separated everything by the time I wrote this question, so the finder fired the image capture, etc.

I took a decision on board to share the number of images, and it also helped.

I also added a try, bypassing the read file.

This was only called from the browser during testing, but now that it is all running, it will be a cron job.

Thanks to Swish and Benubird for your detailed and educational answers. Unfortunately, I had no cooperation with the developers on the backend, where the images come from (a long and complicated story).

In any case, everything is fine now, so thanks. (Swish, as you call the script from the command line, do I know about this field?)

Answer 2

If this is memory related, I would personally use copy() rather than file_get_contents() . It supports wrapping files in the same way, and I see no advantage in loading the entire file in memory, just to write it back to the file system.

Otherwise, your error_log may give you more information on why 500 occurs.

Answer 3

There are three parties involved:

• Remote - server (s) containing the images you are after
• Server - the computer on which your PHP script is running
• The client is your home computer if you use a script from a web browser or the same computer as the server if you use it from Cron.

Is the 500 error that you see generated by 'Remote' and visible by the 'server' (i.e. images are temporarily unavailable); Or it is generated by the "Server" and considered by the "Client" (i.e. there is a problem with your script).

If it is generated by 'Remote', see Ali's answer for retrying.

If it is created by your script on 'Server', you need to determine exactly what the error is. Php error logs should provide you with more information. I can think of two possible reasons:

• Reaching PHP time limit . PHP will spend a certain amount of time before returning a 500 error. You can either set this higher value or regularly reset the timer with set_time_limit () , but this will not work if your server is configured in safe mode.

• Reaching PHP memory limit . It seems you have already activated this, but make sure that you script still did not eat a lot of memory. Consider the output of debugging data (only possible if you set $ config ['debug_mode'] = true or something else). I would suggest:

   try { echo 'Getting '.$va.'...'; $imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs $h = fopen($writeTo,'w'); fwrite($h,$imgcont); fclose($h); echo 'saved. Memory usage: '.(memory_get_usage() / (1024 * 1024)).' <br />'; unset($imgcont); } catch(Exception $e) { $error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />"; } 

I also added a line to remove the image from memory, if PHP does not do it right myself (theoretically, this line is not needed).

You can avoid both problems by making your script process fewer images at the same time and calling it regularly - either using Cron on the server (an ideal solution, although these are not all shared web hosts), or some software on your desktop computer. If you do this, make sure you think what happens if two copies of the script work at the same time - will they simultaneously receive the same image?

Answer 4

So it looks like you are running this process through a web browser. I assume that you can get 500 errors from Apache time, anyway, after a certain period of time or the process dies or something scared. I suggest you do one of the following:

A) Move the image loading to the background process, you can run a script bypass in a browser that will write the URLs of the images that will be loaded in db or something else, and another script will be run through cron and get all the images. You can also use this script function in batches of 100 or so at the same time to reduce memory consumption.

B) Call the script directly from the command line (this is really the preferred method for something similar anyway, and you should still separate the image getting to another script)

C) If for some reason the command line is not a parameter, load your browser script, tap the file and run cron, which runs every minute and looks for the file. Then it runs your script, you can write the output to a file so that you can later check or send an email after it is completed.

Answer 5

Anyway, can I get a 500 response code and send it back to the url so that it reappears? How should all this be an automated process?

Here is a simple version of how I will do this:

 function getImage($va, $writeTo, $retries = 3) { while ($retries > 0) { if ($imgcont = file_get_contents($va)) { file_put_contents($writeTo, $imgcont); return true; } $retries--; } return false; } 

This does not create the file unless we receive our image file and by default repeat three times. Of course, you will need to add any exception handling, error checking, etc.

Answer 6

I would definitely stop using file_get_contents () and write files to chunks, for example:

  $read = fopen($url, 'rb'); $write = fope($local, 'wb'); $chunk = 8096; while (!feof($read)) { fwrite($write, fread($read, $chunk)); } fclose($fp); 

It will be better for your server and hopefully solve your 500 problems. As for catching a 500 error, this is simply not possible. This is a fatal error caused by your script and written by the client by the web server.

Answer 7

I'm with Swish, in fact this is not the task for which PHP is intended - you are much better off using some kind of script on the server side.

Is there anyway I can receive the 500 response code and send it back it to the url to make it go again?

Have you considered using another library? Getting files from an external server seems to me more like working for curl or ftp than file_get_content & etc. If the error is external, and you use curl, you can detect a return code of 500 and handle it accordingly without crashing. If not, then perhaps you should split your program into two files — one that extracts one file / image, and the other that uses curl to recall the first. If the 500 error does not mean that all php runtime failure will fail, you can detect the failure and handle it.

Something like this pseudo code:

file1.php:

 foreach(list_of_files as filename){ do { x = call_curl('file2.php', filename); } while(x == 500); } 

file2.php:

  filename=$_GET['filename']; results = use_curl_to_get_page(filename); echo results;