Reading and analyzing the contents of a very large file

Question

I am trying to parse a ~ 1 GB tab delimited file.

Where I run the script, I get:

Fatal error: Allowed memory size of 1895825408 bytes exhausted (tried to allocate 1029206974 bytes) ... 

My script at the moment is simple:

 $file = file_get_contents('allCountries.txt') ; $file = str_replace(array("\r\n", "\t"), array("[NEW*LINE]", "[tAbul*Ator]"), $file) ; 

I set the memory limit in php.ini to -1, which then gives me:

 Fatal error: Out of memory (allocated 1029963776) (tried to allocate 1029206974 bytes) 

Is it possible to partially open the file and then go to the next part so that less memory is used up at a time?

Answer 1

Yes, you can read it line by line:

 $handle = @fopen("/tmp/inputfile.txt", "r"); if ($handle) { while (($buffer = fgets($handle, 4096)) !== false) { echo $buffer; } fclose($handle); } 

Answer 2

You need to use blocks to read the file. Check the answer to this question. stack overflow

You can also try using this for smaller files.

 ini_set('memory_limit', '32M'); //max size 32m 

Answer 3

Are you sure it fopen not working and not your script timeout? The default is usually around 30 seconds or so, and if your file takes longer than it does to read, it may cause a shutdown.

Another thing to consider may be memory limitation on your script - reading a file into an array can work around this, so check your error log for memory warnings.

If none of the above problems is your problem, you can study fgets to read the file in turn as you go.

 $handle = fopen("/tmp/uploadfile.txt", "r") or die("Couldn't get handle"); if ($handle) { while (!feof($handle)) { $buffer = fgets($handle, 4096); // Process buffer here.. } fclose($handle); } 

Edit

PHP doesn't seem to throw an error, it just returns false.

Is the path to $rawfile regarding where the script works? Perhaps try setting the absolute path here for the file name.

Answer 4

Yes, use fopen and fread / fgets for this:

http://www.php.net/manual/en/function.fread.php

 string fread ( resource $handle , int $length ) 

Set $ ​​length to how many files you want to read. The $ knob keeps the position for new readings, and with fseek you can also set the position later ....