Reading a large file line by line or storing its lines in an array

Question

I have a large file, 100,000 lines. I can read each line and process it, or I can store the lines in an array and then process them. I would prefer to use an array for additional functions, but I'm really worried about the memory usage associated with storing many rows in an array, and if it's worth it.

Answer 1

There are two functions that you should familiarize yourself with.

The first file() , which reads the entire file into an array, with each line being an element of the array. This is useful for shorter files, and probably this is not what you want to use in a 100k file. This function handles its own file management, so you do not need to explicitly open and close the file yourself.

The second is fgets() , which you can use to read the file one line at a time. You can use this for a loop as long as there are more lines to process, and follow your processing line inside the loop. You will need to use fopen() to get the file descriptor, you can track the file pointer yourself to control recovery (i.e. you won’t have to restart processing from scratch if something goes sideways and the script fails), etc. .

Hope this is enough to get you started.

Answer 2

How about a combination of the two? Read 1000 lines in the array, process it, delete the array, then read another 1000, etc. Use monitor memory and adjust how much you read into the array at a time.