How to write a file in a large php application (a few questions)

Question

What is the best way to write to files in a large php application. Let's say a lot of records are required per second. What is the best way to do this.

Can I just open the file and add data. Or should I open, block, write and unlock.

What will happen to the file will work, and other data must be written. Will this activity be lost or will it be saved. and if it is saved, it will stop the application.

If you were, thanks for reading!

Answer 1

I have a high-performance multi-threaded application where all threads write (add) to a single log file. Until now, there have been no problems with this, each stream was recorded several times per second, and nothing was lost. I think just adding to a huge file should not be a problem. But if you want to modify existing content, especially with concurrency - I would go with blocking, otherwise a big mess could happen ...

Answer 2

Here is a simple example that emphasizes the danger of concurrent wites:

<?php for($i = 0; $i < 100; $i++) { $pid = pcntl_fork(); //only spawn more children if we're not a child ourselves if(!$pid) break; } $fh = fopen('test.txt', 'a'); //The following is a simple attempt to get multiple threads to start at the same time. $until = round(ceil(time() / 10.0) * 10); echo "Sleeping until $until\n"; time_sleep_until($until); $myPid = posix_getpid(); //create a line starting with pid, followed by 10,000 copies of //a "random" char based on pid. $line = $myPid . str_repeat(chr(ord('A')+$myPid%25), 10000) . "\n"; for($i = 0; $i < 1; $i++) { fwrite($fh, $line); } fclose($fh); echo "done\n"; 

If the addition was safe, you should get a file with 100 lines, all of which are approximately 10,000 characters long and begin with an integer. And sometimes, when you run this script, this is exactly what you get. Sometimes several add-ons will conflict, but this will be distorted.

You can find damaged lines with grep '^[^0-9]' test.txt

This is because the append file is only atomic if :

• You make one call to fwrite ()
• and that fwrite () is less than PIPE_BUF (somewhere around 1-4k)
• and you write a fully compatible POSIX file system.

If you make more than one call for recording while adding a magazine or write more than 4k, all bets are disabled.

Now about whether it really matters: are you okay with a few corrupt lines in your journal under heavy load? Honestly, most of the time this is acceptable, and you can avoid the overhead of locking files.

Answer 3

If concurrency is a problem, you really should use databases.

Answer 4

If you're just writing logs, you might need to take a look at the syslog function, as syslog provides an api. Should you also delegate entries to a dedicated server and perform work in the asynchronous arena?

Answer 5

This is my 2p.

If for a certain reason a unique file is not needed, I would not add everything to a huge file. Instead, I would wrap the file in time and size. To do this, you can define a pair of configuration parameters (wrap_time and wrap_size).

In addition, I would probably introduce some buffering to avoid waiting for the write operation to complete.

PHP is probably not the most adapted language for this kind of operation, but it will still be possible.

Answer 6

Use flock ()

See question

Answer 7

If you just need to add data, PHP should be fine with this, since the file system should take care of the simultaneous additions.