What is more expensive on every page view - writing to files or writing to files?

Question

What is the most effective solution when you need to write some data to each page view in an application - should it be written to a file or written to a database?

Or maybe none of them - perhaps you should cache data in memory or a file and sometimes write it to a database (or file system if you use a memory cache)?

Answer 1

If it writes cleanly a small amount of data without further searching, direct file I / O is almost guaranteed to be more efficient. You lose all the advantages of a DBMS: indexing, transaction integrity (indeed, ACID in general), simultaneous access, etc.

You seem to be talking about what constitutes simple logging. If this is the case, and you don’t need to make frequent complex queries on the resulting data, you are probably better off working with direct file I / O if performance is a serious problem. However, be careful of problems with simultaneous recording.

If RDBMS features are desired, you might consider using SQLite, which for better performance will have better performance than most RDBMSs with less overhead due to some advantages (high level of simultaneous access and network access to other machines is a couple of “big” ones ) However, it will not be as fast as direct file I / O in the general case.

Your later mention that this is to track pageviews raises my question: are you increasing the counter, but not registering the pageview data? If so, I highly recommend switching to something like SQLite (something like UPDATE tbl SET counter = counter + 1). You really do not want to understand the time issues associated with this manually - if you do not do it right, you will start to lose accounts with simultaneous access (A reads “100”, B reads “100”, A writes “101”, B writes "101", B should have written 102, but does not know this).

Answer 2

Conceptually, writing to a database is always slower than writing to a file. The database must also be written to a file with additional overhead to get data into the database so that it can write it to a file. Therefore, it should be slower.

However, databases do disk I / O very well, probably better than you. Do not be surprised if you find that a simple file logger is slower than writing it to the database. There are a lot of I / O optimizations in the database, and there are some tricks that you cannot (depending on your web language and environment).

Do not be surprised if the answer changes over time. When your site is small, registering in the database is very fast. As your site grows, the logging table can become a serious pain: it consumes a lot of disk space, makes backups forever and consumes all disk I / O when trying to request it. That is why you should compare both methods yourself. You can then retest in the future when conditions change.

Answer 3

Hitting a database is likely to be more expensive than writing to a file.

If your page views per second are high, and if the data does not have to be immediately available in the database, then writing to a file and periodically loading data into the database will be the best solution.

However, it all depends on the nature of the data that you record on each page view, and how important it is for any business function that it serves.

Answer 4

It greatly depends on your data security needs. If you can afford to lose some data in the event of a failure, then storing the data in memory and periodically writing it to the permanent storage is by far the most efficient way.

Edit: You mentioned pageviews. In this case, I would save the counters in memory and periodically update the database table (for example, every minute or so).

Answer 5

It depends.

And this is true: it depends on the DBMS and / or OS + file system used. In other words: your mileage is changing.

If you just add data, somewhere modern DBMS / OS + file systems should handle it equally well and quickly. Problems arise when you want to change data.

Caching - it also depends on what degree of caching you can afford (you need to have each step log with emergency security and potential savings).

Answer 6

Use a hybrid solution like redis designed for this kind of material