NoSQL or RDBMS for audit data

I know that similar questions were asked in the subject, but so far I have not seen anyone who would completely contain all my requests.

I would say that I only have experience with RDBMSs, so I'm sorry I have nothing to do with NoSQL.

I am creating a database in which a large number of audit logs will be stored (about 1 TB).

I use it for:

  • Fast data recording (a huge number of audit logs are recorded all the time)

  • Search - search by audit data (search actions performed by a specific user at a specific time or a specific action ... the database must support a quick search of any of the columns)

  • Analytics and reporting - generate daily, weekly, monthly data reports (they are predetermined at the moment .. if they are more dynamic, does this affect the decision that I have to choose?)

Reliability (failure support or any similar function), Scalability (if I grow above 1 TB to 2 TB, 10 TB or 100 TB - can any of the solutions not support this amount of data?) And, of course, Performance ( in the use cases that I indicated) are very important to me.

I know RDBMS, and it will be my easy way to get started, but I'm really worried that after a while the DB just does not keep up with the pace.

My question is, should I choose a solution for RDBMS or NoSQL and why? If the solution is NoSQL, since they are so different, which one do you think suits my needs?

+3
source share
1 answer

As a rule, there is no right or wrong answer.

Fast data recording, any solution will be fine, although you did not say how much per second you store. Both solutions have everything you need to pay attention to.

Search (very fast) across all columns. For small volumes, say a few hundred Gb, then any solution would be Ok (assuming experienced people put it together). You really did not say how fast / often you search, so if it is many times a minute, this consideration becomes more important. A quick search often slows down the ability to quickly record large volumes because the indexes needed for a search need to be updated.

Audit reports usually have a time component, so a limited time search, for example, over the last 7 days, will significantly speed up the search time compared to searching for all records.

Reporting When you get up to 100Tb, you will need some real tricks or a big budget to get a report quickly. For static reporting, you are likely to end up creating one program that generates several reports at once to save I / O. Dynamic reports will be complex.

My opinion? Since you know the DBMS, I will start with this as a method and submit a solution. This will buy you time to study the real problems that you will encounter (the lack of premature optimization that many people at SO are so keen on). During this initial period of time, you can begin to choose nosql solutions and study them. I assume that you want to run your own hardware / database if you want to use cloud-type solutions, then go straight to them right away.

+7
source

All Articles