Guidelines for Real-Time Data Logging in a NoSQL Database

Question

I got a Java web application that receives some events in real time and pushes them to the user interface level. I want to log all perceived events, and since the amount of information will be huge, I prefer to use NoSQL db.

I installed mongodb for this purpose, which inserts a document for the event. The problem is that this approach (disk access for each event) drastically slows down the whole process.

So what approaches can I take in this situation? what options are available in mongodb for this (e.g. bulk insert, asynchronous insert, caching, ...)? switching to some other NoSQL SQL server implementation makes a difference? what are the best methods here?

Answer 1

I waited a while to see other answers, but lost my patience. I used MongoDB as a log repository for 3 projects (two for Java and one for C #). Based on this, I can determine the following important rules for organizing logging:

• Do not use indexes. If you mostly write, then indexes cause performance degradation. If you need log analysis after processing, copy the information to another database or collection. Unfortunately, you cannot get rid of the primary key _id - just leave it as is (GUID) or replace it with an automatic increment of NumberLong .

• Bottom recording problem. MongoDB has rich control over write operations. You can establish a correspondence between LogLevel and write rules. For example, DEBUG , INFO , WARN can come with WriteConcern.UNACKNOWLEDGED and ERROR , FATAL can be saved with WriteConcern.ACKNOWLEDGED . This way you improve application performance by avoiding pauses when writing low priority emails. At the same time, you are sure that important messages (which are rare) are stored.

• Cache for the collection instance. I mean, every time a message arrives, you do not resolve Mongo objects over getDB or getCollection .

• Reduce the amount of data transmitted over the network. Limit your message to a minimum set of fields. Truncate stack tracing too long. See how Spring 3.x shortens the fully qualified class name swsmmaRequestMappingHandlerMapping instead of some.whatever.sub.main.minimal.agent.RequestMappingHandlerMapping