Finding a simple persistent message buffer in Java

Question

I am looking for a simple persistent buffer as a temporary storage for JSON messages in a Java application. Memory usage should be relatively constant and independent of the number of messages in the buffer. It would be nice to be able to play messages from a point in the past. Removing old messages should be effective. You must be able to process 1 m messages / h.

Currently, my application uses a local RabbitMQ broker, which translates messages to a remote RabbitMQ broker. When a remote broker is disconnected or does not accept messages, local use of the RabbitMQ firewall increases with the length of the queue and eventually stops receiving messages. I want to change this to a local disk buffer and a stream copying messages to a remote RabbitMQ broker.

Does anyone have any idea? I looked at Kafka, but this seems like an unnecessary case for my use case. MongoDB is a feature, but I'm worried about memory usage.

Answer 1

Memory usage is always a problem on any system. I use MongoDB for production, and when I compare with similar solutions (CouchDB, CouchBase, redis.io), MongoDB is really good at managing memory and ease of implementation. But I have to admit that I never had the opportunity to check Riak in more detail.

I save 5,000,000 user records with 4 index fields and the entire user session behind the rest / web service api site that uses the messaging service.

The messaging service is using a different db instance on the same server. My user records have at least 20 fields, and session records have only 5 fields. Our ubuntu servers have never used more than 10 GB baht, even with heavy boot processes.

Hope this helps find out.

ps: it all depends on the data model and how you implement your infrastructure.

Hi,

EDIT:

I think this is a good slideshow about using MongoDB for messaging.

and a nice article on MongoDB and messaging.

You can use the test code and see that the results are approved for your solution. Please remember to share your results if you check.