Message Bus + Event Store + PubSub

Question

I am looking to create an application with many data sources, each of which places events on my system. Events have a well-defined data structure and can be encoded using JSON or XML.

I would like to be able to guarantee the persistence of events, as well as the fact that events are used as part of the publish / subscribe bus with several subscribers available for each event.

For a database, accessibility is very important, even if it scales to multiple nodes, and partition portability is important, so I can scale the number of places that my events can store. A possible sequence is enough for me.

I thought of using a JMS communications bus for messaging (like Mule ) or a communications bus for corporate AMQP messaging (like RabbitMQ or ZeroMQ ).

But for my application, it seems that if I could set up a publishing subscription system using CouchDB or something similar, it would solve my problem without the need to integrate enterprise messaging and persistent storage.

What will work better, CouchDB + scaling + loadbalancing + some PubSub engine or an explicit PubSub messaging system with an ultimately attached, consistent, accessible, partition-compatible storage? Which one is easier to configure, administer and manage? Which solution will have high throughput for a given cost? Why?

In addition, are there any other questions that I must ask before choosing my technology? (BTW, Java are both server and client languages).

Answer 1

I use the CouchDB message queue during production. (This is not pub / sub, so I don't think this answer is complete.)

Currently (June 2011), CouchDB has enormous potential as a messaging substrate:

• Good data retention
• Well balanced for clustering (on a local network using BigCouch or Lounge).
• Well balanced for distribution (between data centers around the world)
• Good platform. Despite the shortcomings listed below, I love CQS because I can reuse my DB, and it works from Erlang, NodeJS and every web browser.
• Request _changes
  • Continuous channels, instant delivery without polling
  • Network shift is not a problem, just try again from the previous position.

However, even the small messaging system in CouchDB requires careful planning and maintenance. CouchDB is potentially a great messaging server. (He's inspired by Lotus notes that handle a lot of email.)

However, these are problems with CouchDB:

• Application-only database files grow fast
  • Remember disk capacity
  • Remember the i / o drives. The seal will read and rewrite all live documents.
• Deleted documents are not actually deleted. They are marked with deleted=true and remain forever, even after compaction! This is really uniquely good about CouchDB because the deleted action will propagate through the cluster, even if the network goes down for a while.
• Distribution (duplication) is great, but what about escalating deleted documents? In the end, he will get ahead of everything else. The solution is to clean them , which effectively removes them from disk. Unfortunately, if you do 2 or more cleanings before asking for a map display / reduction, the view will completely rebuild itself . This may take too long, depending on your needs.

As usual, we hear NoSQL databases screaming “free lunch!”, “Free lunch!”. while CouchDB says "you have to work for it."

Unfortunately, if you don’t have constant pressure to reuse CouchDB, I would use a dedicated messaging platform. I had good experience working with ejabberd as a messaging platform and to communicate with / from Google App Engine.)

Answer 2

I think the best solution would be CouchDB + Jabber / XMPP server (ejabberd) + book: http://professionalxmpp.com

• JSON is CouchDB's Natural Storage Engine
• Jabber / XMPP server includes pubsub support
• Book must read

Answer 3

Although you can use the database as an alternative to the message queuing system, no database is a message queuing system, not even CouchDB. A message queuing system such as AMQP provides more than just saving messages, in fact with RabbitMQ, persistence is just an invisible service under the hood that takes care of all the problems you have to deal with on CouchDB.

Take a look at the RabbitMQ website, where there is a lot of information about AMQP and how to use it. They did a great job collecting articles and blogs about the order of the posts.