Finding a permanent, distributed work queue for erlang

Question

Before reinventing the wheel, I look for pointers to open source projects that meet these requirements.

• implemented in erlang , although you can go either C if not too much baggage (for example: twisting a maze of dependencies.)
• The endpoint or client in erlang (for example: I want erlang code to run when tasks are completed.)
• distributes tasks to nodes and calls the erlang function to perform tasks.
• save work somehow
• no master nodes, no single point of failure
• homogeneous architecture
• manage the queue of jobs that can be copied without dropping jobs to the floor.
• work performed more than once is normal
• a working profile such as Riak or Couchbase (for example: start one node, then start the others and point to it.)

Strong preference for something light. There are many overloaded solutions in the field of higher education that seem to take so long to study, since it would be for me to recreate it from scratch (in fact, I basically archived the solution to this very problem in response to then else the question here is about stackoverflow. I can build what I described, but it looks like one of these needs, which is right in the middle of what erlang was designed for.)

What I reviewed: - ejabbered - more messaging framework - rabitmq - theoretically it does, but every time I go to their site, I drown in a sea of ​​abstractions. Everything seems ready. I can’t even say if he has any perseverance.

Edit to add: Here is a slide deck on a distributed lock using a locker. It seems like it solves a key part of the problem (if someone wants to turn off on their own). http://www.slideshare.net/knutnesheim/locker-distributed-consistent-locking

Further editing: I'm really looking for something easier than RabbitMQ. I know that he can do what I want, but it seems that the cost of training is comparable to the cost of its implementation, when in the end the user solution will be closer to what I really need.

Answer 1

We use RABBITMQ to link all of our applications with a complete set of things. Inside the entire installation is a central RABBITMQ server, to which systems create queues, both permanent and temporary. Due to the availability of RABBITMQ, our entire distribution system runs on top of it. Systems built using different technologies send and receive jobs from other systems through RABBITMQ.

We came up with a message format, which can be in JSON or XML, in which systems exchange data with each other. It is so fast. However, there are so many details that I will not go into, but I had to write an OTP application on top of the RABBITMQ Client to distract all AMQP materials from erlang programmers. All programmers know APi, let's say I send a request to System A , just prepare the message format, M and call the API: zeenode_amqp:req(SystemA,Message) . Systems can send synchronous or asynchronous requests.

What should be your approach: RABBITMQ is very good for queuing systems. Infact in our setup, RABBITMQ pushes messages directly to the servers as soon as they push RABBITMQ from clients. Using a carefully designed queue and exchange naming convention and carefully examining the various AMQP usage patterns, it will be perfect for you.

I think you can roll your own using technology dictionaries like Gproc in combination with the Erlang Queue Module and Poolboy . In any case, I would recommend RABBITMQ. let me know, I can send you some libraries so you can study them and see if they work for you. as soon as you have a good RABBITMQ setup, well configured according to the documents on their website, and then you have the Erlang amqp client installed, then you can dynamically create queues and exchanges (whether you want them to be constant or not depends on from what you do). You can even group RABBITMQ servers for guaranteed availability.

Answer 2

We use RabbitMQ for such tasks. The RabbitMQ Exchange Queue Binding Model supports flexible configuration. You publish to the exchange, the binding ensures that messages are received in the queue (s). One or more workflows can subscribe to a queue. Queues and exchanges may be permanent. Some people say it takes about six months to fully understand RabbitMQ.