Golang Background Processing

Question

How to do background processing / queue in Go?

For example, a user subscribes, and you send them a confirmation email - you want to send a confirmation email in the background, because it can be slow, and the mail server may not be available, etc. etc.

Ruby has a very nice DelayedJob solution that puts your work in a relational database (that is, simple and reliable), and then uses background workers to complete tasks, and retries if the work failed.

I am looking for a simple and reliable solution, not something low, if possible.

Answer 1

While you can just open goroutine and complete any async task that you want, this is not a great solution if you want reliability, that is, the promise that if you complete the task, it will be completed.

If you really need this to be a production assessment, select a distributed work queue. I do not know such queues that are typical for golang, but you can work with rabbitmq, beanstalk, redis or similar queue mechanisms to unload such tasks from your process and add fault tolerance and constancy of queues.

Answer 2

Simple goroutine can do the job: http://golang.org/doc/effective_go.html#goroutines

Open gorutine with email delivery and then respond to an HTTP request or something else

If you want to use the workstation, you can use the Rabbitmq or Beanstalk client, for example: https://github.com/streadway/amqp https://github.com/kr/beanstalk

Or maybe you can create a queue in the process with a FIFO queue running in goroutine https://github.com/iNamik/go_container

But perhaps the best solution is a job queue library, in this library you can set the concurrency limit, etc.: https://github.com/otium/queue

import "github.com/otium/queue" q := queue.NewQueue(func(email string) { //Your mail delivery code }, 20) q.Push(" [e-mail] ") 

Answer 3

I created a library to run asynchronous tasks using the message queue (brokers are currently supported by RabbitMQ and Memcache, but other brokers like Redis or Cassandra can be easily added).

You can take a look. It can be good enough for your use (and also supports chaining and workflows).

https://github.com/RichardKnop/machinery

This is an early project.

Answer 4

You can also use the goworker library to schedule tasks.

http://www.goworker.org/