What is the correct way to handle Redis in Tornado? (Async - Pub / Sub)

Question

I use Redis along with my Tornado application with asyc Brukva client, when I looked at sample applications on the Brukva website, they create a new init "connection in websocket

class MessagesCatcher(tornado.websocket.WebSocketHandler): def __init__(self, *args, **kwargs): super(MessagesCatcher, self).__init__(*args, **kwargs) self.client = brukva.Client() self.client.connect() self.client.subscribe('test_channel') def open(self): self.client.listen(self.on_message) def on_message(self, result): self.write_message(str(result.body)) def close(self): self.client.unsubscribe('test_channel') self.client.disconnect() 

its fine in the case of websocket, but how to handle it in the usual Tornado RequestHandler method, they talk about a lengthy polling operation (publish-subscribe model) . I create a new client approach in each post method of the update handler - is this the right approach? When I checked on the redis console, I see that the clients are increasing in each new post-operation.

Here is an example of my code.

 c = brukva.Client(host = '127.0.0.1') c.connect() class MessageNewHandler(BaseHandler): @tornado.web.authenticated def post(self): self.listing_id = self.get_argument("listing_id") message = { "id": str(uuid.uuid4()), "from": str(self.get_secure_cookie("username")), "body": str(self.get_argument("body")), } message["html"] = self.render_string("message.html", message=message) if self.get_argument("next", None): self.redirect(self.get_argument("next")) else: c.publish(self.listing_id, message) logging.info("Writing message : " + json.dumps(message)) self.write(json.dumps(message)) class MessageUpdatesHandler(BaseHandler): @tornado.web.authenticated @tornado.web.asynchronous def post(self): self.listing_id = self.get_argument("listing_id", None) self.client = brukva.Client() self.client.connect() self.client.subscribe(self.listing_id) self.client.listen(self.on_new_messages) def on_new_messages(self, messages): # Closed client connection if self.request.connection.stream.closed(): return logging.info("Getting update : " + json.dumps(messages.body)) self.finish(json.dumps(messages.body)) self.client.unsubscribe(self.listing_id) def on_connection_close(self): # unsubscribe user from channel self.client.unsubscribe(self.listing_id) self.client.disconnect() 

I appreciate if you provide a sample code for a similar case.

Answer 1

A bit late, but I used tornado-redis . It works with tornado ioloop and the tornado.gen module

Install tornadoredis

It can be installed from pip

 pip install tornadoredis 

or using setuptools

 easy_install tornadoredis 

but you really shouldn't do that. You can also clone storage and retrieve it. Then run

 python setup.py build python setup.py install 

Connect to redis

The following code is in your main.py or equivalent

 redis_conn = tornadoredis.Client('hostname', 'port') redis_conn.connect() 

redis.connect is called only once. This is a blocking call, so it should be called before starting the main ioloop. The same connection object is shared by all handlers.

You can add it to your application settings, for example

 settings = { redis = redis_conn } app = tornado.web.Application([('/.*', Handler),], **settings) 

Use tornadoredis

A connection can be used in handlers in self.settings['redis'] or it can be added as a property of BaseHandler and a subclass of this class for other request handlers.

 class BaseHandler(tornado.web.RequestHandler): @property def redis(): return self.settings['redis'] 

The tornado.web.asynchronous and tornado.gen.engine decorators are used to communicate with redis.

 class SomeHandler(BaseHandler): @tornado.web.asynchronous @tornado.gen.engine def get(self): foo = yield gen.Task(self.redis.get, 'foo') self.render('sometemplate.html', {'foo': foo} 

Additional Information

Additional examples and other functions, such as pooling and pipelines, can be found in the github registry.

Answer 2

You must combine the connections in your application. since it seems that brukva does not support this automatically (redis-py supports this, but blocks by nature, so it does not fit the tornado), you need to write your own connection pool.

The template is pretty simple. something in this direction (this is not a real working code):

 class BrukvaPool(): __conns = {} def get(host, port,db): ''' Get a client for host, port, db ''' key = "%s:%s:%s" % (host, port, db) conns = self.__conns.get(key, []) if conns: ret = conns.pop() return ret else: ## Init brukva client here and connect it def release(client): ''' release a client at the end of a request ''' key = "%s:%s:%s" % (client.connection.host, client.connection.port, client.connection.db) self.__conns.setdefault(key, []).append(client) 

it may be a little more complicated, but the main idea.