Python asyncore uses 100% processor after client connection

Question

I use asyncore to implement a publication subscription. I understand that using twisted or ZMQ may be the best solution, however in this case it should be pure python. When waiting for connections, the CPU usage is ~ 1%, as soon as the client connects to the processor load, write up to 100%. It is not discarded even after disconnecting the client.

My server class:

class Host(asyncore.dispatcher): log = logging.getLogger('Host') def __init__(self, port=7655): asyncore.dispatcher.__init__(self) self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.set_reuse_addr() self.bind(('0.0.0.0', port,)) self.listen(5) self.clients = [] def handle_accept(self): socket, addr = self.accept() self.log.info("Aceepted client at {0}:{1}".format(addr[0], addr[1])) self.clients.append(RemoteClient(self, socket, addr)) def broadcast(self, message): self.log.info("Broadcasting message: {0}".format(message)) for client in self.clients: client.message(message) 

And my handler:

 class RemoteClient(asyncore.dispatcher): log = logging.getLogger('Host') def __init__(self, host, socket, address): asyncore.dispatcher.__init__(self, socket) self.host = host self.outbox = collections.deque() def message(self, message): self.outbox.append(message) def handle_write(self): if not self.outbox: return message = self.outbox.popleft() if len(message) > MAX_MESSAGE_LENGTH: raise ValueError('Message too long') self.send(message) def handle_close(self): self.host.clients.remove(self) self.log.info("Client removed from list") self.close() def handle_error(self): self.log.error("Socket error") 

I tried to look for a solution, but I can not understand what is happening. Any help appreciated!

Answer 1

Explanation

Your problem is that you are not overriding the asyncore.dispatcher.writeable method, by default:

 def writable(self): return True 

What causes asyncore.poll to run in a while True (thus using 100% CPU):

 def poll(timeout=0.0, map=None): if map is None: map = socket_map if map: r = []; w = []; e = [] for fd, obj in map.items(): is_r = obj.readable() is_w = obj.writable() # This is always true if is_r: r.append(fd) if is_w: w.append(fd) if is_r or is_w: e.append(fd) # This always happens if [] == r == w == e: time.sleep(timeout) # This never happens return try: # Here, having w (as function parameter) set to non-empty value # causes select to immediately return without waiting with w set # just to your client r, w, e = select.select(r, w, e, timeout) 

---

Decision

I think a really clean solution would be to rebuild asyncore with some kind of threading.Event mechanism that would provide a wait for the writeable object, but so far this worked for me:

 # Add to RemoteClient def writable(self): ''' It has point to call handle_write only when there something in outbox Having this method always returning true will cause 100% CPU usage ''' return bool(self.outbox) # When you start loop asyncore.loop(timeout=5.0) 

You can also see writable() overridden in the example in the official documentation .

I just prefer to kill, waiting sooner than after 30 seconds.