Associated worker pools in Python

Question

I am trying to set up a system of related work pools that look like this:

╭────────╮ ╭─────────╮ ╭─────────╮ │ ├──> Worker 1 ───> │ ├──> Worker 3 ───> │ │ │ Data ├──> Worker 1 ───> │ Queue ├──> Worker 3 ───> │ Queue ├───> Output │ ├──> Worker 1 ───> │ ├──> Worker 3 ───> │ │ ╰────────╯ ╰─────────╯ ╰─────────╯ ╭────────╮ ^ ^ ^ │ ├──> Worker 2 ────────┘ │ │ │ Data ├──> Worker 2 ──────────┘ │ │ ├──> Worker 2 ────────────┘ ╰────────╯ 

Before I deploy my own generic solution, are there any existing libraries (or pure multiprocessing / threading examples) that I could use? I’m not sure what to call this setting, so my Google searches didn’t give me many useful results.

Any advice is appreciated!

Answer 1

ZeroMQ is a lightweight solution and has Python bindings. http://www.zeromq.org/bindings:python

Answer 2

I looked at Beanstalkd , which is worth working with work queues with several processors, which are the manufacturer and / or consumers, which keeps worrying about the thread.

There is a Python client on beanstalkc

An example taken from their wiki.

 >>> import beanstalkc >>> beanstalk = beanstalkc.Connection(host='localhost', port=14711) >>> beanstalk.put('hey!') 1 >>> job = beanstalk.reserve() >>> job.body 'hey!' >>> job.delete() 

It can meet your needs - IIRC you can also have strong lines.