Implement load balancing using Python

Question

I am doing some research this summer and am working on parallelizing existing code. The focus right now is a way to load code balance so that it works more efficiently in a cluster. The current task is to make a proof of concept that creates several processes, each of which has its own stack, and when the process finishes processing the stack, it asks for the two nearest processes to find out if they have more work on the stack.

I am having difficulty conceptualizing this in python, but I was hoping someone could point me in the right direction or have some kind of example similar to either mpi4py or ParallelPython. Also, if someone knows about a more convenient or more convenient module, then that would be great to find out.

Thank.

Answer 1

Here is an easy way to do this.

• Create a single shared work queue. This application will fill this queue with work.

• Create an application that receives one item from the queue and does the work.

This is a single-manufacturer and multi-user design. It works well and can boot your machine using parallel processes.

, API. http://docs.python.org/library/queue.html. HTTP-, . , .

, RabbitMQ, . http://nathanborror.com/posts/2009/may/20/working-django-and-rabbitmq/

, http://hjb.python-hosting.com/, JMS.

.

, . :

for i in 1 2 3 4 5 6 7 8 9 10
do
    python myapp.py &
done

10 . 10 . , .

---

Peer, node -to- node , O (n * (n-1)/2) .

" " , 2 * n , "- " . , - . , ?

, , node . node . , , () () . ? , - . - .

- , n . , , . , , .

Answer 2

multiprocessing module