Sending C # workers to different virtual machines

Question

My question is very simple, but there can be no answer. I have thousands of workers performing the same task, and I would like them to run in parallel on many remote virtual machines (Cloud or Network).

Here is an idea:

class ThreadManager { public void main() { for (int i = 0; i<300; i++) { myWorker wk; if (i < 100) wk = new myWorker(IP_Computer_1); else if (i < 200) wk = new myWorker(IP_Computer_2); else wk = new myWorker(IP_Computer_3); wk.RunWorkerAsync(); } } internal class myWorker :: BackgroundWorker { public string IP_Computer; {...}//constructor protected override void OnDoWork(DoWorkEventArgs e) { WriteToDatabaseTable("BAZINGA ! Greetings from computer " + Dns.GetHostName()); //SQL server DB is hosted on a publicly accessible domain base.OnDoWork(e); } } } 

Of course, this is pseudo code, you get the idea: Assigning threads / workers to different computers / virtual machines over a network (not just for several cores)

What are the main / easiest options I have? I am open to ANY easy and effective solution (I don’t want to move on to complex workflow planning / reengineering of the workflow, etc.), Let it be as simple as it is, please)

NB: This is NOT an innocent question. I know all the fuss about grid / cloud computing, HPC, Amazon web services, Azure, etc. I spent a lot of time reading and trying things, and my opinion is that there is a lot of (money) business out there. It’s sometimes useful to go back to the basics and ask simple questions to find out if we really need a too complicated / complicated / invasive / expensive solution to solve the main problems (we are not a Fortune 500 with a giant network, but just a small research company with specific / atomic computing )

Answer 1

The easiest way to do this is to create a DoWork model as a WCF service method. Use NetTCP bindings and load balancing across multiple computers . This is suitable if the machines are stateless clones and all state is maintained in the database. In this case, you should not worry about which machine is serving your request, and you do not need to use futz with IP addresses at the application level.

This does not provide any guarantees of communication, reliability and fault tolerance. For example, if a service machine dies during a request, the request will not be restarted on another machine. It can be perfect for your needs.

If you want guaranteed delivery (such as fire-and-forget) and transaction longevity, consider using the MSMQ transport for your WCF service. In this case, the service machine is deactivated transactionally, and the transaction is completed only if the database was updated.

Answer 2

How about NGrid open source grid computing. If you want to do all this yourself, you need very little coding to get the fault tolerance baked in your custom solution. The most difficult thing is local failures that do not affect your entire system.

Answer 3

In your ... well .... question there are many unanswered questions. Conceptually, this is a very simple task, however, as soon as you start to get some details, it becomes more and more difficult.

As stated in your question, I understand that you want to start and forget tasks for distributed processors. This means that each task is atomic and no return is required. The way to do this is to create a listening server on distributed nodes, which your main node will then send messages containing serialized work objects to execute. Then the distributed nodes will start a new thread (or workflow) to complete the processing.

Then the problem begins to expand, based on how much more features are involved.

Answer 4

The first thing to understand is that you can send data to other computers quite easily (e.g. WCF) ... however, you cannot easily send code / logic. Thus, you can approach the problem in one of two ways:

• You have a “setup procedure” in which you manually take the workers and distribute them to each computer that will participate in the exercise, then send them pieces of data for processing via WCF or MSMQ or any other amount of remote communication methods
• Ask the master process to serialize the .net assembly, pass it on to each workflow, and ask that user to upload the assembly to the application domain, and then start sending them the bits to work on.

In any case, you will have to first solve the communication problem (you press something like WCF, or you have slaves from the queue), etc.

Answer 5

.NET has no easy built-in way to do this. You might want to make your workers serializable and send them to nodes through a message queue (possibly using System.Messaging and MSMQ).