Best concurrency structure for low latency, high data throughput on one machine

Question

I am looking for ideas on how a parallel structure can be implemented for my specific architecture using C #:

I have implemented several modules / containers (implemented as classes) that all individually connect to the message bus. Each module mainly produces or mainly consumes, but all modules also implement a request / response template for communication between these two modules. I am very new to parallel and asynchronous programming, but, in fact, I just want to run the entire architecture at the same time, not synchronously. I would really appreciate some guidance on which technologies (TPL, ThreadPool, CTP, open source libraries, ..) should be considered in my particular use case, given the following requirements:

• The whole system works only on the local machine (in the process, even on the message bus)
• At least one module performs heavy IO (reads several million 16-megabyte messages per second from a physical disk) by publishing several blocks of 16 bytes to a blocking collection for the entire time.
• Other modules consume locks from the collection at all times.
• The entry point is the producer, who begins to publish messages, exit when the producer finishes publishing the final set of messages in 16 bytes.
• The only message that bypasses the message bus is posting / consuming to / from the lock collection for bandwidth and latency reasons. (I am glad to hear suggestions to get rid of the message bus, if this is believable).
• Other modules handle operations such as writing to an SQL database, publishing to a GUI server, connecting to APIs that interact with external servers. Such operations are performed less frequently / throttled and can potentially be performed as tasks, rather than using the entire stream while the system is running.
• I am running a 64-bit quad-core, 16-gigabyte memory machine, but ideally I would like to implement a solution that can also work on a dual-core machine.

Given that I like to manage concurrency implementation, would you advise me to concentrate?

EDIT:. I like to emphasize that the biggest problem I encountered is how convenient it is to connect each container / module to the thread / task pool so that each of the modules performs asynchronous as yet complete and external communication between such modules. I am not too interested in optimizing a single producer / consumer pattern before I decided to connect all the modules to a parallel platform that can handle the number of tasks / threads connected dynamically.

Answer 1

I found n-act http://code.google.com/p/n-act/ , an Actors framework for .Net that implements pretty much what I'm looking for. I described in my question that I am looking for more detailed wireframe suggestions, and it seems to me that the Actor Framework decides what I need. I'm not saying that the n-act library will be what I am implementing, but this is a simple example of setting up participants who can communicate asynchronously and can work in their threads. Messaging also supports the new async / wait feature of C # 5.

Disruptor was mentioned above, as well as TPL and a couple of other ideas, and I appreciate the contribution, in fact it really made me think, and I spent a lot of time to understand what each library / structure is trying to configure and what problems it is trying to solve , so the entrance was very fruitful.

However, for my specific case, I think I believe that The Actors Framework is exactly what I need, because my main task is to exchange streams of asynchronous data. Unfortunately, I do not see a significant part of the Actor model implemented in any .NET technology (for now). TPL Dataflow looks very promising, but, as Weismat noted, it is not yet ready for production.

If N-Act is not stable or usable, I will look for a custom implementation through TPL. It is time to fully understand everything that TPL offers and start thinking at the same time at the design stage, rather than trying to transfer synchronous models into an asynchronous structure.

So the "Actor Model" was what I was looking for.

Answer 2

I recommend disruptor-net for a task where you have high bandwidth, low latency and a well-defined data stream.

If you are willing to sacrifice some efficiency for flow control, TPL Dataflow may work for you. It does a great job using TPL for task scheduling.

Answer 3

You can look at Concurrency and Coordination_Runtime if you are looking for a solution based on the concurrency framework. I think this may come in handy for your design ideas.
Otherwise, I will follow the rule that threads should be used when something works throughout the life of your application and tasks for short positions.
I believe that it is more important that the responsibility for consistency is clearly defined so that you can subsequently change the structure.
As usual for writing quick code, there are no thumb rules, but you need a lot of testing with small stubs to measure actual performance.