How to ensure message queuing with multiple instances of WCF

I want to create a WCF service that uses the MSMQ binding because I have a large volume of notifications that the service must handle. It is important that clients are not delayed by the service and that notifications are processed in the order in which they were raised, therefore, the implementation of the queue.

Another consideration is sustainability. I know that I can compile MSMQ to make the queue more reliable, but I want to be able to run an instance of my service on different servers, so if server notification failures are not created in the queue and another server is processing.

I experimented with binding to MSMQ and found that you can have multiple instances of a service listening on the same queue, and leaving for themselves, they end up doing a kind of circuit with load balancing across available services. This is great, but I end up losing the sequence of queues as different instances take a different amount of time to process the request.

I use a simple console experiment application, which is epic code. When it starts, I get this output:

host1 open host2 open S1: 01 S1: 03 S1: 05 S2: 02 S1: 06 S1: 08 S1: 09 S2: 04 S1: 10 host1 closed S2: 07 host2 closed 

I want this to happen:

 host1 open host2 open S1: 01 <pause while S2 completes> S2: 02 S1: 03 <pause while S2 completes> S2: 04 S1: 05 S1: 06 etc. 

I would think that as S2 does not finish, it can still fail and return the message that it was processing to the queue. Therefore, S1 should not be allowed to pull another message out of the queue. My turn is for transactions, and I tried setting TransactionScopeRequired = true in the service, but to no avail.

Is it possible? Am I really wrong? Is there any other way to create a failover service without any central synchronization mechanism?

 class WcfMsmqProgram { private const string QueueName = "testq1"; static void Main() { // Create a transactional queue string qPath = ".\\private$\\" + QueueName; if (!MessageQueue.Exists(qPath)) MessageQueue.Create(qPath, true); else new MessageQueue(qPath).Purge(); // S1 processes as fast as it can IService s1 = new ServiceImpl("S1"); // S2 is slow IService s2 = new ServiceImpl("S2", 2000); // MSMQ binding NetMsmqBinding binding = new NetMsmqBinding(NetMsmqSecurityMode.None); // Host S1 ServiceHost host1 = new ServiceHost(s1, new Uri("net.msmq://localhost/private")); ConfigureService(host1, binding); host1.Open(); Console.WriteLine("host1 open"); // Host S2 ServiceHost host2 = new ServiceHost(s2, new Uri("net.msmq://localhost/private")); ConfigureService(host2, binding); host2.Open(); Console.WriteLine("host2 open"); // Create a client ChannelFactory<IService> factory = new ChannelFactory<IService>(binding, new EndpointAddress("net.msmq://localhost/private/" + QueueName)); IService client = factory.CreateChannel(); // Periodically call the service with a new number int counter = 1; using (Timer t = new Timer(o => client.EchoNumber(counter++), null, 0, 500)) { // Enter to stop Console.ReadLine(); } host1.Close(); Console.WriteLine("host1 closed"); host2.Close(); Console.WriteLine("host2 closed"); // Wait for exit Console.ReadLine(); } static void ConfigureService(ServiceHost host, NetMsmqBinding binding) { var endpoint = host.AddServiceEndpoint(typeof(IService), binding, QueueName); } [ServiceContract] interface IService { [OperationContract(IsOneWay = true)] void EchoNumber(int number); } [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)] class ServiceImpl : IService { public ServiceImpl(string name, int sleep = 0) { this.name = name; this.sleep = sleep; } private string name; private int sleep; public void EchoNumber(int number) { Thread.Sleep(this.sleep); Console.WriteLine("{0}: {1:00}", this.name, number); } } } 
+7
source share
2 answers

batwad

You are trying to manually create a service bus. Why aren't you trying to use an existing one?

NServiceBus, MassTransit, ServiceStack

At least 2 of them work with MSMQ.

In addition, if you absolutely need order, this may be for another reason - you want to be able to send a message, and you do not want dependent messages to be processed before the first message. You are looking for a saga pattern. NServiceBus and MassTransit let you easily manage Sagas, they allow you to simply initiate the initial message and then run the remaining messages based on conditions. This will allow you to realize smooth transfer of your distributed application.

You can then scale to thousands of clients, queue servers, and message handlers without having to write a single line of code and have no problems.

We tried to implement our own msmq service bus here, we gave up because another problem kept sneaking up. We went with NServiceBus, but MassTransit is also a great product (it is 100% open source, NServiceBus is not). ServiceStack is amazing when creating APIs and using message queues - I'm sure you can use it to create services that will act as queue interfaces in minutes.

Oh, did I mention that in the case of NSB and MT, both only require 10 lines of code to fully execute the queues, senders, and handlers?

----- ADDED -----

Udi Dahan (one of NServiceBus’s main contributors) talks about this in: Udi Dahan Messaging: Ordering Messages: Is It Economic? with Udi Dahan

Chris Patterson (one of the main participants in mass transit) Using Sagas to ensure the correct sequential message order

StackOverflow questions / answers: "Keep message order when using MSMQ messages in WCF application"

----- QUESTION -----

I have to say that I am puzzled by why you need to guarantee the order of the messages - would you be in the same position if you were using the HTTP / SOAP protocol? My guess is no, then why is this a problem in MSMQ?

Good luck, hope this helps,

+10
source

Ensuring uninterrupted message delivery is one of the delicate issues with large volume messaging.

In an ideal world, message recipients should be able to process messages out of order. This can be achieved by ensuring that your message source includes some sequence information. Again, ideally, this takes the form of some kind of x-of-n stamp (message 1 out of 10, 2 out of 10, etc.). Then your message is sent to collect the data in order after delivery.

However, in the real world, there is often no way to modify downstream systems to handle failed messages. In this case, you have two options:

  • Go all the way to a single thread - in fact, you can usually find some kind of “grouping identifier”, which means you can go single-threaded in the sense for each group, which means that you still have concurrency for different groups of messages.
  • Deploy a re-sequencer wrapper around each of your consumer systems that you want to receive in message order.

None of the solutions are very good, but I think you can have concurrency and message delivery in order.

+1
source

All Articles