Ensuring Sequential Processing of JMS Messages in an OC4J Cluster

Question

We have an application that processes a JMS message using a bean message. This application is deployed to the OC4J application server. (10.1.3)

We plan to deploy this application on several OC4J application servers that will be configured to operate in a cluster.

The problem is processing the JMS messages in this cluster. We must ensure that only one message is processed in the entire OC4J cluster at a time. This is necessary because messages must be processed in chronological order.

Do you know the configuration parameter that will control message processing in the OC4J cluster?

Or do you think that we need to implement our own synchronization code that will synchronize the beans message through the cluster?

Answer 1

I did sequential message processing in a cluster on a rather large scale - 1.5 million + message / day, using a combination of a competing customer pattern and a rental pattern.

Here is a kicker, although your requirement that you can only process one trance at a time will not allow you to achieve your goals. We had one and the same basic requirement - messages had to be processed in order. At least we thought we did. Then we had an epiphany - since we thought more about the problem, we realized that we did not need a complete streamlining. We really required to order only within each account. Therefore, we could distribute the load on the servers in the cluster by assigning account ranges to different servers in the cluster. Then each server was responsible for processing messages for this account in order.

Here's the second smart part - we used the Lease template, which dynamically assigns account ranges to different servers in the cluster. If one server in the cluster goes down, another will take over the lease and take on the server’s first responsibility.

This worked for us, and this process continued for about 4 years before it was replaced due to a merger.

Edit:

I will explain this solution in more detail here: http://coders-log.blogspot.com/2008/12/favorite-projects-series-installment-2.html

Edit:

Ok, got it. You are already processing at the level you need, but since you are deploying in a cluster, you need to make sure that only one instance of your MDB is actively pulling messages from the queue. In addition, you need the simplest workable solution.

You do not need to give up your MDB mechanism, which you have now, I don’t think. Essentially, what we're talking about here is a requirement for a distributed blocking mechanism, not to give it a too fake phrase.

So let me suggest this. At the moment when your MDB is registered to receive messages from the queue, it should check the distributed lock and see if it can capture it. The first MDB to capture the lock wins, and only it will register to receive messages. So now you have serialization. What shape should this castle make? There are many possibilities. Well, how about this. If you have access to the database, its transactional lock already provides some of what you need. Create a table with one row. The line shows the identifier of the server on which the lock is currently stored, and the expiration time. This is a server rental. Each server should have a way to generate its unique identifier, possibly a server name and a stream identifier, for example.

If the server can access the update for the row, and the lease has expired, it must capture it. Otherwise, he refuses. If he captures the lease, he needs to update the line with time in the near future, for example, five minutes or so, and fix the update. The active server must renew the lease before it expires. I recommend updating it when half the remaining time is left, so every 2-1 / 2 minutes if the lease expires in five. In this case, you have a transition to another resource. If the active MDB dies, another MDB (and only one) will be captured.

It should be pretty simple, I think. Now you want sleeping MDBs to sometimes check the lock to see if it is released.

So, active MDBs and inactive MDBs all have to do something. Perhaps they can create a separate thread for this. Many application manufacturers will not be happy if you do this, but adding a single thread doesn't really matter, especially since it spends most of its time. Another option would be to link the timer mechanism provided by many engines and periodically wake up your MDB to check your rental.

Oh, and by the way, make sure server administrators use NTP to synchronize the clock.

Answer 2

First point: this is a pretty crappy design, and you seriously limit performance by only being able to process one message at a time. I assume that you are clustering only for fault tolerance, because you will not get performance improvements?

Do you use the default JMS implementation with OC4J or another?

I used IBM MQ in the past, and it had the function that the queue can be marked as exclusive, which means that only one client can connect to it. This would seem to offer what you want.

An alternative would be to enter a sequence identifier (as simple as an increment counter), and the client processing the message will verify that the sequence identifier is the next expected value, if not after that, the message is returned. This approach requires that different clients retain the last valid sequence identifier that they saw in some centralized data warehouse, such as a database.

Answer 3

I agree with Stevendik: Maybe you are not aware of the design. Regarding sequence identifiers or similar approaches, I suggest you familiarize yourself with the messaging architecture using enterprise integration patterns: designing, building, and deploying messaging solutions (written by Gregor Hohpe y Bobby Woolf). This is a great book, many useful templates ... I am sure that the forces and the problem that you encounter are well described there.