Re-alignment strategies for ordered delivery in BizTalk:
I recently answered a question from a LinkedIn user regarding BizTalk ordered delivery options.
I thought it would be useful for people to understand some strategies for re-sequence of messages using BizTalk.
Often as a BizTalk developer, you need to integrate into linear business systems that are immutable. This may be for one or several different reasons. As an example, the cost of changing a system may be too high, or the manufacturer’s license states that support may be canceled if the system is changed.
This usually does not present a problem when the provider provides a well-thought-out API as the endpoint of the integration point. However, as many integration developers quickly find out, this is very rare.
What do I mean by a smart API? Well, except for all SODA participants (composition of services, fault contracts, etc.), the most important feature of the API is support for the consumption of data coming in the wrong order.
This is a pretty simple thing. For example, if you are a provider and you provide an HTTP operation as an integration point, then one of the fields that you could set in your operation is a timestamp or, even better, a serial number. This means that if the call is made with an outdated payload, then the corresponding compensation mechanism can be hacked, which can be as simple as dropping data.
This article discusses what to do when the provider has not created this function in the API, and therefore, as an integrator, you are forced to implement end-to-end ordered delivery as part of your integration solution.
As indicated in a response to a user’s message on LinkedIn (see link above), BizTalk ordered delivery in any, but the simplest of the scenarios is more difficult at best, and in the worst case can be a huge cost with increased complexity, as from a development point of view, so and support. The main reason is that BizTalk is designed to be massive at the same time to provide high throughput, and there is a direct and inevitable conflict between concurrency and order. The E2E training download for ordering a BizTalk solution is based on artifacts, such as peer-to-peer orchestrations, which introduce complexity and increase both the number of failures and the cost of failure.
The best solution is to keep the parallel processing as close as possible to the endpoints of the line of the business system, and then implement the so-called re-sequencer shell around each of the endpoints, which requires data that must be delivered in the correct order.
How to implement such a shell in BizTalk depends on some factors, which are described in the following table:
|Sequencing |Messages|Database |Wrapper | |field |are |integration?|strategy | | |deltas? | | | |--------------|--------|------------|----------------------------------| |n of a total m| N | Y |Stored procedure | |n of a total m| N | N |Singleton orchestration | |n of a total m| Y | Y |Batched singleton orchestration | |n of a total m| Y | N |Batched singleton orchestration | |Timestamp | N | Y |Stored procedure | |Timestamp | N | N |Singleton orchestration | |Timestamp | Y | Y |Buffer table with staggered reader| |Timestamp | Y | N |Buffer table with staggered reader|
The first factor in the Sequencing field relates to the idea that in order to implement any re-sequencer wrapper, at a minimum you need your message data to contain some sequence information. This may be the form of the original stamp; however, the best, albeit rarer, sequence information consists of a sequence number combined with the total number of messages, for example, message 1 out of 10, 2 out of 10, etc.
The second factor is delta? whether your message payload really contains one state change for the data or the sum of all past data changes. In other words, is it possible to restore the full current state of the data from this message? If the message payload contains only one change, you may not be able to restore the data state from one message, in which case your message will be delta.
The third factor in database integration? refers to whether the integration-entry point into the system is a database. The reason is that database-level integration is a fairly common integration scenario and, if available, can greatly simplify re-sequence processing.
The strategies from the table above are described in detail below:
Stored Procedure Storage This is the easiest re-alignment strategy. A new stored procedure is created that requests the target data before deciding whether to update the target data. Can the solution be as simple as the data that I have above the data in the target system?
Of course, to implement this strategy, the target data should also include the source data sequence field, although, if necessary, approximation can be made by relying on existing timestamps that may already exist in the target data. A stored procedure wrapper can be contained either in the target database, or ideally in a separate database.
Singleton Orchestration Essay The idea behind this strategy is a singleton orchestration. This is a template that you can implement to ensure that only one instance of orchestration will exist at any given time. There are many articles on the Internet that demonstrate how to implement this template in BizTalk.
The core of the idea is that singleton simply keeps track of the most recent successfully processed message sequence (or timestamp). If singleton receives a message that is older than the last sequence, it is simply discarded. This works because the messages are not delta, so the target system can only perform the most recent of several messages, and the data will be in the most recent state. Only when the data is successfully completed, is the last updated sequence contained in the updated singlet.
Singleton Orchestration Combination Wrap This strategy is based on the Singleton Orchestra wrap above, in addition to being more complex. Instead of storing only the latest sequence information in memory, a singleton is required to create and store a working set of messages in memory that it will reorder, and then process after all the expected messages from the packet arrived. This is because messages are deltas, so the target system MUST receive each message in the order in which they were intended. Once the packet has been successfully sent, singleton can complete.
For this, it is necessary that the source data contain the correlation identifier of some description, which allows you to define a message package. For example, when processing a specific set of orders from a client, incoming messages must contain an identifier for the client. This can then be used to route messages to a singleton orchestration instance correlated with this client. In addition, the available message sequence field should consist of n general forms m.
After initializing the singleton, it collects a working set of messages in memory and continues to fill it as new messages arrive. One way I saw this is to use System.Collections.Generic.List as a container for the working set. After the list is completely filled (list length = m), it is assumed that all messages in the package have been received, and the orchestration then cyclically moves around the working set and processes the messages in the target system.
One of the advantages of a batch peer-to-peer orchestra shell is the simultaneous processing with a correlation identifier. In the example above, this means that messages from two clients will be processed simultaneously.
Buffer table with a stepped reader shell. Perhaps the most difficult of the presented strategies, this solution should be used when you use delta messaging with a timestamp-based sequencing field. It can be implemented using a database of some description, which acts as a re-sequence buffer.
It is worth noting here that this wrapper for re-sequence does not guarantee an ordered delivery, but it is used well, which makes an ordered delivery very likely.
As messages arrive, they are written to the buffer and in the same operation, the buffer is reordered, so the order of messages stored in the buffer is always correct.
To create a buffer reader, find the receiving location that reads the messages in the buffer before sending messages to the send port with ordered delivery enabled, which will then process the messages in the target system. You can also use peer-to-peer orchestration as an intermediary if the semantics of the target system API are too complex for the send port.
However, using this wrapper, as I described above, will not include ordered delivery, since messages will almost certainly be sent to the buffer in the wrong order, which will lead to the processing of messages in the target system in the same (wrong) order. In this case, a scattered request occurs. This is a fancy way of saying that your buffer request should only select data at time intervals T, and select only those lines where the number of lines is less than the total number of lines in the buffer minus C.
This allows you to resolve the sequence of actions for an appropriate period of time. T will be familiar to most BizTalk developers as a polling interval for some adapters (for example, WCF-SQL adapter). C is a little more difficult to install, but by increasing this number, you reduce the likelihood that you will miss a message that is older than the last in your restored dataset when polling.
The fact that T and C depends on many factors, although these values should be based on your SLA delay and your message volume (or bandwidth). As a guideline, if you have an SLA to deliver data to the target system in 30 seconds, and you process 10 messages per second, then T should be about 10 seconds and C should be about 100 lines.
Of course, this only works if your messages for a given correlation ID are sent by the source system for a short period of time (ideally back-to-back). The longer the interval between gears, the greater the value of C and the less effective the shell.
One of the advantages of this strategy is that you can also deduplicate messages in the buffer if your data source is prone to sending duplicate messages and the endpoint of the end system is not idempotent. You can also use the buffer to implement FILO and other non-standard queue semantics.
Conclusions The strategies I discussed here are ways to bend BizTalk to a task that was not designed for this. As a result, each of them has reservations about the cost and complexity of support, and may also not work in certain scenarios. I would like to hear from everyone who has implemented other templates for ordered delivery in BizTalk.