Using DDD, how to implement batch processing?

Question

I have a logic that you select a large number of records from one system, perform several transformations (based on business rules) and insert them into another system.

It seems that high performance (and memory) is striking in creating each of these records as an object, performing transformations on them, and then inserting all these objects into another system.

Is this the best way to achieve this in DDD to skip classes / objects and do it right through SQL, possibly a stored procedure?

Is there a better way to use DDD to achieve this?

Note. Systems use SQL databases; currently, objects such as CouchDB are not parameters.

Answer 1

Many distributed systems built on DDD use an event-driven architecture instead of waiting for all conversions to complete in one batch, since each object undergoes a state change that will cause it to be converted by your system, the object receives an event that is published in some or the message bus (for example, Mule for Java, MassTransit for .NET). Your transformation system will be subscribed to these events, and as each message arrives in your system, it will transform into the entities identified in the message, and then publish another message in the target system.

This "trickle treatment" can work continuously, all day long, without putting a load on your system, which would require that the work be performed after hours are closed. If you are concerned about performance, such an architecture can lead to a system whose last record will be converted 5 minutes after COB, where the batch job may not even work until 3 a.m. (after all the other batch jobs have finished).

If you really do not want the target system to be updated before midnight, for example, just send messages to the queue before midnight, and then publish them at the endpoint of the end system.

Greg Young published a blog and introduced this architecture widely. Check out his work at InfoQ.