I have a scenario where I need to process a csv file containing some simulation data from a device. Each line is an output representing the state of the device at a particular point in time. When processing each row, certain columns are checked for deviations / anomalies. If there are anomalies, an email should be sent to a handful of people with an anomaly detected. However, to avoid spamming them (sometimes csv can be several 100 thousand lines), I have to maintain the threshold of X seconds. If a message was sent for the same anomaly from the same condition (from the same device that is being simulated) X seconds ago, I should just ignore sending mail.
Currently, the solution I'm using seems awkward to me, where
1) I save the email and device ID with anomaly detection time.
2) Create one “warning” for each email identifier with the creation time stamp, sent time stamp, message identifier (from step 1) and device identifier with the status “NEW”.
3) Before sending each mail, I make a database to find out if the last message with the "SENT" status has a time stamp that exceeds the threshold for ignoring. (now - the send-time-stamp threshold) If yes, then I receive all warnings using the message identifier and send them and update all their status to SENT-else, just ignoring it.
I started with the thread pool executor and realized halfway that the read-send condition might fail if there are several threads trying to send emails and update the sent timestamp. Therefore, at the moment, I have set the thread pool size to 1, which exceeds the goal of the performer. (I don't have row level locking as I use Mongo as db support). The underlying data store should be nosql storage, as the fields can vary greatly and will not correspond to machine disks, as more simulations will be connected.
The application is distributed - so the csv file can be selected by any random node for processing and notification.
Will Akka be a good candidate for this process? Any ideas or lessons from previous experience implementing this are welcome (I should stick with the JVM).