I just wanted to get some ideas on how to detect repeated tuples, so that they could be added to some dead letter queue at their last but unsuccessful attempt. I have an idea of how I would like to handle exceptions / rejections in general from a group of Google groups ; in particular, the idea of separating discarded and non-regressive exceptions.
- Exceptions that cannot be recovered would be an error that would not have been successful in replaying. I would like to catch them and immediately add the tuple to the dead letter queue. The problem can be analyzed and the tuple will be processed when the error / problem is fixed.
- With returned exceptions, this is not so simple. You may have an intermittent network problem that will succeed on restarting. On the other hand, you may have a break with a follow-up service that lasts for hours.
For the latter scenario (a thrown exception due to a downstream trip), this will result in the tuple repeating X times. I want to detect the tuple that fails for the Xth time, and then add it to the dead letter queue so that the tuple is not lost forever, but is reproduced when the problem has been resolved. Is there any way to achieve this? I looked at Tuple, but there is no member variable for the number of retries.
source share