I'm currently wondering how to handle application errors in Apache Flink streaming applications. In general, I see two cases:
- Transient errors in which you want the original data to be reproduced and processing can be successful on the second try. An example is a dependency on an external service that is temporarily unavailable.
- Permanent errors in which reprocessing will still work; for example, invalid input.
In the first case, it seems that the general solution is to simply throw some kind of exception. Or is there a better way, for example. a special kind of exception for more efficient management, such as FailedExceptionfrom Apache Storm Trident (see Error Handling in Storm Trident Topologies ).
For constant errors, I could not find information on the Internet. For example, an operation map()should always return something, so you can just silently discard messages, since it will be in Trident.
What are the available APIs or best practices? Thank you for your help.
source
share