If Flink runs iterative programs, the data flow graph is not a DAG, but it allows loops. However, these cycles are not arbitrary and must follow a certain pattern that allows Flink to control this cyclic flow to some extent.
In other systems, often there is no strict technical reason for non-supporting cycles. Allowing loops in a general way is usually forbidden, as this can lead to an infinite loop (i.e. that the tuple will rotate the loop forever, and the program will not be called term).
Flink tracks the loop by counting the number of iterations. In this way, Flink can track which tuple belongs to that iteration, and can, for example, avoid tuples from the new iteration "take over" tuples from the older one once. In addition, it allows Flink to determine if the results of iterations n and n+1 are equal or not. An equal result shows a completed calculation, allowing Flink to break the endless loop and end (this is done for the so-called iteration of the fix point).
Read more in this article: https://dl.acm.org/citation.cfm?id=2350245
Using iteration in your program is described here: https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/programming_guide.html#iteration-operators
source share