Could not find implicit value of proof parameter of type org.apache.flink.api.common.typeinfo.TypeInformation [...]

Question

I am trying to write some use cases for Apache Flink. One mistake I often encounter is

could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[SomeType] 

My problem is that I cannot really nail my head when they happen, and when they do not.

The most recent example of this:

 ... val largeJoinDataGen = new LargeJoinDataGen(dataSetSize, dataGen, hitRatio) val see = StreamExecutionEnvironment.getExecutionEnvironment val newStreamInput = see.addSource(largeJoinDataGen) ... 

where LargeJoinDataGen extends GeneratorSource[(Int, String)] and GeneratorSource[T] extends SourceFunction[T] are both defined in separate files.

When I try to build it, I get

 Error:(22, 39) could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[(Int, String)] val newStreamInput = see.addSource(largeJoinDataGen) 

1. Why is there an error in this example?

2. What will be the general guidance when these errors occur and how to avoid them in the future?

PS: first scala project and first flink project, please be patient

Answer 1

This mainly happens when you have a user code, that is, a source or function of a map, or something like that that has a common parameter. In most cases, you can fix this by adding something like

 implicit val typeInfo = TypeInformation.of(classOf[(Int, String)]) 

If your code is inside another method that has a common parameter, you can also try adding a context associated with the general parameter of the method, as in

 def myMethod[T: TypeInformation](input: DataStream[Int]): DataStream[T] = ... 

Answer 2

You can import instead of implicits

 import org.apache.flink.streaming.api.scala._ 

It will also help.

Answer 3

My problem is that I cannot really nail my head when they happen, and when they do not.

They occur when an implicit parameter is required. If we look at the definition of a method, we will see:

 def addSource[T: TypeInformation](function: SourceFunction[T]): DataStream[T] 

But we do not see any implicit parameter, where is it?

When you see a polymorphic method, where the type parameter has the form

 def foo[T : M](param: T) 

Where T is a type parameter, and M is a context binding . This means that the creator of the method requests an implicit parameter of type M[T] . This is equivalent to:

 def foo[T](param: T)(implicit ev: M[T]) 

In the case of your method, it actually expands to:

 def addSource[T](function: SourceFunction[T])(implicit evidence: TypeInformation[T]): DataStream[T] 

This is why you see a compiler complaint, since it cannot find the implicit parameter that the method requires.

If we go to the Apache Flink Wiki, under Enter information , we can understand why this is happening:

No implicit value for proof parameter error

In the event that TypeInformation cannot be created, programs cannot compile with an error, stating that "it was not possible to find an implicit value for the proof parameter of type TypeInformation". A common reason if the code generating the TypeInformation not been imported. Be sure to import the entire flink.api.scala package. import org.apache.flink.api.scala._

For general methods, you will need to require them to generate TypeInformation on the call site website:

For general methods, the data types of the function parameters and the return type may not be the same for each call and are not known on the site where the method is defined. The above code will lead to an error in which not enough hidden data will be received. In such cases, type information should be generated on the call site and passed to the method. Scala offers implicit parameters for this.

For your types, this means that if the invocation method is generic, it also needs to request a context binding for this type parameter.