Geek Answers Handbook

Connection Error While Performing Apache Flink Jobs

I have a work developed in Flink 0.9 that uses a graph module (Gelly). The job succeeds in the IDE (Eclipse), but after exporting to the JAR using maven (mvn clean install), it cannot execute on the local flink instance with the following error

"Myclass" entry point class could not be loaded due to communication failure

java.lang.NoClassDefFoundError: org/apache/flink/graph/GraphAlgorithm

Any idea why this is happening and how to solve it?

+5
source share
1 answer

It looks like flink-gelly did not get into your jar file. The most obvious cause of this problem is the lack of a maven dependency in the pom project file. But I assume that there is a dependency, otherwise development of work in the IDE would be impossible.

Most likely, the jar file was created by maven-jar-plugin , which does not include dependencies. Try adding the following snippet to pom.xml :

  <build> <plugins> <!-- We use the maven-shade plugin to create a fat jar that contains all dependencies except flink and it transitive dependencies. The resulting fat-jar can be executed on a cluster. Change the value of Program-Class if your program entry point changes. --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <executions> <!-- Run shade goal on package phase --> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <filters> <filter> <artifact>org.apache.flink:*</artifact> <excludes> <exclude>org/apache/flink/shaded/**</exclude> <exclude>web-docs/**</exclude> </excludes> </filter> </filters> <transformers> <!-- add Main-Class to manifest file --> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>YOURMAINCLASS</mainClass> </transformer> </transformers> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> <profiles> <profile> <!-- A profile that does everyting correctly: We set the Flink dependencies to provided --> <id>build-jar</id> <activation> <activeByDefault>false</activeByDefault> </activation> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>0.9-SNAPSHOT</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-core</artifactId> <version>0.9-SNAPSHOT</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>0.9-SNAPSHOT</version> <scope>provided</scope> </dependency> </dependencies> </profile> </profiles>

Now you can create the jar using mvn clean package -Pbuild-jar . Now the jar file will be in the target/ directory.

You can manually check if the jar (zip) file contains class files in /org/apache/flink/graph/

+4
source

More articles:


All Articles