Geek Answers Handbook

Java.lang.NoSuchMethodError in Flink

I am trying to read a file using:

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");

I get the following error:

java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found

I am using flink version 1.3.2, java version "1.8.0_91"

+1
source share
6 answers

One of the possible causes of the "java.lang.NoSuchMethodError" error is when we use a different version of flink, and then what we installed on our system. For me, I have Flink 1.3.2, and the version I used was 1.1.1. So I updated my pom file to have the same version.

0
source

There is a conflict with dependencies. Apache Flink loads many default classes into its class path.

https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html

Flink maven-shade

Apache Flink . , Flink,

IllegalAccessExceptions


NoSuchMethodError

- .

, pom.xml maven-shade-plugin ,

<relocation>
  <pattern>org.codehaus.plexus.util</pattern>
  <shadedPattern>org.shaded.plexus.util</shadedPattern>
  <excludes>
    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
  </excludes>
</relocation>
+2

IntelliJ Dashboard, IntelliJ, , Flink pom.xml, ,

 <build>

        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>

                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>

                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.codehaus.plexus.util</pattern>
                                    <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                    <excludes>
                                        <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                        <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                    </excludes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>


        </plugins>


</build>>

, maven clean install . , Dashboard intelliJ,

+2

, ,

0

, , Flink 1.3.2, pom, :

<build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.4.1</version>
            <configuration>
               <!-- get all project dependencies -->
               <descriptorRefs>
                  <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
               <!-- MainClass in mainfest make a executable jar -->
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
            <executions>
               <execution>
                  <id>make-assembly</id>
                  <!-- bind to the packaging phase -->
                  <phase>package</phase>
                  <goals>
                     <goal>single</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
         </plugin>
         <!--added newly-->
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
               <source>1.8</source>
               <target>1.8</target>
               <compilerId>jdt</compilerId>
            </configuration>
            <dependencies>
               <dependency>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-compiler-jdt</artifactId>
                  <version>0.21.0</version>
               </dependency>
            </dependencies>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                           <mainClass>FlinktoLambda</mainClass>
                        </transformer>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                           <resource>reference.conf</resource>
                        </transformer>
                     </transformers>
                     <relocations>
                        <relocation>
                           <pattern>org.codehaus.plexus.util</pattern>
                           <shadedPattern>org.shaded.plexus.util</shadedPattern>
                           <excludes>
                              <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                              <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                           </excludes>
                        </relocation>
                     </relocations>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   <name>my-app</name>
   <url>http://maven.apache.org</url>
   <dependencyManagement>
      <dependencies>
         <dependency>
            <groupId>FlinktoLambda</groupId>
            <artifactId>my-app</artifactId>
            <exclusions>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-remote_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-actor_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-slf4j_2.10</artifactId>
               </exclusion>
            </exclusions>
         </dependency>
      </dependencies>
   </dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-table_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-java</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-java_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-clients_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
   </dependencies>

, shade.

0

scala Flink ( , scala).

For example, in my case, I used Flink 1.7.1, and I had to update my scala dependencies from 2.11 to 2.12; I updated artifcatId of the respective dependencies as follows: from flink-scala_2.11to flink-scala_2.12, flink-table_2.11to flink-table_2.12, etc.

See here for more information.

0
source

More articles:


All Articles