Geek Answers Handbook

IllegalAccessError for guava StopWatch from org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus

I try to run a small spark application and get the following exception:

Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262) at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:217) at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) at scala.Option.getOrElse(Option.scala:120)

corresponding gradle dependency section:

 compile('org.apache.spark:spark-core_2.10:1.3.1') compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.6.2') {force = true} compile('org.apache.hadoop:hadoop-mapreduce-client-app:2.6.2') {force = true} compile('org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.6.2') {force = true} compile('com.google.guava:guava:19.0') { force = true }
+6
source share
5 answers

version 2.6.2 from hadoop:hadoop-mapreduce-client-core cannot be used together with guava new versions (I tried 17.0 - 19.0 ), since the guava StopWatch cannot be accessed (by calling IllegalAccessError above)

using hadoop-mapreduce-client-core latest version - 2.7.2 (in which they do not use guava StopWatch in the above method, but use org.apache.hadoop.util.StopWatch ), solved the problem with two additional dependencies that were are necessary

 compile('org.apache.hadoop:hadoop-mapreduce-client-core:2.7.2') {force = true} compile('org.apache.hadoop:hadoop-common:2.7.2') {force = true} // required for org.apache.hadoop.util.StopWatch compile('commons-io:commons-io:2.4') {force = true} // required for org.apache.commons.io.Charsets that is used internally

Note: there are two org.apache.commons.io packages: commons-io: commons-io (ours here) and org.apache.commons: commons-io (old, 2007). be sure to include the correct one.

+18
source

It looks like you have a version mismatch of Guava.

Something in your code base is trying to call the Stopwatch constructor, but the constructors have been removed from Guava 17.0 in favor of the static factory methods ( createStarted() and createUnstarted() ) that were added in Guava 15.0.

You should update any code that tries to use constructors to use static factory methods.

+2
source

I had this problem with Spark 1.6.1 because one of our additional dependencies supplanted Guava 14.0.1 and replaced it with 18.0. Spark has a basic dependency for hadoop client version 2.2. See [Maven Repo] ( https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10/1.6.1 )

The solution that worked was to add the following to sbt libraryDependencies : "org.apache.hadoop" % "hadoop-client" % "2.7.2"

+2
source

In my case, adding guava 21.0 will result in an error.

  <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>21.0</version> </dependency>

After that, I use guava 15.0 or remove the above dependency. My code works well.

+1
source

I just changed my version of guava from 19.0 to 15.0 and it worked. I am currently using spark 2.2

 <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>15.0</version> </dependency>
+1
source

More articles:


All Articles