ReduceByKey method not found in IntelliJ

Question

Here is the code I'm trying to do for reduceByKey:

import org.apache.spark.rdd.RDD import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import scala.math.random import org.apache.spark._ import org.apache.spark.storage.StorageLevel object MapReduce { def main(args: Array[String]) { val sc = new SparkContext("local[4]" , "") val file = sc.textFile("c:/data-files/myfile.txt") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) } } 

Provides a compiler error: "cannot resolve the reduceByKey character"

When I hover over the reduceByKey implementation, it gives three possible implementations, so it seems like it was found ?:

Answer 1

You need to add the following import to the file:

import org.apache.spark.SparkContext._

Spark Documentation:

"In Scala, these operations are automatically available on RDDs containing Tuple2 objects (built-in tuples in a language created by simply writing (a, b)) while you import org.apache.spark.SparkContext._ into your program to include implicit Sparks transformations Key-value pair operations are available in the PairRDDFunctions class, which automatically wraps RDD tuples if you import conversions.

Answer 2

I noticed that sometimes IJ cannot allow methods that are imported implicitly through PairRDDFunctions https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions. scala

Implicitly imported methods include the reduceByKey * and reduceByKeyAndWindow * methods. I currently do not have a general solution - except that you can safely ignore intellisense errors.

Answer 3

It seems that documentary behavior has changed in Spark 1.4.x. In order for IntelliJ to recognize implicit conversions, now you need to add the following import:

 import org.apache.spark.rdd.RDD._