Geek Answers Handbook

Why is the Scala character not accepted as a column reference?

Having tried the Spark SQL examples, they seem to work well, except when expressions are needed:

scala> val teenagers = people.where('age >= 10).where('age <= 19).select('name)
<console>:23: error: value >= is not a member of Symbol
       val teenagers = people.where('age >= 10).where('age <= 19).select('name)

scala> val teenagers = people.select('name)
<console>:23: error: type mismatch;
 found   : Symbol
 required: org.apache.spark.sql.catalyst.expressions.Expression
       val teenagers = people.select('name)

It seems I need an import that is not documented.

If I import all the content

import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.dsl._
import org.apache.spark.sql.catalyst.errors._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._ 
import org.apache.spark.sql.catalyst.types._
import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.execution
import org.apache.spark.sql.hive._

EDIT: ... and

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._

it works.

+4
source share
2 answers

There is an implicit conversion that you are missing.

val sqlContext: org.apache.spark.sql.SQLContext = ???
import sqlContext._

However, this has changed in recent (and supported) versions of Spark.

+4
source

Spark 2.0 is here where SparkSession is replaced SQLContext.

There is an implicit conversion from Scala Symboland Spark SQL Columnthat is not available in scope.

SparkSession import, implicits .

val spark: SparkSession = ...
import spark.implicits._

scaladoc implicits.

spark-shell , .

+2

More articles:


All Articles