Spark-submit Sql Context Create expression does not work

Question

My code below does not work with Spark-submit.

sqlContext.sql(s""" create external table if not exists landing ( date string, referrer string) partitioned by (partnerid string,dt string) row format delimited fields terminated by '\t' lines terminated by '\n' STORED AS TEXTFILE LOCATION 's3n://....' """) 

It gives an error: An exception in the stream "main" java.lang.RuntimeException: [1.2] failure: `` with '' is expected, but the identifier creates the one found

This code works in Spark-shell, but not in Spark-submit. What is the reason?

Answer 1

The "sqlContext" in the default shell is "HiveContext". Maybe you need a new HiveContext instead of sqlContext in the script.

You can do it like this:

 import SparkContext._ import org.apache.spark.sql.hive._ val sc = new SparkContext() val sqlContext = new HiveContext(sc)