Spark: saving a Dataframe in ORC format

Question

In the previous version, we used the saveAsOrcFile () method for RDD. It is now! How to save data in a DataFrame in ORC format?

def main(args: Array[String]) { println("Creating Orc File!") val sparkConf = new SparkConf().setAppName("orcfile") val sc = new SparkContext(sparkConf) val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) val people = sc.textFile("/apps/testdata/people.txt") val schemaString = "name age" val schema = StructType(schemaString.split(" ").map(fieldName => {if(fieldName == "name") StructField(fieldName, StringType, true) else StructField(fieldName, IntegerType, true)})) val rowRDD = people.map(_.split(",")).map(p => Row(p(0), new Integer(p(1).trim))) //# Infer table schema from RDD** val peopleSchemaRDD = hiveContext.createDataFrame(rowRDD, schema) //# Create a table from schema** peopleSchemaRDD.registerTempTable("people") val results = hiveContext.sql("SELECT * FROM people") results.map(t => "Name: " + t.toString).collect().foreach(println) // Now I want to save this Dataframe(peopleSchemaRDD) in ORC Format. How do I do that? 

}

Answer 1

Starting with Spark 1.4, you can simply use the DataFrameWriter and set the format to orc :

 peopleSchemaRDD.write.format("orc").save("people") 

or

 peopleSchemaRDD.write.orc("people")