RDD spark saveAsTextFile gzip

Question

Is it possible to save a text file with spark rdd as gzip?

Can I run it somehow: combPrdGrp3.repartition(10).saveAsTextFile("Combined") and save it as gzip files?

+4

apache-spark

lightweight Aug 26 '15 at 16:12

source share

1 answer

banjara · Answer 1 · 2015-08-27T05:53:22+0000

using

 import org.apache.hadoop.io.compress.GzipCodec combPrdGrp3.repartition(10).saveAsTextFile("Combined", classOf[GzipCodec])

or

 sc.hadoopConfiguration.setClass(FileOutputFormat.COMPRESS_CODEC, classOf[GzipCodec], classOf[CompressionCodec])