Explode in pyspark

Question

I would like to convert from a DataFrame that contains lists of words into a DataFrame with each word in its own line.

How do I explode in a column in a DataFrame?

Here is an example with some of my attempts where you can uncomment every line of code and get the error mentioned in the following comment. I am using PySpark in Python 2.7 using Spark 1.6.1.

from pyspark.sql.functions import split, explode DF = sqlContext.createDataFrame([('cat \n\n elephant rat \n rat cat', )], ['word']) print 'Dataset:' DF.show() print '\n\n Trying to do explode: \n' DFsplit_explode = ( DF .select(split(DF['word'], ' ')) # .select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;" # .map(explode) # AttributeError: 'PipelinedRDD' object has no attribute 'show' # .explode() # AttributeError: 'DataFrame' object has no attribute 'explode' ).show() # Trying without split print '\n\n Only explode: \n' DFsplit_explode = ( DF .select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;" ).show() 

I ask for advice

Answer 1

explode and split are SQL functions. Both work on SQL Column . split takes a Java regular expression as the second argument. If you want to split the data into arbitrary spaces, you will need something like this:

 df = sqlContext.createDataFrame( [('cat \n\n elephant rat \n rat cat', )], ['word'] ) df.select(explode(split(col("word"), "\s+")).alias("word")).show() ## +--------+ ## | word| ## +--------+ ## | cat| ## |elephant| ## | rat| ## | rat| ## | cat| ## +--------+ 

Answer 2

To break into spaces and also remove blank lines, add a where clause.

 DF = sqlContext.createDataFrame([('cat \n\n elephant rat \n rat cat\nmat\n', )], ['word']) >>> (DF.select(explode(split(DF.word, "\s")).alias("word")) .where('word != ""') .show()) +--------+ | word| +--------+ | cat| |elephant| | rat| | rat| | cat| | mat| +--------+