How to show the full contents of a column in the Spark framework?

Question

I use spark-csv to load data into a DataFrame. I want to make a simple request and display the contents:

val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("my.csv") df.registerTempTable("tasks") results = sqlContext.sql("select col from tasks"); results.show() 

The call seems truncated:

 scala> results.show(); +--------------------+ | col| +--------------------+ |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-06 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:21:...| |2015-11-16 07:21:...| |2015-11-16 07:21:...| +--------------------+ 

How to show the full contents of a column?

Answer 1

results.show(20, false) will not be truncated. Check Source

Answer 2

If you put results.show(false) , the results will not be truncated

Answer 3

Other solutions are good. If these are your goals:

1. No column truncation
2. Without losing lines,
3. Fast and
4. effective

These two lines are useful ...

  df.persist df.show(df.count, false) // in Scala or 'False' in Python 

By saving 2 artist actions — counting and displaying — are faster and more efficient when using persist or cache to support the temporary data structure in artists. See more about persistence and cache .

Answer 4

Below code will help to view all rows without truncation in each column

 df.show(df.count(), False) 

Answer 5

results.show(20, False) or results.show(20, False) depending on whether you run it in Java / Scala / Python

Answer 6

results.show(false) will show you the entire contents of the column.

Show the default method, limiting to 20, and adding a number before false will show more lines.

Answer 7

try this command:

 df.show(df.count()) 

Answer 8

results.show(20,false) helped me in Scala.

Answer 9

Inside the Databricks, you can visualize the data frame in a tabular format. With the command:

 display(results) 

It will look

Answer 10

I use the plugin, the Chrome extension works pretty well:

[ https://userstyles.org/styles/157357/jupyter-notebook-wide.BIZ[1]

Answer 11

I had the same request, and I looked through a lot of materials available on the Internet and found something interesting that removes all doubts. So I was thinking of sharing with you guys as well. Try this code, it works:

 results.show(results.count.toInt) 

Answer 12

Try the following: df.show (some no) will work, but df.show (df.count ()) will not work. df.count gives the output type long, which is not accepted by df.show (), because it accepts an integer type.