How to save a file in the format Feather \ storage from Spark?

Question

Is it possible to export a data frame from Apache Spark to a pen ( https://github.com/wesm/feather ) file?

+4

Dmitry Petrov Jul 08 '16 at 16:02

1 answer

jezrael · Answer 1 · 2016-07-08T16:13:17+0000

IIUC you can try:

import feather
import pandas as pd

#df is spark df
pdf = df.toPandas()
feather.write_dataframe(pdf, 'test.feather')