How to add pandas data to an existing csv file?

Question

I want to know if the pandas to_csv() function can be used to add a data frame to an existing csv file. The csv file has the same structure as the downloaded data.

Answer 1

You can add in csv opening the file in add mode:

 with open('my_csv.csv', 'a') as f: df.to_csv(f, header=False) 

If it was your csv, foo.csv :

 ,A,B,C 0,1,2,3 1,4,5,6 

If you read this, then add, for example, df + 6 :

 In [1]: df = pd.read_csv('foo.csv', index_col=0) In [2]: df Out[2]: ABC 0 1 2 3 1 4 5 6 In [3]: df + 6 Out[3]: ABC 0 7 8 9 1 10 11 12 In [4]: with open('foo.csv', 'a') as f: (df + 6).to_csv(f, header=False) 

foo.csv becomes:

 ,A,B,C 0,1,2,3 1,4,5,6 0,7,8,9 1,10,11,12 

Answer 2

You can specify python write mode in pandas to_csv . To add, this is a.

In your case:

 df.to_csv('my_csv.csv', mode='a', header=False) 

The default mode is "w".

Answer 3

A small helper function that I use with some header checking tools to handle all of this:

 def appendDFToCSV_void(df, csvFilePath, sep=","): import os if not os.path.isfile(csvFilePath): df.to_csv(csvFilePath, mode='a', index=False, sep=sep) elif len(df.columns) != len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns): raise Exception("Columns do not match!! Dataframe has " + str(len(df.columns)) + " columns. CSV file has " + str(len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns)) + " columns.") elif not (df.columns == pd.read_csv(csvFilePath, nrows=1, sep=sep).columns).all(): raise Exception("Columns and column order of dataframe and csv file do not match!!") else: df.to_csv(csvFilePath, mode='a', index=False, sep=sep, header=False) 

Answer 4

A bit late, but you can also use the context manager if you open and close your file several times, or register data, statistics, etc.

 from contextlib import contextmanager import pandas as pd @contextmanager def open_file(path, mode): file_to=open(path,mode) yield file_to file_to.close() ##later saved_df=pd.DataFrame(data) with open_file('yourcsv.csv','r') as infile: saved_df.to_csv('yourcsv.csv',mode='a',header=False)` 

Answer 5

Initially, starting with pyspark data frames - I got errors when converting types (when converting to pandas df and then adding to csv), considering the types of schemas / columns in my pyspark data frames

I solved the problem by forcing all the columns in each df to be of type string, and then adding this to csv as follows:

 with open('testAppend.csv', 'a') as f: df2.toPandas().astype(str).to_csv(f, header=False) 

Answer 6

 with open(filename, 'a') as f: df.to_csv(f, header=f.tell()==0) 

• Create a file if it does not exist, otherwise add
• Add a header if the file is created, otherwise skip it