Geek Answers Handbook

How to split a large csv file on a unix command line

I am just breaking a very large csv file into parts. When I run the following command. it is not completely broken, but returns the following error to me. how can i avoid splitting the whole file.

awk -F, '{print > $2}' test1.csv awk: YY1 makes too many open files input record number 31608, file test1.csv source line number 1
+7
unix awk printf
source share
2 answers

Just close files after writing:

 awk -F, '{print > $2; close($2)}' test1.csv
+15
source share

You should have many lines. Are you sure the second line is repeated enough to put these entries in a separate file? Anyway, awk keeps files open until the end. You will need a process that can close file descriptors when not in use.

Pearl for salvation. Yet again.

 #!perl while( <> ) { @content = split /,/, $_; open ( OUT, ">> $content[1]") or die "whoops: $!"; print OUT $_; close OUT; }

usage: script.pl your_monster_file.csv

prints the entire line to a file with the same name as the value of the second CSV column in the current directory if no fields are specified, etc.

0
source share

More articles:


All Articles