Sed, how to delete the first 17 lines and the last 8 lines in a file

Question

I have a CSV file of a large 150GB file, and I would like to delete the first 17 lines and the last 8 lines. I tried the following but it doesn't seem to work correctly

sed -i -n -e :a -e '1,8!{P;N;D;};N;ba' 

and

 sed -i '1,17d' 

I wonder if anyone can help with sed or awk, will one liner be big?

Answer 1

 awk -v nr="$(wc -l < file)" 'NR>17 && NR<(nr-8)' file 

Answer 2

head and tail better for the job than sed or awk .

 tail -n+18 file | head -n-8 > newfile 

Answer 3

All awk:

 awk 'NR>y+x{print A[NR%y]} {A[NR%y]=$0}' x=17 y=8 file 

Answer 4

 Try this : sed '{[/]<n>|<string>|<regex>[/]}d' <fileName> sed '{[/]<adr1>[,<adr2>][/]d' <fileName> 

Where

• /.../= delimiters

• n = line number

• string = string found in string

• regex = regular expression matching pattern

• addr = string address (number or pattern)

• d = delete

Send link

Answer 5

 LENGTH=`wc -l < file` head -n $((LENGTH-8)) file | tail -n $((LENGTH-17)) > file 

Edit: as mtk posted a comment, this will not work. If you want to use wc and the length of the tracking file, you should use:

 LENGTH=`wc -l < file` head -n $((LENGTH-8)) file | tail -n $((LENGTH-8-17)) > file 

or

 LENGTH=`wc -l < file` head -n $((LENGTH-8)) file > file LENGTH=`wc -l < file` tail -n $((LENGTH-17)) file > file 

What makes this solution less elegant than published by choroba :)

Answer 6

I found out about it today for the shell.

 { ghead -17 > /dev/null sed -n -e :a -e '1,8!{P;N;D;};N;ba' } < my-bigfile > subset-of 

You need to use a non-consuming head , therefore using ghead from GNU coreutils.

Answer 7

Like Thor answer , but a little shorter:

 sed -i '' -e $'1,17d;:a\nN;19,25ba\nP;D' file.txt 

-i '' tells sed to modify the file in place. (The syntax may be slightly different on your system. Check the manual page.)

If you want to remove the front lines from the front and tail from the end, you will need to use the following numbers:

 1,{front}d;:a\nN;{front+2},{front+tail}ba\nP;D 

(I put them in braces here, but it's just a pseudo-code. You will have to replace them with actual numbers. Also, it should work with {front+1} , but this is not on my machine (macOS 10.12.4). I think what a mistake.)

I will try to explain how the team works. Here's a human readable version:

 1,17d # delete lines 1 ... 17, goto start :a # define label a N # add next line from file to buffer, quit if at end of file 19,25ba # if line number is 19 ... 25, goto start (label a) P # print first line in buffer D # delete first line from buffer, go back to start 

First we skip 17 lines. It's simple. The rest is complicated, but basically we keep a buffer of eight lines. We are just starting to print lines when the buffer is full, but we stop printing when we get to the end of the file, so at the end there are still eight lines left in the buffer that we did not print - in other words, we deleted them.