Delete the first N lines of the file on the unix command line

Question

I am trying to delete the first 37 lines from a very large file. I started trying sed and awk, but they seem to require copying data to a new file. I am looking for a "delete lines in place" method, which unlike sed -i does not make copies of any type, but simply deletes lines from an existing file.

Here is what I did ...

 awk 'NR > 37' file.xml > 'f2.xml' sed -i '1,37d' file.xml 

Both of them seem to be making a complete copy. Is there another simple CLI that can do this quickly without a full crawl of the document?

Answer 1

There is no easy way to do in-place editing using UNIX utilities, but there is one solution for modifying in-place files that you can modify to work for you (courtesy of Robert Bonomy at https://groups.google.com/forum/ #! topic / comp.unix.shell / 5PRRZIP0v64 ):

 bytes=$(head -37 "$file" |wc -c) dd if="$file" bs="$bytes" skip=1 conv=notrunc of="$file" 

The final file should be $count bytes less than the original (since the goal was to remove the $count bytes from the beginning), so to finish we must delete the last bytes of $count . We use conv=notrunc above to make sure that the file is not completely cleaned, but only truncated (see, for example, below). On a GNU system such as Linux, subsequent truncation can be done with:

 truncate -s "-$bytes" "$file" 

For example, to remove the first 5 lines from this file of 12 lines

 $ wc -l file 12 file $ cat file When chapman billies leave the street, And drouthy neibors, neibors, meet; As market days are wearing late, And folk begin to tak the gate, While we sit bousing at the nappy, An' getting fou and unco happy, We think na on the lang Scots miles, The mosses, waters, slaps and stiles, That lie between us and our hame, Where sits our sulky, sullen dame, Gathering her brows like gathering storm, Nursing her wrath to keep it warm. 

First, use dd to remove the target 5 lines (actually the $ bytes bytes) from the beginning of the file and copy the rest from the end to the foreground, but leave the ending bytes of $ bytes as they are:

 $ bytes=$(head -5 file |wc -c) $ dd if=file bs="$bytes" skip=1 conv=notrunc of=file 1+1 records in 1+1 records out 253 bytes copied, 0.0038458 s, 65.8 kB/s $ wc -l file 12 file $ cat file An' getting fou and unco happy, We think na on the lang Scots miles, The mosses, waters, slaps and stiles, That lie between us and our hame, Where sits our sulky, sullen dame, Gathering her brows like gathering storm, Nursing her wrath to keep it warm. s, waters, slaps and stiles, That lie between us and our hame, Where sits our sulky, sullen dame, Gathering her brows like gathering storm, Nursing her wrath to keep it warm. 

and then use truncate to remove these remaining bytes from the end:

 $ truncate -s "-$bytes" "file" $ wc -l file 7 file $ cat file An' getting fou and unco happy, We think na on the lang Scots miles, The mosses, waters, slaps and stiles, That lie between us and our hame, Where sits our sulky, sullen dame, Gathering her brows like gathering storm, Nursing her wrath to keep it warm. 

If we tried the above without dd... conv=notrunc :

 $ wc -l file 12 file $ bytes=$(head -5 file |wc -c) $ dd if=file bs="$bytes" skip=1 of=file dd: file: cannot skip to specified offset 0+0 records in 0+0 records out 0 bytes copied, 0.0042254 s, 0.0 kB/s $ wc -l file 0 file 

Check out the Google Groups thread I referenced for other suggestions and information.

Answer 2

The semantics of a Unix file do not trim the front of the file.

All decisions will be based either on:

• Reading a file into memory and then writing it ( ed , ex , other editors). This should be good if your file is 1 GB or if you have a lot of RAM.
• Recording a second copy and optional replacement of the original ( sed -i , awk / tail > foo ). This is great if you have enough free disk space for a copy, and don't mind waiting.

If the file is too large for any of them to work for you, you can work around it depending on what your file is reading.

Perhaps your reader skips comments or blank lines? If so, you can process a message that the reader ignores, make sure that it has as many bytes as the first 37 lines in your file, and overwrite the beginning of the file with dd if=yourdata of=file conv=notrunc .

Answer 3

ed is a standard editor:

 ed -s file <<< $'1,37d\nwq' 

Answer 4

A copy should be created at some point - why not while reading the “modified” file; streaming a modified copy instead of saving it?

What I think is to create a named pipe "file2" that is the result of the same awk 'NR> 37' file.xml or something else; then someone who reads file2 will not see the first 37 lines.

The downside is that it will run awk every time the file is processed, so this is only possible if it is rarely read.