Get the last n lines or bytes of a huge file on Windows (e.g. Unix tail). Avoid laborious options

Question

I need to get the last n lines of huge files (1-4 Gb) in Windows 7. Due to corporate restrictions, I cannot run any command that is not built-in. The problem is that all the solutions I found seem to read the entire file, so they are very slow.

Can this be done quickly?

Notes:

• I managed to get the first n lines, fast.
• This is normal if I get the last n bytes. (I have used this https://stackoverflow.com/a/3189605/ for the first n bytes)

The solutions are here. The Unix tail equivalent command in Windows Powershell does not work. Using -wait does not speed -wait up. I don't have -tail (and I don't know if it will work fast).

PS: For head and tail there are quite a few related issues, but not focused on the speed issue. Therefore, useful or accepted answers there may be useful. For example.

Windows equivalent of the tail command

CMD.EXE package script to display the last 10 lines from a txt file

Extract N lines from file using single window command

https://serverfault.com/questions/490841/how-to-display-the-first-n-lines-of-a-command-output-in-windows-the-equivalent

powershell to get the first x MB of file

https://superuser.com/questions/859870/windows-equivalent-of-the-head-c-command

Answer 1

How about this (reads the last 8 bytes for a demonstration):

 $fpath = "C:\10GBfile.dat" $fs = [IO.File]::OpenRead($fpath) $fs.Seek(-8, 'End') | Out-Null for ($i = 0; $i -lt 8; $i++) { $fs.ReadByte() } 

---

UPDATE To interpret bytes as a string (but be sure to select the correct encoding - UTF8 is used here):

 $N = 8 $fpath = "C:\10GBfile.dat" $fs = [IO.File]::OpenRead($fpath) $fs.Seek(-$N, [System.IO.SeekOrigin]::End) | Out-Null $buffer = new-object Byte[] $N $fs.Read($buffer, 0, $N) | Out-Null $fs.Close() [System.Text.Encoding]::UTF8.GetString($buffer) 

---

UPDATE 2 . To read the last M lines, we will read the file in parts until the result is no longer char sequences:

 $M = 3 $fpath = "C:\10GBfile.dat" $result = "" $seq = "`r`n" $buffer_size = 10 $buffer = new-object Byte[] $buffer_size $fs = [IO.File]::OpenRead($fpath) while (([regex]::Matches($result, $seq)).Count -lt $M) { $fs.Seek(-($result.Length + $buffer_size), [System.IO.SeekOrigin]::End) | Out-Null $fs.Read($buffer, 0, $buffer_size) | Out-Null $result = [System.Text.Encoding]::UTF8.GetString($buffer) + $result } $fs.Close() ($result -split $seq) | Select -Last $M 

Try playing with large $buffer_size - this is ideally equal to the expected average line length to make fewer disk operations. Also pay attention to $ seq - it could be \r\n or just \n . This is very dirty code without error handling and optimization.

Answer 2

If you have PowerShell 3 or higher, you can use the -Tail parameter for Get-Content to get the last lines of n .

 Get-content -tail 5 PATH_TO_FILE; 

In a 34 MB text file on my local SSD, this returned 1 millisecond versus 8.5 seconds for get-content |select -last 5

Answer 3

With the amazing answer of Aziz Kabyshev that solves the speed problem, and with some search engines, I ended up using this script

 $fpath = $Args[1] $fs = [IO.File]::OpenRead($fpath) $fs.Seek(-$Args[0], 'End') | Out-Null $mystr = '' for ($i = 0; $i -lt $Args[0]; $i++) { $mystr = ($mystr) + ([char[]]($fs.ReadByte())) } $fs.Close() Write-Host $mystr 

which I call from a batch file containing

 @PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '.\myscript.ps1' %1 %2" 

(thanks to How to run a PowerShell script from a batch file ).

Answer 4

This is not an answer, but a great comment in response to sancho.s's answer.

If you want to use small PowerShell scripts from a batch file, I suggest you use the method below, which simplifies and allows you to save all the code in the same batch file:

 @PowerShell ^ $fpath = %2; ^ $fs = [IO.File]::OpenRead($fpath); ^ $fs.Seek(-%1, 'End') ^| Out-Null; ^ $mystr = ''; ^ for ($i = 0; $i -lt %1; $i++) ^ { ^ $mystr = ($mystr) + ([char[]]($fs.ReadByte())); ^ } ^ Write-Host $mystr %End PowerShell% 

Answer 5

try

 Get-Content <file name> | Select-Object -Last 3 # it prints last 3 lines of the file