Quickly read the last line of a text file?

Question

What is the fastest and most efficient way to read the last line of text from a [very, very large] file in Java?

Answer 1

Take a look at my answer to a similar question for C # . The code would be very similar, although coding support is slightly different in Java.

Basically, this is not a very simple thing in general. As MSalter points out, UTF-8 makes it easy to specify \r or \n , since the representation of these UTF-8 characters is the same as ASCII, and these bytes will not occur in a multibyte character.

So, take a 2K buffer (say) and progressively read backward (skip to 2K before you were before, read the next 2K), checking for line endings. Then go to the right place in the stream, create an InputStreamReader on top and a BufferedReader on top of it. Then just call BufferedReader.readLine() .

Answer 2

Below are two functions, one of which returns the last non-empty line of a file without loading or iterating over the entire file, and the other, which returns the last N lines of the file without going through the entire file:

Which tail is the zoom right to the last character of the file, then a step back, character by character, recording what it sees until it finds a line break. As soon as it finds a line break, it exits the loop. Pays attention to what was written, and inserts it into a string and returns. 0xA is the new line, and 0xD is the carriage return.

If your lines end with \r\n or crlf or some other “double line new line new line”, you will need to specify n * 2 lines to get the last n lines, because they count 2 lines for each line.

 public String tail( File file ) { RandomAccessFile fileHandler = null; try { fileHandler = new RandomAccessFile( file, "r" ); long fileLength = fileHandler.length() - 1; StringBuilder sb = new StringBuilder(); for(long filePointer = fileLength; filePointer != -1; filePointer--){ fileHandler.seek( filePointer ); int readByte = fileHandler.readByte(); if( readByte == 0xA ) { if( filePointer == fileLength ) { continue; } break; } else if( readByte == 0xD ) { if( filePointer == fileLength - 1 ) { continue; } break; } sb.append( ( char ) readByte ); } String lastLine = sb.reverse().toString(); return lastLine; } catch( java.io.FileNotFoundException e ) { e.printStackTrace(); return null; } catch( java.io.IOException e ) { e.printStackTrace(); return null; } finally { if (fileHandler != null ) try { fileHandler.close(); } catch (IOException e) { /* ignore */ } } } 

But you probably don't need the last line, you need the last N lines, so use this instead:

 public String tail2( File file, int lines) { java.io.RandomAccessFile fileHandler = null; try { fileHandler = new java.io.RandomAccessFile( file, "r" ); long fileLength = fileHandler.length() - 1; StringBuilder sb = new StringBuilder(); int line = 0; for(long filePointer = fileLength; filePointer != -1; filePointer--){ fileHandler.seek( filePointer ); int readByte = fileHandler.readByte(); if( readByte == 0xA ) { if (filePointer < fileLength) { line = line + 1; } } else if( readByte == 0xD ) { if (filePointer < fileLength-1) { line = line + 1; } } if (line >= lines) { break; } sb.append( ( char ) readByte ); } String lastLine = sb.reverse().toString(); return lastLine; } catch( java.io.FileNotFoundException e ) { e.printStackTrace(); return null; } catch( java.io.IOException e ) { e.printStackTrace(); return null; } finally { if (fileHandler != null ) try { fileHandler.close(); } catch (IOException e) { } } } 

Call the above methods as follows:

 File file = new File("D:\\stuff\\huge.log"); System.out.println(tail(file)); System.out.println(tail2(file, 10)); 

Caution In the wild west of Unicode, this code may cause an error in the exit of this function. For example, "Mary" instead of "Mary." Symbols with hats, accents, hieroglyphs , etc. They can lead to an incorrect result, because accents are added after the character as modifiers. Inverse composite characters change the character’s identity when reversing. You will need to perform a full test verification in all languages ​​in which you plan to use this.

For more information about this Unicode U-turn issue, read the following: http://msmvps.com/blogs/jon_skeet/archive/2009/11/02/omg-ponies-aka-humanity-epic-fail.aspx

Answer 3

Apache Commons has an implementation using RandomAccessFile .

It is called ReversedLinesFileReader .

Answer 4

Using FileReader or FileInputStream will not work - you will need to use FileChannel or RandomAccessFile to scroll back from the end. Encodings will be a problem though, as John said.

Answer 5

You can easily change the code below to print the last line.

MemoryMappedFile to print the last 5 lines:

 private static void printByMemoryMappedFile(File file) throws FileNotFoundException, IOException{ FileInputStream fileInputStream=new FileInputStream(file); FileChannel channel=fileInputStream.getChannel(); ByteBuffer buffer=channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size()); buffer.position((int)channel.size()); int count=0; StringBuilder builder=new StringBuilder(); for(long i=channel.size()-1;i>=0;i--){ char c=(char)buffer.get((int)i); builder.append(c); if(c=='\n'){ if(count==5)break; count++; builder.reverse(); System.out.println(builder.toString()); builder=null; builder=new StringBuilder(); } } channel.close(); } 

RandomAccessFile to print the last 5 lines:

 private static void printByRandomAcessFile(File file) throws FileNotFoundException, IOException{ RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r"); int lines = 0; StringBuilder builder = new StringBuilder(); long length = file.length(); length--; randomAccessFile.seek(length); for(long seek = length; seek >= 0; --seek){ randomAccessFile.seek(seek); char c = (char)randomAccessFile.read(); builder.append(c); if(c == '\n'){ builder = builder.reverse(); System.out.println(builder.toString()); lines++; builder = null; builder = new StringBuilder(); if (lines == 5){ break; } } } } 

Answer 6

In C # you should set the position of the stream:

From: http://bytes.com/groups/net-c/269090-streamreader-read-last-line-text-file

 using(FileStream fs = File.OpenRead("c:\\file.dat")) { using(StreamReader sr = new StreamReader(fs)) { sr.BaseStream.Position = fs.Length - 4; if(sr.ReadToEnd() == "DONE") // match } } 

Answer 7

 try(BufferedReader reader = new BufferedReader(new FileReader(reqFile))) { String line = null; System.out.println("======================================"); line = reader.readLine(); //Read Line ONE line = reader.readLine(); //Read Line TWO System.out.println("first line : " + line); //Length of one line if lines are of even length int len = line.length(); //skip to the end - 3 lines reader.skip((reqFile.length() - (len*3))); //Searched to the last line for the date I was looking for. while((line = reader.readLine()) != null){ System.out.println("FROM LINE : " + line); String date = line.substring(0,line.indexOf(",")); System.out.println("DATE : " + date); //BAM!!!!!!!!!!!!!! } System.out.println(reqFile.getName() + " Read(" + reqFile.length()/(1000) + "KB)"); System.out.println("======================================"); } catch (IOException x) { x.printStackTrace(); } 

Answer 8

As far as I know, the fastest way to read the last line of a text file is to use the Apache FileUtils class, which is located in "org.apache.commons.io". I have a file with two million lines, and with this class it took me less than one second to find the last line. Here is my code:

 LineIterator lineIterator = FileUtils.lineIterator(newFile(filePath),"UTF-8"); String lastLine=""; while (lineIterator.hasNext()){ lastLine= lineIterator.nextLine(); } 

Answer 9

 Path path = Paths.get(pathString); List<String> allLines = Files.readAllLines(path); return allLines.get(allLines.size()-1);