How can I read FIrst efficiently Multiple lines from many files in Delphi

Question

I have a Find Files feature in my program that will find text files with the .ged index that my program reads. I show the results found in a window similar to explorer, which looks like this:

I use standard FindFirst / FindNext methods and it works very fast. The 584 files shown above will be found and displayed within a few seconds.

Now I need to add two columns to the display, which shows the "Source" and "Version" that are contained in each of these files. This information is usually found in the first 10 lines of each file in lines that look like this:

1 SOUR FTM 2 VERS Family Tree Maker (20.0.0.368) 

Now I have no problem dealing with this very quickly, and this is not what I am asking.

I need help, just how to quickly load the first 10 or so lines from these files so that I can analyze them.

I tried to make StringList.LoadFromFile, but loading large files takes too much time, for example, above 1 MB.

Since I only need the first 10 lines, how should I get them?

I am using Delphi 2009, and my input files may or may not be Unicode, so this should work for any encoding.

---

Follow-up: Thanks Antonio,

I ended up working fine:

 var CurFileStream: TStream; Buffer: TBytes; Value: string; Encoding: TEncoding; try CurFileStream := TFileStream.Create(folder + FileName, fmOpenRead); SetLength(Buffer, 256); CurFileStream.Read(Buffer[0], 256); TEncoding.GetBufferEncoding(Buffer, Encoding); Value := Encoding.GetString(Buffer); ... (parse through Value to get what I want) ... finally CurFileStream.Free; end; 

Answer 1

Use TFileStream and use the Read method to read the number of bytes needed. Here is an example of reading bitmap information, which is also saved when the file starts.

http://www.delphidabbler.com/tips/19

Answer 2

Just open the file yourself to read the blocks (without using the built-in TStringList functions) and read the first block of the file, and then you can, for example, load this block into a string list with strings. SetText () (if you're using block functions) or just strings. LoadFromStream () if you load blocks using streams.

I would just go with the file functions FileRead / FileWrite and load the block into the buffer. You can also use the similair winapi functions, but this is just more code for no reason.

The OS reads files in blocks of at least 512 bytes in size on almost any platform / file system, so you can read 512 bytes first (and hope you get all 10 lines, which will be true if your lines are generally short enough). It will be (practically) as fast as reading 100 or 200 bytes.

Then, if you notice that your string objects only have less than 10 lines, just read the next 512-byte block and try to parse again. (Or just go with blocks of 1024, 2048, etc., On many systems, this will probably be as fast as 512 blocks, since file system clusters are usually larger than 512 bytes in size).

PS. In addition, using streams or asynchronous functions in the functions of the winapi file (CreateFile, etc.), you can load this data from files asynchronously, and the rest of your application works. In particular, the interface does not freeze while reading large directories.

This will speed up the loading of your information (since the list of files will be downloaded directly, and then after a few milliseconds the rest of the information will be displayed), although it does not actually increase the real reading speed.

Do this only if you have tried other methods, and it seems to you that you need an additional impulse.

Answer 3

You can use TStreamReader to read single lines from any TStream object, such as TFileStream . For even faster file I / O, you can use memory mappings using TCustomMemoryStream .

Answer 4

Ok, I deleted my first answer. Using the first Remy sentence above, I tried again with the embedded material. I do not like that you need to create and release two objects. I think I will make my own class to wrap this:

 var fs:TFileStream; tr:TTextReader; filename:String; begin filename := 'c:\temp\textFileUtf8.txt'; fs := TFileStream.Create(filename, fmOpenRead); tr := TStreamReader.Create(fs); try Memo1.Lines.Add( tr.ReadLine ); finally tr.Free; fs.Free; end; end; 

If someone is interested in what I had here before, he has a problem not working with Unicode files.

Answer 5

Sometimes the oldschool pascal style is not so bad. Although accessing a file without accessing a window does not seem to be very popular, ReadLn(F,xxx) still works pretty well in situations like yours.

The code below loads the information (file name, source and version) into TDictionary so that you can easily view it or use the list in virtual mode and look in this list when ondata even fires.

Warning: the code below does not work with Unicode.

 program Project101; {$APPTYPE CONSOLE} uses IoUtils, Generics.Collections, SysUtils; type TFileInfo=record FileName, Source, Version:String; end; function LoadFileInfo(var aFileInfo:TFileInfo):Boolean; var F:TextFile; begin Result := False; AssignFile(F,aFileInfo.FileName); {$I-} Reset(F); {$I+} if IOResult = 0 then begin ReadLn(F,aFileInfo.Source); ReadLn(F,aFileInfo.Version); CloseFile(F); Exit(True) end else WriteLn('Could not open ', aFileInfo.FileName); end; var FileInfo:TFileInfo; Files:TDictionary<string,TFileInfo>; S:String; begin Files := TDictionary<string,TFileInfo>.Create; try for S in TDirectory.GetFiles('h:\WINDOWS\system32','*.xml') do begin WriteLn(S); FileInfo.FileName := S; if LoadFileInfo(FileInfo) then Files.Add(S,FileInfo); end; // showing file information... for FileInfo in Files.Values do WriteLn(FileInfo.Source, ' ',FileInfo.Version); finally Files.Free end; WriteLn; WriteLn('Done. Press any key to quit . . .'); ReadLn; end.