Should I use TMemoryStream as an effective buffer before writing to a file?

Question

I use D6 Professional and you need to create a text file in a specific format from many small lines that are already in memory. For performance reasons, I consider using TMemoryStream to sort file data and then write it to disk in one go through TFileStream.

But I have a half-forgotten memory (possibly from pre-D6 days) of reading somewhere that TMemoryStream is inefficient, especially after it has reached its size. My Delphi (and Windows API) skill is not good enough to check the Classes.pas code for myself.

(OFFTOPIC) especially this code: (line 5152 of Classes.pas):
NewCapacity: = (NewCapacity + (MemoryDelta - 1)), not (MemoryDelta - 1);
(/ OFFTOPIC)

Adding to my concern is that the output from a related question is Using a MemoryStream for Writing in XML
was not for using TMemoryStream, but did not say why - because of the TMemoryStream itself or because of the presence of sufficient buffering in the TFileStream device driver or I / O device or simply in particular the code in question.

Thanks for any advice.
Yours faithfully,
PhilW.

Answer 1

A regular TFileStram also performs buffering, which is enough to optimize I / O. Putting a MemoryStream in front only increases overhead.

Answer 2

TFileStream itself does not perform buffering, which is processed by the OS and, as a rule, is sufficient for most purposes.

My suggestion was to create a method to write your data to the stream, and then pass the TSTream parameter to this method. Thus, you can easily test various options without affecting your program.

For instance:

Procedure TForm1.StreamMyObjects(aStream : tStream); begin aStream.Write( MyString[1], Length( MyString ) * SizeOf( Char )); aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char )); aStream.Write( MyOtherString[1], Length( MyOtherString ) * SizeOf( Char )); aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char )); end; 

In JCL, as mentioned earlier, there is a TJclBufferedStream, which you can then check to see if there is any performance advantage that will depend on what you write and how much you write. For example, the following will check TFileStream and tJCLBufferedStream to find out what the differences are (yes, I know I miss TRY / FINALLY):

 var fstm : tFileSTream; fBufStm : tJCLBufferedStream; iTicks : Cardinal; fModes : word; // for SO formatting. begin fModes := fmOpenReadWrite or fmCreate or fmShareExclusive; iTicks := GetTickCount; fstm := tFilestream.create('test1.txt',fModes); StreamMyObjects( fStm ); fstm.free; ShowMessage('TEST1='+IntToSTr(GetTickCount-iTicks)); iTicks := GetTickCount; fstm := tFilestream.create('test2.txt',fModes); fBufStm := tJclBufferedStream.create( fStm ); StreamMyObjects( fBufStm ); fBufStm.free; fstm.free; ShowMessage('TEST2='+IntToSTr(GetTickCount-iTicks)); end; 

in my test, the following procedure:

 procedure TForm1.StreamMyObjects(aSTream: tStream); var St : string; ix : integer; begin for ix := 0 to 10000 do begin St := 'This is a string which is written to a stream. ' + IntToStr(ix); aStream.Write(st[1], Length(st) * SizeOf(Char) ); end; end; 

returned 47 for tFilestream and 16 for tJCLBufferedStream. Without a loop, time is short, so you need to test your data ... and how much you write.

Answer 3

You can use TStringList and call the SaveToFile method. TStringList has the advantage that memory is not copied when a row is added to it.

Another option is the TJclBufferedStream JJD class.

Answer 4

As already mentioned, adding a TMemoryStream in front of a TFileStream will probably not lead you to where you need to get.

In a Delphi Magazine article, Julian Bucknall released a set of free modules and classes to add functionality to TStream objects. I find them extremely useful in most of my transactions with TStream objects (especially TFileStream).

The following is a link from a Google code search and provides direct access to the aaStrms.pas file. To use the aaStrms.pas file, you will need other units (aaIntDeq.pas, aaIntList.pas, aaRegEx.pas and aaStrBld.pas). These are not visual classes, so all you have to do is enable the block and instantiate the class.

http://www.google.com/codesearch/p?hl=en#7-hAM65i1Xc/disks/dmag70.zip|alfresco/AAStrms.pas&q=lang:pascal%20aaStrms

I would suggest that the TaaWriteBufferFilter class is the one you want to use.

NTN

Ryan

Answer 5

I would investigate whether to create your own buffer of a limited size. It writes small bits slowly to TFileStream (especially when write-delayed recording is disabled on the storage volume), because the disk driver will need to find a sector to write to each bit. Therefore, if you use a buffer of several sectors in size (a safe guess is 512 KB or 1 MB), and write a buffer when it is full, this will speed up your work without worrying about what to do with the giant bloat TMemoryStream.