Efficient way to write to a text file in VB.NET

Question

We have the information that we need to write (about 18 kilobytes) to a .txt file stored on one of our network drives. A file is rewritten about once every 15 minutes, but it is read almost at least every second. We are currently using StreamWriter to record a file.

The file server is located in a remote location, and ping with a reverse pass varies from less than 1 μs to 15 ms.

The problem is that it only takes six seconds to write the contents to a file, which is certainly too long even after we look at the network speed.

So I'm just wondering if there is an efficient way to write a file using VB.NET to improve performance? Java has a very nice BufferedOutputStream tool, which, unfortunately, is not available in VB.NET (or I just could not find it).

Answer 1

The fastest option:

Collect all the text in one large line, and then use System.IO.File.WriteAllText(text) .

Answer 2

There are a couple of things. But I found that using the IO.File.AppendText function significantly speeds up the writing process.

In addition, using the System.Text.Stringbuilder class as opposed to the traditional line (provided that you link the text to write to the file) improves the speed even more.

See Speeding up file writing .

Answer 3

This function was written to be read from the database and output to a text file. Please feel free to use it as a starting point.

  Sub MakeFile(ByVal Obj As Object) Dim CountRow As Integer = 0 Dim TableName As String = CType(Obj, String())(0) Dim CommandText As String = CType(Obj, String())(1) Dim csvFileName As String = InitilizationSettings.DirectoryPath & TableName & ".txt" If File.Exists(csvFileName) Then File.Delete(csvFileName) End If Dim buffer() As Byte = {255} Dim FileObject As New FileStream(csvFileName, FileMode.OpenOrCreate) Dim MStream As New MemoryStream() Dim StreamWriterObj As New StreamWriter(MStream) Dim sb As New System.Text.StringBuilder Dim x As Integer = 0 Dim reader As SqlDataReader Dim cmd As New SqlCommand() Dim conn As New SqlConnection(IOUtilities.GetConnectionString()) conn.Open() With cmd .CommandText = CommandText .CommandTimeout = 1200 .CommandType = CommandType.Text .Connection = conn End With reader = cmd.ExecuteReader() Do While reader.Read() 'System.Console.Write("Loading rows to memory.../" & vbCr) sb.Append(Chr(34)) sb.Append(reader.Item(0)) 'System.Console.Write("Loading rows to memory...|" & vbCr) sb.Append(Chr(34)) For i = 1 To reader.FieldCount - 1 sb.Append(",") sb.Append(Chr(34)) sb.Append(reader.Item(i)) sb.Append(Chr(34)) Next 'System.Console.Write("Loading rows to memory...\" & vbCr) sb.AppendLine() 'Write every 10000 rows of data to the file from the buffer If x = 50000 Then StreamWriterObj.Write(sb.ToString().ToCharArray()) MStream.Seek(0, SeekOrigin.Begin) MStream.WriteTo(FileObject) sb = New StringBuilder() CountRow = CountRow + x x = 0 End If 'System.Console.Write("Loading rows to memory...-" & vbCr) x = x + 1 'LogEvents("Dumped " & strFileName & " to " & GetFilePath() & " at " & Now.ToString & vbCrLf & vbCrLf) Loop conn.Close() reader.Close() 'Write any remaining data from the buffer to the file StreamWriterObj.Write(sb.ToString().ToCharArray()) MStream.WriteTo(FileObject) FileObject.Close() System.Console.WriteLine(String.Format(vbCrLf & "Finished writing data to {1}", CountRow, csvFileName)) End Sub 

Answer 4

Consider them in turn:

• Create file locally
• Copy it to a remote folder with a temporary extension
• Rename the deleted file to the original file name

Step 2 and 3 are as follows (using System.IO):

 string OriginalExtension = ".ok", TemporaryExtension = ".dat"; string tmpFileRemote = RemoteFile.Replace(TemporaryExtension, OriginalExtension); File.Copy(fileName, RemoteFile, true); File.Copy(RemoteFile, tmpFileRemote, true); File.Delete(RemoteFile); 

The first File.Copy file takes time. But since it does not block the actual file that people use, it does not block. The second File.Copy actually just renames the file and replaces the real file with the newly loaded one. File.Delete deletes the downloaded temporary file.

Hope this helps.

Answer 5

Old thread, but I think there is still something to add:

Write all the text at once using:
System.IO.File.WriteAllText (path As String, contents As String).

However, it will still take time to write the file if it is being recorded by a slow connection to a remote location. To prevent users from reading a partially written file, you must write data to a temporary file on a remote server. After all the data has been written, copy the temporary file on top of the old file and then delete the temporary file.

If you want to be sure that no one is reading a partially written file, you can delete the old file and then move / rename the temporary file to replace the deleted file. In this case, you must make sure that the client programs gracefully process files that do not detect errors that will inevitably occur from time to time.

Answer 6

First think about writing to a local drive, and then move this file to a network drive.

Answer 7

 Sub writefile() Dim file As System.IO.StreamWriter file = My.Computer.FileSystem.OpenTextFileWriter("N:\GeneratedNumber.txt", False) file.WriteLine("Player1 Skill is " & Skill(0)) file.WriteLine("Player1 Strength is " & Skill(1)) file.WriteLine("Player2 Skill is " & Skill(2)) file.WriteLine("Player2 Strength is " & Skill(3)) file.Close() End Sub