Read UTF-8 files correctly with PowerShell

Question

The following situation:

• PowerShell script creates a UTF-8 encoded file
• The user may or may not edit the file, possibly losing the specification, but must save the encoding as UTF-8 and possibly change the line separators
• The same PowerShell script reads the file, adds some more content and writes all of it as UTF-8 back to the same file
• This can be repeated many times.

With Get-Content and Out-File -Encoding UTF8 , I have trouble reading it. He stumbled upon a specification that he had written earlier (putting it in the content, breaking my parsing regular expression), doesn't use UTF-8 encoding, and even removes line breaks in the original part of the content.

I need a function that can read any UTF-8 encoded file, ignore and delete the specification, and not modify the contents. What should i use?

Update

I added a small test script that shows what I'm trying to do and what happens instead.

 # Read data if exists $data = "" $startRev = 1; if (Test-Path test.txt) { $data = Get-Content -Path test.txt if ($data -match "^[0-9-]{10} - r([0-9]+)") { $startRev = [int]$matches[1] + 1 } } Write-Host Next revision is $startRev # Define example data to add $startRev = $startRev + 10 $newMsgs = "2014-04-01 - r" + $startRev + "`r`n`r`n" + ` "Line 1`r`n" + ` "Line 2`r`n`r`n" # Write new data back $data = $newMsgs + $data $data | Out-File test.txt -Encoding UTF8 

After starting several times, new sections should be added to the beginning of the file, existing content should not be changed in any way (it currently loses line breaks), and no additional new lines should be added at the end of the file (it seems to happen sometimes) .

Instead, a second run causes an error.

Answer 1

If the file should be UTF8, why don't you try reading its UTF8 decoding:

 Get-Content -Path test.txt -Encoding UTF8 

Answer 2

Indeed, JPBlanc is right. If you want it to read as UTF8, specify when the file is being read.

On the other hand, you lose formatting here with the material [String] + [String]. Not to mention that your regular match doesn’t work. Check the regex search changes and the changes made to $ newMsgs and how the data is output to the file.

 # Read data if exists $data = "" $startRev = 1; if (Test-Path test.txt) { $data = Get-Content -Path test.txt #-Encoding UTF8 if($data -match "\br([0-9]+)\b"){ $startRev = [int]([regex]::Match($data,"\br([0-9]+)\b")).groups[1].value + 1 } } Write-Host Next revision is $startRev # Define example data to add $startRev = $startRev + 10 $newMsgs = @" 2014-04-01 - r$startRev`r`n`r`n Line 1`r`n Line 2`r`n`r`n "@ # Write new data back $newmsgs,$data | Out-File test.txt -Encoding UTF8 

Answer 3

Get-Content does not seem to process UTF files without specification (if you omit the Encoding flag). System.IO.File.ReadLines seems like an alternative, examples:

 PS C:\temp\powershellutf8> $a = Get-Content .\utf8wobom.txt PS C:\temp\powershellutf8> $b = Get-Content .\utf8wbom.txt PS C:\temp\powershellutf8> $a2 = Get-Content .\utf8wbom.txt -Encoding UTF8 PS C:\temp\powershellutf8> $a ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ <== This doesnt seem to be right at all PS C:\temp\powershellutf8> $b ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ PS C:\temp\powershellutf8> $a2 ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ PS C:\temp\powershellutf8> PS C:\temp\powershellutf8> $c = [IO.File]::ReadLines('.\utf8wbom.txt'); PS C:\temp\powershellutf8> $c ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ PS C:\temp\powershellutf8> $d = [IO.File]::ReadLines('.\utf8wobom.txt'); PS C:\temp\powershellutf8> $d ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ <== Works!