Pipeline text in an external program adds a trailing line

Question

I compared hash values ​​between several systems and was surprised to find that the values ​​of the PowerShells hash function are different from the values ​​of other terminals.

Linux terminals (CygWin, Bash for Windows, etc.) and the Windows command line display the same hash, where PowerShell displays a different hash value.

This was tested using SHA256, but found the same problem when using other algorithms such as md5.

Encoding update:

I tried changing the encoding of PShell, but had no effect on the returned hash values.

[Console]::OutputEncoding.BodyName iso-8859-1 [Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8 utf-8 

GitHub PowerShell Problem

https://github.com/PowerShell/PowerShell/issues/5974

Answer 1

TL; DR:

The key is to avoid the PowerShell pipeline in favor of the native shell to prevent the implicit addition of a trailing newline :

• If you are using your Unix- based command (using PowerShell Core):

 sh -c "printf %s 'string' | openssl dgst -sha256 -hmac authcode" 

^{printf %s is a portable alternative to echo -n . If the string contains ' chars., Double them or use a quote instead of `"...`" .}

• If you need to do this on Windows through cmd.exe , things get even more complicated, because cmd.exe does not support echo support without a final new line:

 cmd /c "<NUL set /p =`"string`"| openssl dgst -sha256 -hmac authcode" 

^{Please note that there must be no spaces before | . For an explanation and limitations of this solution, see this answer .}

Encoding problems will only occur if the string contains non-ASCII characters and you are running Windows PowerShell; in this case, first set $OutputEncoding to the encoding that the target utility expects, usually UTF-8: $OutputEncoding = [Text.Utf8Encoding]::new()

---

• PowerShell, like Windows PowerShell v5.1 / PowerShell Core v6.0.0, invariably adds a final new line when you send a line without one through the pipeline to an external utility that causes the differences you are seeing (this final new product will be LF only on platforms Unix and CRLF sequence on Windows).

  • You can track efforts to resolve this issue in this GitHub issue , open by OP.

• In addition, the PowerShell pipeline is consistently text-based when it comes to transferring data to external programs; PowerShell (.NET) strings embedded in UTF-16LE are transcoded based on the encoding stored in the $OutputEncoding automatic variable, which by default uses only ASCII encoding in Windows PowerShell and UTF-8 encoding in PowerShell Core (both on Windows and Unix-like platforms).

  • PowerShell Core discusses a change to feed raw byte streams between external programs.

• The fact that echo -n in PowerShell does not create a line without a trailing newline is therefore random for your problem; for completeness, here is the explanation:

  • echo is an alias for PowerShell Write-Output cmdlet, which - in the context of connecting to external programs - writes text to standard program input in the next pipeline segment (similar to Bash / cmd.exe echo ).
  • -n interpreted as the (unique) abbreviation for Write-Output -NoEnumerate .
  • -NoEnumerate only applies when writing multiple objects, so it has no effect here.
  • Therefore, in short: in PowerShell, echo -n "string" same as Write-Output -NoEnumerate "string" , which - since only one line is output - this is the same as Write-Output "string" , which, in its the queue is the same as just using "string" , relying on the implicit behavior of PowerShell output.
  • Write-Output does not have the ability to suppress the final new line, and even if this happened, using a pipeline to connect to an external program would add it back.

Answer 2

Linux terminals and PowerShell use different encodings. Thus, the real bytes generated by echo -n "string" are different. I tried this on my Linux Mint and Windows 10 PowerShell. Here is what I got:

Linux Mint:

 73 74 72 69 6E 67 

Windows 10:

 FF FE 73 00 74 00 72 00 69 00 6E 00 67 00 0D 00 0A 00 

Linux terminals seem to use UTF-8, and Windows PowerShell uses UTF-16 with a specification. Also in PowerShell, you cannot use the -n option to echo. Thus, the echo puts the newline characters \r\n ( 0D 00 0A 00 ) at the end of the line.

Edit: As mklement0 below, Windows PowerShell uses ASCII on pipelines by default.