How can I decode UTF-16 data in Perl when I don't know the byte order?

Question

If I open the file (and specify the encoding directly):

open(my $file,"<:encoding(UTF-16)","some.file") || die "error $!\n"; while(<$file>) { print "$_\n"; } close($file); 

I can read the contents of the file beautifully. However, if I do this:

 use Encode; open(my $file,"some.file") || die "error $!\n"; while(<$file>) { print decode("UTF-16",$_); } close($file); 

I get the following error:

 UTF-16:Unrecognised BOM d at F:/Perl/lib/Encode.pm line 174 

How can I make it work with decode ?

EDIT: here are the first few bytes:

 FF FE 3C 00 68 00 74 00 

Answer 1

If you just specify “UTF-16,” Perl will look for the byte character (BOM) to figure out how to parse it. If there is no specification, it will explode. In this case, you must specify an Encode that is in byte order, specifying either "UTF-16LE" for little-endian or "UTF-16BE" for big-endian.

Something else is happening with your situation, but it's hard to say without seeing the data that you have in the file. I get the same error with both fragments. If I don't have a specification and I don't specify the byte order, my Perl complains anyway. What Perl do you use and what platform do you have? Does your platform have your own source code for your file? I think the behavior that I see is correct according to the docs.

Furthermore, you cannot just read the string in some unknown encoding (no matter what was the default for Perl), and then pass it to decode . You may be in the middle of a multibyte sequence. You should use Encode::FB_QUIET to save the portion of the buffer that you could not decode and add to the following piece of data:

 open my($lefh), '<:raw', 'text-utf16.txt'; my $string; while( $string .= <$lefh> ) { print decode("UTF-16LE", $string, Encode::FB_QUIET) } 

Answer 2

You need to specify UTF-16BE or UTF-16LE. See http://perldoc.perl.org/Encode/Unicode.html#Size%2c-Endianness%2c-and-BOM

Answer 3

What you are trying to make impossible.

You read lines of text without specifying an encoding, so each byte containing a newline character (by default \x0a ) ends the line. But this newline character may well be in the middle of the UTF-16 character, in which case your next line cannot be decrypted. If your data is UTF-16LE, this will happen in all time channels \x0a \x00 . If you have UTF16-BE, you can get lucky (newline characters \x00 \x0a ) until you get the character with \x0a in the high byte.

So, do not do this, open the file in the correct encoding.