My goal is to convert external input sources to a common internal UTF-8 encoding, as it is compatible with many libraries that I use (like RE2) and is compact. Since I don’t need to do line cutting except for pure ASCII, UTF-8 is the perfect format for me. Now from the external input formats that I have to decode includes UTF-16.
To test the reading of UTF-16 (either high or low) in C ++, I converted the test file UTF-8 to UTF-16 LE and UTF-16 BE. The file is a simple gibberish in CSV format with many different source languages (English, French, Japanese, Korean, Arabic, Spanish, Thai) to create a fairly complex file:
"This","佐藤 幹夫","Mêmes","친구" "ภควา"," كيبورد للكتابة بالعربي","ウゥキュ,","🛂"
UTF-8 Example
Now parsing this file encoded in UTF-8 using the following code gives the expected result (I understand that this example is mostly artificial, since my system encoding is UTF-8, and therefore no actual conversion to wide characters, and then back for bytes):
#include <sstream> #include <locale> #include <iostream> #include <fstream> #include <codecvt> std::wstring readFile(const char* filename) { std::wifstream wif(filename, std::ios::binary); wif.imbue(std::locale(wif.getloc(), new std::codecvt_utf8<wchar_t, 0x10ffff>)); std::wstringstream wss; wss << wif.rdbuf(); return wss.str(); } int main() { std::wstring read = readFile("utf-8.csv"); std::cout << read.size() << std::endl; using convert_type = std::codecvt_utf8<wchar_t>; std::wstring_convert<convert_type, wchar_t> converter; std::string converted_str = converter.to_bytes( read ); std::cout << converted_str; return 0; }
When the file compiles and runs (on Linux, therefore the UTF-8 system encoding), I get the following output:
$ g++ utf8.cpp -o utf8 -std=c++14 $ ./utf8 73 "This","佐藤 幹夫","Mêmes","친구" "ภควา"," كيبورد للكتابة بالعربي","ウゥキュ,","🛂"
UTF-16 Example
However, when I try to use a similar example with UTF-16, I get a truncated file, despite loading the files correctly in text editors, Python, etc.
#include <fstream> #include <sstream> #include <iostream> #include <locale> #include <codecvt> #include <string> std::wstring readFile(const char* filename) { std::wifstream wif(filename, std::ios::binary); wif.imbue(std::locale(wif.getloc(), new std::codecvt_utf16<wchar_t, 0x10ffff>)); std::wstringstream wss; wss << wif.rdbuf(); return wss.str(); } int main() { std::wstring read = readFile("utf-16.csv"); std::cout << read.size() << std::endl; using convert_type = std::codecvt_utf8<wchar_t>; std::wstring_convert<convert_type, wchar_t> converter; std::string converted_str = converter.to_bytes( read ); std::cout << converted_str; return 0; }
When the file compiles and runs (on Linux, so the system encoding is UTF-8), I get the following output for the small endian format:
$ g++ utf16.cpp -o utf16 -std=c++14 $ ./utf16 19 "This","PO
For the big-endian format, I get the following:
$ g++ utf16.cpp -o utf16 -std=c++14 $ ./utf16 19 "This","OP
Interestingly, the CJK character must be part of the Basic Multilingual Plane, but obviously not properly converted, and the file has been truncated before. The same problem occurs with line building.
Other resources
I have checked the following resources before, this answer is most noteworthy, as well as this answer . None of their decisions turned out to be fruitful for me.
Other features
LANG = en_US.UTF-8 gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)
Any other details, and I will be happy to provide them. Thanks.
edits
Adrian mentioned in the comments should provide the hexdump that is shown for "utf-16le", a UTF-16 restricted encoding file:
0000000 0022 0054 0068 0069 0073 0022 002c 0022 0000010 4f50 85e4 0020 5e79 592b 0022 002c 0022 0000020 004d 00ea 006d 0065 0073 0022 002c 0022 0000030 ce5c ad6c 0022 000a 0022 0e20 0e04 0e27 0000040 0e32 0022 002c 0022 0020 0643 064a 0628 0000050 0648 0631 062f 0020 0644 0644 0643 062a 0000060 0627 0628 0629 0020 0628 0627 0644 0639 0000070 0631 0628 064a 0022 002c 0022 30a6 30a5 0000080 30ad 30e5 002c 0022 002c 0022 d83d dec2 0000090 0022 000a 0000094
qexyn mentioned removing the std::ios::binary flag, which I tried but didn't change anything.
Finally, I tried using iconv to make sure these are valid files using both the command line utility and the C module.
$ iconv -f="UTF-16BE" -t="UTF-8" utf-16be.csv "This","佐藤 幹夫","Mêmes","친구" "ภควา"," كيبورد للكتابة بالعربي","ウゥキュ,","🛂"
Obviously, iconv has no problems with the source files. This leads me to use iconv as it is cross-platform, easy to use and tested, but if anyone has an answer with a standard library, I will gladly accept it.