Nokogiri does not allow XML with UTF-16 declaration (understanding and working around)

Question

Summary

Attempting to read and serialize XML documents encoded and declared in UTF-16 forces Nokogiri to generate garbage after a certain point.

• Is this a mistake, or is there a reasonable explanation for this?
• What is the best way to avoid this?

Environment

C:\>nokogiri -v # Nokogiri (1.5.5) --- warnings: [] nokogiri: 1.5.5 ruby: version: 1.9.3 platform: i386-mingw32 description: ruby 1.9.3p194 (2012-04-20) [i386-mingw32] engine: ruby libxml: binding: extension compiled: 2.7.7 loaded: 2.7.7 

More details

I have an XML file encoded using UTF-16 (LE) and it also includes a PI XML declaration at the top indicating that the encoding is UTF-16. To summarize, it looks like this:

 <?xml version="1.0" encoding="UTF-16" ?> <Foo> <Bar><![CDATA[ Lorem ipsum dolor ...about 3900 more bytes of content here... ]]></Bar> <Jim>Oh! Hello there.</Jim> </Foo> 

When I use Nokogiri to read this document, everything looks good:

 xml = File.open('Simplified.xml','rb:utf-16le',&:read) p xml.encoding # #<Encoding:UTF-16LE> p xml.valid_encoding? # true doc1 = Nokogiri.XML(xml,&:noblanks) xml1 = doc1.to_xml.encode('utf-8') p xml1.encoding # #<Encoding:UTF-8> p xml1.valid_encoding? # true 

However, the serialization output of the document fades out after a certain point:

 p xml1 # Correct contents of CDATA removed from the following output #=> "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n<Foo>\n <Bar><![CDATA[\n...\n\t]]></Bar>\n <Jim>Oh! Hello there.\uFFFE\u3C00\u0000\u2F00\u0000\u4A00\u0000\u6900\u0000\u6D00\u0000\u3E00\u0000\u0A00\u0000\u3C00\u0000\u2F00\u0000\u4600\u0000\u6F00\u0000\u6F00\u0000\u3E00\u0000\u0A00\u0000" 

_{(The limit is apparently related to the number of characters. I can add and remove a few words from Lorem ipsum text without changes, but deleting the text at a certain point suddenly corrects the output.) Sub>}

However, the Nokogiri document is not violated. I can independently serialize <Jim> successfully:

 puts doc1.at('Jim').to_xml.encode('utf-8') #=> <Jim>Oh! Hello there.</Jim> 

The only workaround I found was to remove the XML declaration at the top of the document before parsing it. In this case, everything works as desired:

 decl = '<?xml version="1.0" encoding="UTF-16" ?>'.encode('UTF-16LE') doc2 = Nokogiri.XML(xml.sub(decl,''),&:noblanks) puts doc2.to_xml.encode('utf-8') #=> <?xml version="1.0"?> #=> <Foo> #=> <Bar><![CDATA[ #=> Lorem ipsum dolor...and more... #=> ]]></Bar> #=> <Jim>Oh! Hello there.</Jim> #=> </Foo> 

Full XML

Here is the full file to check:

 <?xml version="1.0" encoding="UTF-16" ?> <Foo> <Bar><![CDATA[ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam ac augue arcu, eget laoreet lorem. Quisque ac augue velit. Integer consectetur suscipit vehicula. Etiam et convallis enim. Etiam varius massa sit amet lacus rhoncus varius in non ante. Sed dictum, metus eu bibendum ornare, ligula dui commodo urna, ut dignissim felis dolor eget nisl. Proin sit amet nisi nunc. Vestibulum a urna sed dui dignissim blandit nec vel enim. Vivamus tincidunt nulla id dui hendrerit hendrerit. Aliquam neque orci, luctus sit amet fringilla eu, varius vitae diam. Suspendisse varius rutrum lorem eget malesuada. Sed dapibus dapibus nisl, in cursus ante lacinia non. Aenean id sagittis ipsum. Suspendisse elit nunc, porta sit amet blandit ut, laoreet sed est. Nunc eget sem vitae nisl elementum ullamcorper ut sit amet urna. Sed ligula quam, fringilla in facilisis tincidunt, vehicula in nisi. Maecenas a augue in augue semper scelerisque sit amet ut arcu. Praesent hendrerit, enim in elementum ornare, lorem nisi euismod dolor, sit amet ornare mi sem sodales lacus. Fusce et tempor mauris. In non quam nisl, non consequat diam. Duis sit amet massa ultrices massa cursus iaculis. Nunc ullamcorper malesuada sem dignissim semper. Fusce aliquet lacus quis nisi tincidunt sodales. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque posuere commodo aliquet. Aliquam blandit vestibulum facilisis. Sed pellentesque viverra dignissim. Etiam est lacus, mollis eu pretium vitae, lacinia eleifend augue. Mauris vitae quam nisl. In venenatis nunc ac eros elementum cursus. Sed a metus sit amet nunc euismod condimentum id non orci. Curabitur velit turpis, lacinia non eleifend sed, rhoncus id est. Fusce ut massa dolor, ut sodales odio. Donec aliquam convallis tellus, eu pharetra tortor iaculis non. Integer imperdiet feugiat ipsum a gravida. Mauris sapien ipsum, ultricies ac placerat ut, imperdiet eu justo. Quisque quis consectetur velit. Etiam facilisis sapien nec enim tincidunt pulvinar. Duis fermentum faucibus felis, sed consequat libero pretium at. Phasellus nibh purus, suscipit in vestibulum vel, blandit at leo. Suspendisse placerat elit sed enim bibendum vel hendrerit mauris pretium. Maecenas ut lacus eu nisi euismod pretium. Aliquam feugiat felis id massa aliquam pharetra sed non eros. Morbi interdum molestie iaculis. Curabitur varius ante ac dui dapibus non laoreet risus blandit. Nunc sit amet magna lacus. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Phasellus egestas nunc sed turpis imperdiet a rhoncus massa aliquam. Nulla facilisi. Phasellus sit amet neque felis, nec vestibulum massa. Donec luctus fringilla dolor et gravida. Phasellus euismod lectus eget elit hendrerit non vehicula tellus venenatis. Phasellus sit amet ligula et purus dignissim feugiat at vitae libero. Proin ut tortor eros, quis laoreet lectus. Quisque nec urna mattis ante gravida fermentum eu at nibh. Phasellus sapien elit, tincidunt quis laoreet id, lobortis sed magna. Aliquam pulvinar erat eu sapien pretium bibendum. Maecenas eleifend, leo quis sodales tincidunt, leo felis tristique dolor, vitae ultrices neque felis ut metus. Etiam dignissim egestas ipsum, eget tempor ipsum rutrum eu. Donec vehicula eleifend ullamcorper. Mauris justo nulla, varius a mattis a, cursus sit amet risus. Phasellus rutrum interdum blandit. Donec ut justo eros, ut auctor dolor. Suspendisse potenti. Cras ultricies, dui eget mattis bibendum, leo dui luctus purus, sit amet rhoncus libero metus eget purus. Pellentesque scelerisque ornare sapien faucibus tempor. Suspendisse potenti. Proin fermentum bibendum dapibus. Pellentesque facilisis aliquam. Nam egestas tellus non mauris scelerisque feugiat pellentesque lacus dignissim. Quisque id nulla felis. Mauris justo mauris, posuere sed facilisis in, venenatis nec risus. Mauris eu dui sed tellus laoreet tempor a in turpis volutpat. ]]></Bar> <Jim>Oh! Hello there.</Jim> </Foo> 

Answer 1

Instead of serializing xml and then calling encode on a string, you can specify the encoding that will be used in the parameters for to_xml .; instead

 xml1 = doc1.to_xml.encode('utf-8') 

using:

 xml1 = doc1.to_xml(:encoding => 'utf-8') 

This seems to fix the issues.

---

As for what is going on, I can only offer some observations.

Firstly, the encoding of the string created by to_xml without specifying the encoding is UTF-16 , which in Ruby is a "dummy encoding" (no matter what it means):

 xml1 = doc1.to_xml p xml1.encoding #=> #<Encoding:UTF-16 (dummy)> 

The docs talk about dummy encodings :

A dummy encoding is an encoding for which character processing is not performed properly. It is used for stateful encodings.

Another thing I noticed is that the values ​​in the bent part of the output actually correspond to the code points that should appear.

 Hello there.\uFFFE\u3C00\u0000\u2F00\u0000\u4A00\u0000\u6900... 

3C is < , 2F is / , 4A is J , 69 is i , etc., producing (if you ignore zeros and the extra specification)

 Hello there.</Ji... 

If you write out the XML created by Nokogiri before encoding in UTF-8) and click on the hex editor on it, the beginning looks like this:

 0000000 ff fe 3c 00 3f 00 78 00 6d 00 6c 00 20 00 76 00 

Starts with FF FE , i.e. bit endian bom.

At the start of the search, it looks like this:

 0001f20 3c 00 4a 00 69 00 6d 00 3e 00 4f 00 68 00 21 00 0001f30 20 00 48 00 65 00 6c 00 6c 00 6f 00 20 00 74 00 0001f40 68 00 65 00 72 00 65 00 2e 00 fe ff 00 3c 00 00 0001f50 00 2f 00 00 00 4a 00 00 00 69 00 00 00 6d 00 00 0001f60 00 3e 00 00 00 0a 00 00 00 3c 00 00 00 2f 00 00 

fe ff is where the released result begins (on the midline). fe ff also a large-end specification, and the rest of the characters seem to be BE (you can see how zeros don't line up before and after fe ff . There are additional pairs of null bytes between the characters.