Unicode text decoding back

Question

Many text encodings have the property that you can go back through encoded text and still be able to decode it. ASCII, UTF-8, UTF-16, and UTF-32 have this property. This allows you to do convenient things, such as reading the last line of a file without reading all the lines before it or moving back several lines from the current position in the file.

Unfortunately, Python does not seem to be able to decode the file back. You cannot read back or seek by the number of characters in the encoded file. Decoders in the codecs module support incremental decoding forward, but not backward. It seems that there is no "UTF-8-back" codec, I could feed the UTF-8 bytes in reverse order.

I could probably implement my own character synchronization on the codec, read the binary fragments back and pass the correctly aligned fragments to the corresponding decoders from the codecs module, but it seems like a non-expert will skip some subtle details and not notice that the result is incorrect .

Is there an easy way to decode text in Python using existing tools?

---

Several people seem to have missed the fact that reading the entire file to do this defeats the target . Although I clarify things, I could also add that this should work for variable-length encodings . Support for UTF-8 is required .

Answer 1

The lack of a general purpose solution, here is one specific to utf-8:

 def rdecode(it): buffer = [] for ch in it: och = ord(ch) if not (och & 0x80): yield ch.decode('utf-8') elif not (och & 0x40): buffer.append(ch) else: buffer.append(ch) yield ''.join(reversed(buffer)).decode('utf-8') buffer = [] utf8 = 'ho math\xc4\x93t\xc4\x93s hon \xc4\x93gap\xc4\x81 ho I\xc4\x93sous' print utf8.decode('utf8') for i in rdecode(reversed(utf8)): print i, print "" 

Result:

 $ python x.py ho mathētēs hon ēgapā ho Iēsous suos ē I oh ā pag ē nohs ē t ē htamoh