Python: Convert from ISO-8859-1 / latin1 to UTF-8

Question

I have this line that has been decrypted from Quoted-printable to ISO-8859-1 using the email module. This gives me strings like "\ xC4pple" that match "Äpple" (Apple in Swedish). However, I cannot convert these lines to UTF-8.

>>> apple = "\xC4pple" >>> apple '\xc4pple' >>> apple.encode("UTF-8") Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128) 

What should I do?

Answer 1

Try to decode it first and then encode:

 apple.decode('iso-8859-1').encode('utf8') 

Answer 2

This is a common problem, so here is a fairly thorough illustration.

For non-unicode strings (i.e., without the u prefix like u'\xc4pple' ), it is necessary to decode from the native encoding ( iso8859-1 / latin1 , unless modified with the mysterious sys.setdefaultencoding function ) to unicode , then encode to a character set, which can display the characters you want, in which case I would recommend UTF-8 .

Firstly, here is a handy utility function that helps highlight Python 2.7 string and unicode patterns:

 >>> def tell_me_about(s): return (type(s), s) 

Simple line

 >>> v = "\xC4pple" # iso-8859-1 aka latin1 encoded string >>> tell_me_about(v) (<type 'str'>, '\xc4pple') >>> v '\xc4pple' # representation in memory >>> print v ?pple # map the iso-8859-1 in-memory to iso-8859-1 chars # note that '\xc4' has no representation in iso-8859-1, # so is printed as "?". 

Decoding a string iso8859-1 - converting a simple string to unicode

 >>> uv = v.decode("iso-8859-1") >>> uv u'\xc4pple' # decoding iso-8859-1 becomes unicode, in memory >>> tell_me_about(uv) (<type 'unicode'>, u'\xc4pple') >>> print v.decode("iso-8859-1") Äpple # convert unicode to the default character set # (utf-8, based on sys.stdout.encoding) >>> v.decode('iso-8859-1') == u'\xc4pple' True # one could have just used a unicode representation # from the start 

A few more illustrations - with "Ä"

 >>> u"Ä" == u"\xc4" True # the native unicode char and escaped versions are the same >>> "Ä" == u"\xc4" False # the native unicode char is '\xc3\x84' in latin1 >>> "Ä".decode('utf8') == u"\xc4" True # one can decode the string to get unicode >>> "Ä" == "\xc4" False # the native character and the escaped string are # of course not equal ('\xc3\x84' != '\xc4'). 

Coding in utf

 >>> u8 = v.decode("iso-8859-1").encode("utf-8") >>> u8 '\xc3\x84pple' # convert iso-8859-1 to unicode to utf-8 >>> tell_me_about(u8) (<type 'str'>, '\xc3\x84pple') >>> u16 = v.decode('iso-8859-1').encode('utf-16') >>> tell_me_about(u16) (<type 'str'>, '\xff\xfe\xc4\x00p\x00p\x00l\x00e\x00') >>> tell_me_about(u8.decode('utf8')) (<type 'unicode'>, u'\xc4pple') >>> tell_me_about(u16.decode('utf16')) (<type 'unicode'>, u'\xc4pple') 

The relationship between unicode and UTF and latin1

 >>> print u8 Äpple # printing utf-8 - because of the encoding we now know # how to print the characters >>> print u8.decode('utf-8') # printing unicode Äpple >>> print u16 # printing 'bytes' of u16    pple >>> print u16.decode('utf16') Äpple # printing unicode >>> v == u8 False # v is a iso8859-1 string; u8 is a utf-8 string >>> v.decode('iso8859-1') == u8 False # v.decode(...) returns unicode >>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16') True # all decode to the same unicode memory representation # (latin1 is iso-8859-1) 

Unicode Exceptions

  >>> u8.encode('iso8859-1') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) >>> u16.encode('iso8859-1') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128) >>> v.encode('iso8859-1') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128) 

You can get around them by going from a specific encoding (latin-1, utf8, utf16) to unicode, for example. u8.decode('utf8').encode('latin1') .

Therefore, perhaps the following principles and generalizations could be used:

• type str is a set of bytes that can have one of several encodings, such as Latin-1, UTF-8 and UTF-16
• The unicode type is a set of bytes that can be converted to any number of encodings, most often UTF-8 and latin-1 (iso8859-1)
• The print command has its own encoding logic , is set to sys.stdout.encoding and defaults to UTF-8
• Before converting to another encoding, it is necessary to decode a str to unicode.

Of course, all this changes in Python 3.x.

Hope this is an insight.

Further reading

• Characters vs. Bytes , Tim Bray.

And the very illustrative teachings of Armin Ronacher:

• Updated Unicode Guide to Python (July 2, 2013)
• Learn more about Unicode in Python 2 and 3 (January 5, 2014)
• UCS vs UTF-8 as an internal string encoding (January 9, 2014)
• Everything you didn't want to know about Unicode in Python 3 (May 12, 2014)

Answer 3

Unicode decoding, encoding of results in UTF8.

apple.decode ('latin1'). Encode ('utf8')

Answer 4

For Python 3:

 bytes(apple,'iso-8859-1').decode('utf-8') 

I used this for text incorrectly encoded as iso-8859-1 (showing words like VeÅ \ x99ejnà ©) instead of utf-8. This code creates the correct version of Veřejné.

Answer 5

concept = concept.encode ('ascii', 'ignore') concept = MySQLdb.escape_string (concept.decode ('latin1'). Encode ('utf8'). Rstrip ())

I do this, I'm not sure if this is a good approach, but it works every time!