PGError: ERROR: invalid byte sequence for encoding "UTF8

Question

I get the following PGError when swallowing Rails emails from Cloudmailin:

PGError: ERROR: invalid byte sequence for encoding "UTF8": 0xbb HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". : INSERT INTO "comments" ("content") VALUES ('Reply with blah blah                                                       ..... 

So, it seems pretty clear that some invalid UTF8 characters get in the email? So I tried to clear it, but something else sneaked in. Here is what I still have:

 message_all_clean = params[:message] Iconv.conv('UTF-8//IGNORE', 'UTF-8', message_all_clean) message_plain_clean = params[:plain] Iconv.conv('UTF-8//IGNORE', 'UTF-8', message_plain_clean) @incoming_mail = IncomingMail.create(:message_all => Base64.encode64(message_all_clean), :message_plain => Base64.encode64(message_plain_clean)) 

Any ideas, thoughts or suggestions? Thanks

Answer 1

When we encounter this problem in Heroku, we convert to US-ASCII to properly clear incoming data (i.e. inserted from Word):

 Iconv.conv("UTF-8//IGNORE", "US-ASCII", content) 

With this, we had no more problems with character encoding.

Also, check to see if there are any other fields that need the same conversion, as this can affect everything that sends a block of text to the database.