PGError: ERROR: invalid byte sequence for encoding "UTF8

Question

I'm getting the following PGError while ingesting Rails emails from Cloudmailin:

PGError: ERROR: invalid byte sequence for encoding "UTF8": 0xbb HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". : INSERT INTO "comments" ("content") VALUES ('Reply with blah blah  ����������������������������������������������������� .....

So it seems pretty clear I have some invalid UTF8 characters getting into the email right? So I tried to clean that up but something is still Sneaking through. Here's what I have so far:

message_all_clean = params[:message]
Iconv.conv('UTF-8//IGNORE', 'UTF-8', message_all_clean)
message_plain_clean = params[:plain]
Iconv.conv('UTF-8//IGNORE', 'UTF-8', message_plain_clean)

@incoming_mail = IncomingMail.create(:message_all => Base64.encode64(message_all_clean), :message_plain => Base64.encode64(message_plain_clean))

Any ideas, thoughts or suggestions? Thanks

Dominic · Accepted Answer · 2011-01-23 18:42:45Z

8

When encountering this issue on Heroku, we converted to US-ASCII to sanitize incoming data appropriately (i.e. pasted from Word):

Iconv.conv("UTF-8//IGNORE", "US-ASCII", content)

With this, we had no more issues with character encoding.

Also, double check that there's no other fields that need the same conversion, as it could affect anything that's passing a block of text to the database.

answered Jan 23, 2011 at 18:42

Dominic

3,30421 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

AnApprentice Over a year ago

Thanks but do I need to do something like CLEAN = Iconv.conv("UTF-8//IGNORE", "US-ASCII", content) or does Iconv convert the content var and I can just send content to the DB?

Dominic Over a year ago

Ah, there I am not sure. In actual practice, I created an Iconv object (ic = Iconv.new("UTF-8//IGNORE", "US-ASCII")), and then used the iconv method: content = ic.iconv(content). Iconv.conv() seems to be a shorthand for that, but I preferred having a reusable object.

Bogdan Gusiev Over a year ago

How to convert it into Ruby 1.9.3 where Iconv is deprecated?

Collectives™ on Stack Overflow

PGError: ERROR: invalid byte sequence for encoding "UTF8

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related