I'm getting the following error:
ActiveRecord::StatementInvalid: PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0xf66e6bf6 : INSERT INTO "response_sets" ("city") VALUES ('Jönköping') RETURNING "id"
Database is PostgreSQL 9.0.6 on a Heroku app.
Not sure how to get around that error when there are odd characters.
Your database isn't set to the same encoding scheme as the string you're trying to insert. I would imagine Postgres on Heroku is set up to use UTF-8 by default and your input might be one of the latin variations if I had to guess. You can either set your database to accept the encoding scheme you're supplying, e.g.:
SET CLIENT_ENCODING 'ISO-8859-2'
Or you can transcode your input to UTF-8 (this is probably better)
"my string".encode('UTF-8')
It's likely your database isn't set to the same encoding as the string you're inserting. Postgres is typically UTF-8.
You'll have to set the proper encoding on your string.
That could be as simple as
"string".encode('UTF-8')
Or if the string is improperly tagged you may also have to force_encoding first. ie. it's stored as 'Windows-1252' but not marked as such by Ruby.
"string".force_encoding('Windows-1252').encode('UTF-8')
We had this problem working with Sendgrid + Heroku Rails http://blog.zenlike.me/2013/04/06/sendgrid-parse-incoming-email-encoding-errors-for-rails-apps-using-postgresql/
This seems to work for me:
"Hern\xE1ndez".encode('UTF-8','ISO-8859-1')
The first argument is the encoding that you want the string to be in and the second argument is the encoding that you think the string is in.
There are also options in the documentation on how to deal with invalid or undefined characters.
This is what I ended up using (just to be safe):
"Hern\xE1ndez".encode('UTF-8','ISO-8859-1', :invalid => :replace, :undef => :replace, :replace => "?")
You can also make up a helper method:
def convert_to_utf_8(string)
string.encode('UTF-8','ISO-8859-1', :invalid => :replace, :undef => :replace, :replace => "?")
end
The problem I was having was loading data from the Amazon Merchant Services API.
