CSV file encoding in Rails with S3 and Heroku

Question

My rails app uploads CSV files to S3, then subsequently pulls them down into a tempfile to send each row's data to a Sidekiq worker. I'm using Carrierwave and fog to handle the uploading.

This all worked beautifully until recently switching to Heroku, and now, when trying to create my tempfile I get the following error:

Error type Encoding::UndefinedConversionError
Error message "\xA2" from ASCII-8BIT to UTF-8

I've tried setting the encoding when creating the tempfile as well as working with the CSV file and continue to get the same error. I cannot reproduce this error on my local machine, which has made this entire process that much more fun :)

Currently, my Sidekiq worker calls the following method:

def upload_csv(filename, file_path)
  file = Tempfile.new(filename, Rails.root.join('tmp'), encoding: "ISO8859-1:utf-8").tap do |f|
   open(file_path).rewind
   f.write(open(file_path).read)
   f.close
  end

  CSV.foreach(file, headers: true, encoding: "ISO8859-1:utf-8")do |row|
   #do stuff to rows
  end
end

I understand the very basics of encoding, but I'm super stuck on this. Any insight would be appreciated.

Thanks!

I should add that after setting the encoding on the tempfile, its charset still comes back as us-ascii not sure if that makes a difference.... — chicjean
– chicjean, Commented Mar 1, 2016 at 23:03

chicjean · Accepted Answer · 2016-03-04 18:18:33Z

1

Not sure if this will help anyone else, but I found a solution that works for me:

def upload_csv(filename, file_path)
  file = Tempfile.new(filename, Rails.root.join('tmp')).tap do |f|
   open(file_path).rewind
   f.write(open(file_path).read.force_encoding('utf-8'))
   f.close
  end

  CSV.foreach(file, headers: true)do |row|
   #do stuff to rows
  end
end

Even though I could confirm that the file was UTF-8 encoded before it was uploaded, open(@file_path).read.encoding returning an ASCII-8BIT encoding. It was getting confused on how to write the file and convert it from ASCII-8BIT to UTF-8.

answered Mar 4, 2016 at 18:18

chicjean

776 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

CSV file encoding in Rails with S3 and Heroku

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related