Node.JS scrape encoding?

Question

I'm fetching this page with with this request library in Node.JS, and parsing the body using cheerio.

Calling $.html() on the parsed response body reveals that the title attribute for the page is:

<title>Le Relais de l'Entrec?te</title>

... when it should be:

<title>Le Relais de l'Entrecôte</title>

I've tried setting the options for the request library to include encoding: 'utf8', but that didn't seem to change anything.

How do I preserve these characters?

cheerio might also just exhibit this bug, which incorrectly outputs certain characters in certain situations — Dan Dascalescu
– Dan Dascalescu, Commented Aug 13, 2014 at 4:09

JJJ · Accepted Answer · 2017-03-17 11:44:33Z

33

You can use iconv (or better iconv-lite) for the conversion itself, but to detect the encoding you should check out the charset and jschardet modules. Here's an example of them both in action:

var charset = require('charset'),
    jschardet = require('jschardet'),
    Iconv = require('iconv').Iconv;

request.get({url: 'http://www.example.com', encoding: 'binary'}, function(err, res, body) {
    var enc = charset(res.headers, body) || jschardet.detect(body).encoding.toLowerCase();

    if(enc !== 'utf8') {
        var iconv = new Iconv(enc, 'UTF-8//TRANSLIT//IGNORE');
        body = iconv.convert(new Buffer(body, 'binary')).toString('utf8');
    }

    console.log(body);
});

edited Mar 17, 2017 at 11:44

JJJ

33.2k20 gold badges95 silver badges103 bronze badges

answered Sep 10, 2013 at 6:27

Ben Dowling

17.6k8 gold badges93 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

leesei Over a year ago

I think this is a better answer as it takes response header into consideration.

nialna2 Over a year ago

Yes this is definitely a better answer and should be the accepted one

dkasipovic Over a year ago

This should be the correct answer. It cleverly uses all available means (apart from asking the developer of the site) to detect the encoding and it succeeds!

dpmott Over a year ago

Note that jschardet v2.2.1 will return 'UTF-8', which won't match 'utf8' after making it lowercase.

wulong · Accepted Answer · 2012-09-08 00:37:56Z

The page appears to be encoded with iso-8859-1. You'll need to tell request to hand you back an un-encoded buffer by passing encoding: null and use something like node-iconv to convert it.

If you're writing a generalized crawler, you'll have to figure out how to detect the encoding of each page you encounter to decode it correctly, otherwise the following should work for your case:

var request = require('request');                                               
var iconv = require('iconv');                                                   

request.get({                                                                   
  url: 'http://www.relaisentrecote.fr',                                         
  encoding: null,                                                               
}, function(err, res, body) {                                                   
  var ic = new iconv.Iconv('iso-8859-1', 'utf-8');                              
  var buf = ic.convert(body);                                                   
  var utf8String = buf.toString('utf-8');  
  // .. do something with utf8String ..                                                                             
});

Collectives™ on Stack Overflow

Node.JS scrape encoding?

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related