Saving HTML to Mongodb encoding issue

Question

I'm writing a web crawler and need to save the html from the webpage I crawled into my MongoDB database. This is what I'm trying to do(I'm using pymongo):

        c=urllib2.urlopen(myUrl)
        html=c.read()
        db.urls.insert(
                {
                    "url":myUrl,
                    "HTML":html
                }
        )

When I run my script, I get the following error:

InvalidStringData: strings in documents must be valid UTF-8

I tried looking up my problem and figured out that I need to process the HTML somehow before saving it, so it's UTF-8 compatible, but I couldn't find how.

I don't think my question is a duplicate of python encoding utf-8 since I do not see how that question is related to HTML. If I'm wrong, or my problem has nothing to do with HTML, please direct me.

possible duplicate of python encoding utf-8

user3561036
– user3561036

2015-06-15 14:20:41 +00:00
Commented Jun 15, 2015 at 14:20 — user3561036
– user3561036, Commented Jun 15, 2015 at 14:20

silviud · Accepted Answer · 2015-06-15 14:37:52Z

0

To transform from string to utf

html.decode('utf8')

This encodes to utf8 your string content.

answered Jun 15, 2015 at 14:37

silviud

1,0251 gold badge12 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user3561036 Over a year ago

Thanks for copying from the duplicate and by the way it's "encode" and not "decode" that is required here.

silviud Over a year ago

it says is not a duplicate ... anyways if you have a string s = 'asdasdc' ; type(s) str; s.encode('utf8') 'asdasdc' ; s.decode('utf8') u'asdasdc' so the last one is unicode

user2980055 Over a year ago

@user3561036 this is not a duplicate dude, neither encode or decode work for me. With both of them I get UnicodeDecodeError

silviud Over a year ago

what python version do you use ? i tested on 2.6 dude.

silviud Over a year ago

btw - you can try to convert the data to binary ... will insert into the db but not sure will allow text searches - use api.mongodb.org/python/current/api/bson/binary.html

Collectives™ on Stack Overflow

Saving HTML to Mongodb encoding issue

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related