json serialization outputting accented character incorrectly, python / django

Question

I have a Django app, using tasypie to serialize some data.

There is a name

"Glòria"

(with an accented 'o') in the database, but this is not being serialized correctly. In the json produced by tasypie, it comes out as

"GlÃ²ria"

The serializer class looks like this:

import json as simplejson


class PrettyJSONSerializer(Serializer):
    json_indent = 2
    def to_json(self, data, options=None):
        options = options or {}
        data = self.to_simple(data, options)
        return simplejson.dumps(data, cls=json.DjangoJSONEncoder,
            sort_keys=True, ensure_ascii=False, indent=self.json_indent)

Changing the attribute on the simplejson.dumps to

ensure_ascii=True

returns the following:

"Gl\u00f2ria"

Is this Python 2 or 3? If it's Python 2, is the name represented by a str or a unicode object? — patrys
– patrys, Commented Aug 11, 2015 at 9:52
Python 2.7, its stored as unicode internally, the debugger shows: u'Gl\xf2ria ' — wobbily_col
– wobbily_col, Commented Aug 11, 2015 at 9:58
The "Gl\u00f2ria" version is actually a valid JSON representation of Glòria. Are you sure the problem with ensure_ascii=False is with the serializer and not the client? — patrys
– patrys, Commented Aug 11, 2015 at 10:05
I don't see a problem with"Gl\u00f2ria" but its not what I want to return. I would like to set ensure_ascii=False, and have it output a ò'' rather than 'Ã²' — wobbily_col
– wobbily_col, Commented Aug 11, 2015 at 10:10
Hmmm. I don't know Django or tastypie, so there might be a proper way to fix this, but FWIW, you can easily convert that Unicode escape to proper Unicode. Eg, s="this is a Gl\u00f2ria test".decode('unicode-escape');print s,repr(s) prints this is a Glòria test u'this is a Gl\xf2ria test'. At least, it'll print that if your console is set to use utf-8 encoding. :) — PM 2Ring
– PM 2Ring, Commented Aug 11, 2015 at 10:20

voneiden · Accepted Answer · 2015-08-11 10:35:16Z

3

I can't comment (yet..) so I'm posting a reply. Python 2 isn't exactly fun with encodings.

GlÃ²ria is the correct utf-8 encoded representation of the data in bytes. Gl\u00f2ria is Python 2 internal representation of unicode strings. json.dumps returns a python unicode string. What you probably want to do is encode the output of json.dumps in utf8.

import json
data = u'Gl\xf2ria'
encoded_data = json.dumps(s, ensure_ascii=False).encode("utf8")
print(encoded_data)

prints Glòria.

Edit: Just to make sure

GlÃ²ria = Gl\xc3\xb2ria. Printed with the print statement both should display correctly as Glòria.

edited Aug 11, 2015 at 10:35

answered Aug 11, 2015 at 10:30

voneiden

4974 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

wobbily_col Over a year ago

You are right, its maybe a web browser based problem, as using curl on the command line displays it properly.

voneiden Over a year ago

Maybe you're familiar with the matter, but in case somebody reading this later isn't: it's necessary to define the content encoding. text/html provides html tags for this, but something like application/json may require adding the encoding to Content-type HTTP headers to display correctly on browsers (Content-type: application/json; charset=utf-8).

Collectives™ on Stack Overflow

json serialization outputting accented character incorrectly, python / django

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related