UnicodeDecodeError with Django Rest Framework on Model's str field

Question

I'm banging my head on the table regarding this error, and I've scoured the internet but either 1) haven't been able to find the answer or 2) am not knowledgable enough to figure out what's going on.

The tldr summary of my problem:

When I store a non-ASCII character (such as è) in a models.TextField and serialize it with serializers.CharField, I do not encounter and problems. However, if I have a str method on the model that returns that same text field, upon serialization I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

First of all...

I'm running Python 2.7.10 on OS X.
I'm using Django==1.11.7 and djangorestframework==3.7.3.
My LANG and LC_ALL environment variables are both set to en_US.UTF-8.
Lastly, I'm using PostgreSQL 9.5.6 and I verified that my database is encoded in UTF8.

The details of my problem:

I have a simple model as shown below. It has a name variable that is a TextField as well as a __str__ method that simply returns self.name. I also have defined a testing variable that also simply returns self.name for demonstration purposes.

from __future__ import unicode_literals

from django.db import models
from django.utils.encoding import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _


@python_2_unicode_compatible
class Supplier(models.Model):
    name = models.TextField(help_text=_('Store Name'))

    class Meta:
        verbose_name = 'Supplier'
        verbose_name_plural = 'Suppliers'

    def testing(self):
        return self.name

    def __str__(self):
        return self.name

Next, my serializer is defined as follows.

class SupplierSerializer(serializers.ModelSerializer):

    class Meta:
        model = Supplier
        fields = ('name', 'testing', '__str__')

Finally, I have the following viewset:

class SupplierViewSet(viewsets.ModelViewSet):
    queryset = Supplier.objects.all()
    serializer_class = SupplierSerializer

The problem:

When I make a GET request to the view, I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)

Observation #1:

This error only seems to occur when I try to serialize __str__. If I remove __str__ from the serializer (and thus serializer only name and testing) I don't get any errors. What's odd is that testing returns the same exact thing as __str__ - self.name.

Observation #2: I verified that inside the __str__ method, self.name is type unicode. Inside the testing method, self.name is also type unicode.

Observation #3: The full stack trace is given below.

Traceback (most recent call last):
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 217, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users//github///lib/python2.7/site-packages/django/core/handlers/base.py", line 215, in _get_response
    response = response.render()
  File "/Users//github///lib/python2.7/site-packages/django/template/response.py", line 107, in render
    self.content = self.rendered_content
  File "/Users//github///lib/python2.7/site-packages/rest_framework/response.py", line 72, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
  File "/Users//github///lib/python2.7/site-packages/rest_framework/renderers.py", line 105, in render
    allow_nan=not self.strict, separators=separators
  File "/Users//github///lib/python2.7/site-packages/rest_framework/utils/json.py", line 28, in dumps
    return json.dumps(*args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 210, in encode
    return ''.join(chunks)

It seems to be an error related to the json encoder?

Does anyone with a deeper understanding of django and unicode have an idea what's going on?

Quite apart from anything else, you should consider upgrading to Python 3. — Daniel Roseman
– Daniel Roseman, Commented Aug 27, 2018 at 20:12
Why do you put fields for both name and __str__ on your serializer? It doesn't make sense. I think we need to see a bit more of the original context here. Methods need to use a SerializerMethodField. — wim
– wim, Commented Aug 28, 2018 at 1:58
@wim Currently my str method just returns self.name for simplicity sake. The str method actually returns other pieces of data - I'm just not showing it to keep things simple. But regarding your second point - that's a good point... I got into the habit of putting model fields in the Meta.fields tuple, but you bring up a good point that str is technically not a field. But is it a method? I feel like it is a "special" method. — jmtoung
– jmtoung, Commented Aug 28, 2018 at 2:28

jmtoung · Accepted Answer · 2018-08-31 15:28:33Z

4

While I do not entirely understand the reason why this fixes the error, setting the UNICODE_JSON setting to False fixes the error.

REST_FRAMEWORK = {
    'UNICODE_JSON': False,
}

answered Aug 31, 2018 at 15:28

jmtoung

1,0221 gold badge15 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

UnicodeDecodeError with Django Rest Framework on Model's str field

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related