Django + Postgres: save JSON string directly into model as JSON type

Question

I have a Django model in which I would like to save schema-less key-value metadata, so I'm using a django.contrib.postgres.fields.JSONField.

The key-value data is a Pandas series, which is not JSON-serializable by default (due to numpy.int64, numpy.float64 types), so I use the handy series.to_json() which returns an already serialized JSON string.

To save it to my model, I run json.loads on it. But I know that json.dumps is called on this when the data is saved to Postgres.

How can I avoid this unnecessary deserialization/re-serialization step?

Code example:

def create_model(pandas_series):
    mdl = Model()
    metadata = pandas_series.to_json()  # gives a JSON string
    mdl.metadata = json.loads(metadata) # string->dict, then dict->string
    mdl.save()

Could you post the model definition (at least the metadata = ... field defintion)? Is this django.contrib.postgres.fields.jsonb.JSONField or a third-party package? — Lucas Wiman
– Lucas Wiman, Commented Dec 9, 2016 at 2:05

e4c5 · Accepted Answer · 2016-12-09 06:10:27Z

11

+50

Where is the bottleneck

I created a dictionary with thousand elements programatically it looks something like this:

d = { ... , "a240": 240, "a243": 243, "a242", ...}

Then I json.loads(json.dumps(d)) a 1000 times in a loop. On average the 1000 iterations took 510 milliseconds.

Then I created the following model:

class JsonModel(models.Model):
    jfield = JSONField()

Inserted the dictionary 1000 times into the database as follows:

JsonModel.objects.create(jfield = d)

This operation took 21 seconds on average. So it appears that the real bottleneck is in saving the data to the database rather than in the serialization deserialization step.

So in conclusion, saving to database is roughly 40 times slower than json dumps/loads (but your mileage will vary depending on size of dictionary, load on the db etc).

Avoiding the extra serialize / desrialize steps

It may not be worth the effort to not dumps/loads based on the numbers above. Nevetheless if you are keen to try it, the thing to do is to create a custom field. It can be something as simple as the one below:

from django.contrib.postgres.fields import JSONField
class MyJsonField(JSONField):
    def get_prep_value(self, value):
        if isinstance(value, str):
            return value
        return super(MyJsonField,self).get_prep_value(value)

Then your model can be

class JsonModel(models.Model):
    metadata = MyJsonField()
    ....

And then at the time of creating the db record, you can pass in the json serialized sting instead of calling json.loads

mdl.metadata = metadata # no string->dict, then dict->string

answered Dec 9, 2016 at 6:10

e4c5

53.9k11 gold badges110 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

glifchits Over a year ago

Thanks for providing the solution using the custom field. I also appreciate you doing some perf profiling, I didn't identify a bottleneck, simply a redundant step.

e4c5 Over a year ago

glad to have been of use. Yes, we all hate to see a redundant step.

Collectives™ on Stack Overflow

Django + Postgres: save JSON string directly into model as JSON type

1 Answer 1

Where is the bottleneck

Avoiding the extra serialize / desrialize steps

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Where is the bottleneck

Avoiding the extra serialize / desrialize steps

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related