Random database errors with Django 1.7, uwsgi and PostgreSQL

Question

After upgrading my Django 1.6 app to Django 1.7 I started to get random errors while fetching data from PostgreSQL:

DatabaseError: server sent data ("D" message) without prior row description ("T" message)
lost synchronization with server: got message type "�", length -1244613424

DatabaseError: lost synchronization with server: got message type "0", length 842674226

ProgrammingError: no results to fetch

ValueError: invalid literal for int() with base 10: 'feuj3f47jvsdv7tgnj43g63j'

When I quickly open 10 tabs in browser, half of the tabs load normally, half of them get a DB error. When I refresh the tabs where errors occurred, they load normally.

I'm running Django through uwsgi and nginx, the version of psycopg2 is 2.5.4.

Overall it looks like somehow communication with Postgres is completely broken and results of different queries get mixed.

Edit:

After several hours of troubleshooting I found out the following:

Django 1.6 + uwsgi - works
Django 1.7 + gunicorn - works
Django 1.7 + uwsgi - doesn't work, throws database errors. So the problem seems to be with specifically uwsgi and Django 1.7 combination. Which is strange, I have another Django 1.7 project running on the same server with the same uwsgi and it has no problems.

Any ideas?

(I don't really mind switching over to gunicorn, will probably have to go this way, but it's still interesting why is this happening)

Update 2: Closer examination shows completely insane things happening inside Django, like a model's primary key being substituted with current user's session_id (that's the source of "invalid literal for int() with base 10" error) and Django issuing queries to DB "forgetting" to specify WHERE clause. I would probably call that a memory corruption of some kind.

Update 3: We switched from uwsgi to gunicorn and the problems are now gone. Everything works great. I'm probably still looking for a proper solution though.

libpq mismatched to psycopg2? It's something low level, in psycopg2 or in libpq its self, anyway. — Craig Ringer
– Craig Ringer, Commented Oct 9, 2014 at 8:11
My old (Django 1.6 production) and new (Django 1.7 development) installations share the same server, so I can't see how one can have problems with lipbq while other doesn't — Spc_555
– Spc_555, Commented Oct 9, 2014 at 8:15
But you are right, maybe psycopg2 is misconfigured at compilation, I'll need to check it — Spc_555
– Spc_555, Commented Oct 9, 2014 at 8:16
Well, looks like the problem is somehow with Django itself, see the update. — Spc_555
– Spc_555, Commented Oct 9, 2014 at 8:48
just for being paranoid: are you sure you are not running a super-old broken uWSGI version ? (like anything < 1.2.8) — roberto
– roberto, Commented Oct 14, 2014 at 1:54

judu · Accepted Answer · 2015-03-27 20:24:30Z

6

I think lazy-apps=true should do the trick. From uwsgi documentation:

uWSGI tries to (ab)use the Copy On Write semantics of the fork() call whenever possible. By default it will fork after having loaded your applications to share as much of their memory as possible. If this behavior is undesirable for some reason, use the lazy-apps option. This will instruct uWSGI to load the applications after each worker’s fork(). Beware as there is an older options named lazy that is way more invasive and highly discouraged (it is still here only for backward compatibility)

If you do not switch lazy-apps on, the workers will share memory, and most likely corrupt it:

a model's primary key being substituted with current user's session_id.

When it comes to connections and stuff, not setting lazy-apps is dangerous.

The drawback is that every worker will have a full connection pool (if connection pooling is taking place), and you might end up using a lot of connections.

I'm no python expert, but I think using something like gevent to handle the connection pooling in a centralized way. You might then even not need lazy-apps.

answered Mar 27, 2015 at 20:24

judu

1311 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Spc_555 Over a year ago

I don't have the time to test this idea now, but it sounds plausible

Avamander Jan 26 at 16:58

These errors can happen in all such cases where the connection pool is unsafely shared between threads. So Celery, Flask, all can be impacted. In the case of Celery you can just dispose of the engine when a new worker is started. Described in-depth here: yangster.ca/post/not-the-same-pre-fork-worker-model

Collectives™ on Stack Overflow

Random database errors with Django 1.7, uwsgi and PostgreSQL

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related