{"id":935,"hash":"183f0a700566271a96b616a08397f92d65a12047450ee2b580b89540e5e67ed1","pattern":"InterfaceError: connection already closed (using django + celery + Scrapy)","full_message":"I am getting this when using a Scrapy parsing function (that can take till 10 minutes sometimes) inside a Celery task.\n\nI use:\n- Django==1.6.5\n- django-celery==3.1.16\n- celery==3.1.16\n- psycopg2==2.5.5 (I used also psycopg2==2.5.4)\n\n[2015-07-19 11:27:49,488: CRITICAL/MainProcess] Task myapp.parse_items[63fc40eb-c0d6-46f4-a64e-acce8301d29a] INTERNAL ERROR: InterfaceError('connection already closed',)\nTraceback (most recent call last):\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/celery/app/trace.py\", line 284, in trace_task\n    uuid, retval, SUCCESS, request=task_request,\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/celery/backends/base.py\", line 248, in store_result\n    request=request, **kwargs)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/djcelery/backends/database.py\", line 29, in _store_result\n    traceback=traceback, children=self.current_task_children(request),\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/djcelery/managers.py\", line 42, in _inner\n    return fun(*args, **kwargs)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/djcelery/managers.py\", line 181, in store_result\n    'meta': {'children': children}})\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/djcelery/managers.py\", line 87, in update_or_create\n    return get_queryset(self).update_or_create(**kwargs)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/djcelery/managers.py\", line 70, in update_or_create\n    obj, created = self.get_or_create(**kwargs)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/query.py\", line 376, in get_or_create\n    return self.get(**lookup), False\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/query.py\", line 304, in get\n    num = len(clone)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/query.py\", line 77, in __len__\n    self._fetch_all()\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/query.py\", line 857, in _fetch_all\n    self._result_cache = list(self.iterator())\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/query.py\", line 220, in iterator\n    for row in compiler.results_iter():\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py\", line 713, in results_iter\n    for rows in self.execute_sql(MULTI):\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py\", line 785, in execute_sql\n    cursor = self.connection.cursor()\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/backends/__init__.py\", line 160, in cursor\n    cursor = self.make_debug_cursor(self._cursor())\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/backends/__init__.py\", line 134, in _cursor\n    return self.create_cursor()\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/utils.py\", line 99, in __exit__\n    six.reraise(dj_exc_type, dj_exc_value, traceback)\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/backends/__init__.py\", line 134, in _cursor\n    return self.create_cursor()\n  File \"/home/mo/Work/python/pb-env/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py\", line 137, in create_cursor\n    cursor = self.connection.cursor()\nInterfaceError: connection already closed","ecosystem":"pypi","package_name":"django","package_version":null,"solution":"Unfortunately this is a problem with django + psycopg2 + celery combo.\n  It's an old and unsolved problem.\n\n  \n  Take a look on this thread to understand:\n  https://github.com/celery/django-celery/issues/121\n\n  \n  Basically, when celery starts a worker, it forks a database connection\n  from django.db framework. If this connection drops for some reason, it\n  doesn't create a new one. Celery has nothing to do with this problem\n  once there is no way to detect when the database connection is dropped\n  using django.db libraries. Django doesn't notifies when it happens,\n  because it just start a connection and it receives a wsgi call (no\n  connection pool). I had the same problem on a huge production\n  environment with a lot of machine workers, and sometimes, these\n  machines lost connectivity with postgres server.\n\n  \n  I solved it putting each celery master process under a linux\n  supervisord handler and a watcher and implemented a decorator that\n  handles the psycopg2.InterfaceError, and when it happens this function\n  dispatches a syscall to force supervisor restart gracefully with\n  SIGINT the celery process.\n\nEdit:\n\nFound a better solution. I implemented a celery task baseclass like this:\n\nfrom django.db import connection\nimport celery\n\nclass FaultTolerantTask(celery.Task):\n    \"\"\" Implements after return hook to close the invalid connection.\n    This way, django is forced to serve a new connection for the next\n    task.\n    \"\"\"\n    abstract = True\n\n    def after_return(self, *args, **kwargs):\n        connection.close()\n\n@celery.task(base=FaultTolerantTask)\ndef my_task():\n    # my database dependent code here\n\nI believe it will fix your problem too.","confidence":0.95,"source":"stackoverflow","source_url":"https://stackoverflow.com/questions/31504591/interfaceerror-connection-already-closed-using-django-celery-scrapy","votes":35,"created_at":"2026-04-19T04:52:02.703320+00:00","updated_at":"2026-04-19T04:52:02.703320+00:00"}