How to prevent django from loading objects in memory when using `delete ()`?

Question

I'm having memory problems because it seems like Django is loading objects into memory when using delete() . Is there a way to prevent using Django?

In the Django docs:

Django must retrieve objects in memory to send signals and process cascades. However, if there are no cascades and no signals, then Django can take a quick path and delete objects without loading them into memory. For large deletions, this can lead to a significant reduction in memory usage. The number of completed queries can also be reduced.

https://docs.djangoproject.com/en/1.8/ref/models/querysets/#delete

I do not use signals. I have foreign keys on a model that I am trying to delete, but I do not understand why Django will need to load objects into memory. This seems to be happening because my memory grows as the request runs.

Answer 1

You can use this function to iterate over a huge number of objects without using too much memory:

 import gc def queryset_iterator(qs, batchsize = 500, gc_collect = True): iterator = qs.values_list('pk', flat=True).order_by('pk').distinct().iterator() eof = False while not eof: primary_key_buffer = [] try: while len(primary_key_buffer) < batchsize: primary_key_buffer.append(iterator.next()) except StopIteration: eof = True for obj in qs.filter(pk__in=primary_key_buffer).order_by('pk').iterator(): yield obj if gc_collect: gc.collect() 

Then you can use the function to iterate over deleted objects:

 for obj in queryset_iterator(HugeQueryset.objects.all()): obj.delete() 

For more information, you can check out this blog post .

Answer 2

You can import the django database connection and use it with sql to delete. I had the same problem as you, and it helps me a lot. Here are some snippets (I use mysql, by the way, but you can run any SQL query):

 from django.db import connection sql_query = "DELETE FROM usage WHERE date < '%s' ORDER BY date" % date cursor = connection.cursor() try: cursor.execute(sql_query) finally: c.close() 

This should only perform the delete operation in this table, without affecting any of your model relationships.