Django REST Framework Serialization Extremely Slow

Question

I am in Python 2.7 and Django 1.7.1 with django-restframework I have an API that returns me some specific values ​​taken from a database, it uses a custom serializer as follows:

class InventarioSerializer(serializers.ModelSerializer): item = serializers.RelatedField(source='producto.item') ubicacion = serializers.RelatedField(source='ubicacion.nombre') class Meta: model = Inventario fields = ('epc','item','cantidad','ubicacion') 

My API is called like this:

 class ItemEnInventarioViewSet(InventarioListModelMixin, viewsets.ModelViewSet): serializer_class = InventarioSerializer renderer_classes = (UnicodeJSONRenderer,) 

and my ListModelMixin:

 class InventarioListModelMixin(object): def list(self, request, *args, **kwargs): item = request.QUERY_PARAMS.get('item', None) inventario = Inventario.objects.filter(producto__item = item) if inventario.count() == 0: return HttpResponse(u"El item %s no se encuentra en el inventario" % item,status=400) self.object_list = inventario # Switch between paginated or standard style responses page = self.paginate_queryset(self.object_list) if page is not None: serializer = self.get_pagination_serializer(page) else: serializer = self.get_serializer(self.object_list, many=True) <<--THIS IS THE PROBLEM return Response(serializer.data) 

It works fine, but when I try to create a database of about 1000 or more records, the serializer makes it very slow, arround from 25 to 35 seconds.

The query in the database is very simple, so the database is not a problem at all.

If I serialize a query with this function " data = serializers.serialize('json', myQuerySet) ", it takes no more than 3 seconds, but I don’t get the information on my own, so I use Custom Serializer

Is there a quickest way to get so many values? Maybe with another Serializer? any idea?

** ANSWER Thanks to Kevin ** Change request:

 inventario = Inventario.objects.select_related('producto__item','ubicacion__nombre').filter(producto__item = item) 

... forces the Serializer not to hit the database every row of results to get external values.

Answer 1

The query in the database is very simple, so the database is not a problem at all.

Make sure you do not have an N + 1 question with your queries. They may be simple, but if there are a lot of them, then it will take a lot of time. I wrote a little about troubleshooting performance issues in the Django REST Framework here, and you can find out a lot about it by doing a search.

Is there a quickest way to get so many values? Maybe with another Serializer? any idea?

If your data does not change so often, or you can handle any possible caching problems, you can greatly benefit from adding caching to your API. drf-extensions provides quite a few useful cache mixins that can help you if your problem is not related to your requests.

when I try to GET a DB arround form of 1000 or more records

I understand that your code has a built-in pagination, but I want to emphasize the value of using pagination when working with large amounts of data. Query performance tends to be very linear, and the more data you need to get, the more time it will take to get it.

Answer 2

For me, N + 1 queries for the database were not the answer. It took profiling time to determine, but after that the answer turned out, unfortunately, to be several DecimalField fields in my serializer.

My usage example was simple: 3000-4000 instances that needed to be serialized. All select_related optimizers were executed, however I still watched 2-3 seconds of serialization time, and did not expect .5-1.5 seconds. After several hours of trial and error (commenting / uncommenting fields), I saw a huge (50%) drop at runtime when I had all my DecimalField.

The solution for me was to change my DecimalField to FloatField. Of course, you do it at the cost of losing accuracy, but for my purposes it was good.