When to use Haystack / ElasticSearch vs Django ORM

Question

So, I implemented Haystack with ElasticSearch a week ago in our BETA application. One thing that I can notice is that getting some data (large volumes) back to our users (e.g. listing all users in an application) is much faster, going through Haystack, then Django ORM. Now I will release the REST service (with TastyPie) to service possible tablets over the next weeks, since I want to have access to information from iPads, Nexus tablets, etc.

One thing that I was wondering about when should I ask ORM vs Haystack / ElasticSearch? For example, if a user on a tablet requests a specific set of users, should we allow TastyPie to request ORM or go to ElasticSearch?

If we look at this answer from Django: Haystack or ORM , we can all agree that the database is created to retrieve and write data. However, can we say that a search can be faster with Haystack / ElasticSearch after a search engine upgrade?

I'm a little confused when we should not request Haystack if it is much faster ?!

Answer 1

To make everything clear, I think you are talking about querying Elasticsearch through Haystack without subsequently creating any objects for your search results with data from your database.

Some points to consider, in addition to the points mentioned in others :

• A search engine such as Elasticsearch is very optimized when working with full-text searches (when doing something with SQL, it is very dependent on the database / engine used)

• Queries that involve a large number of relationships / joins are most convenient to use with ORM, but on the other hand, you can, for example, save data from external links in denormalized mode when using ES, which can give you a performance boost. Of course, you can denormalize your database tables, but this is often considered bad practice as long as you know what you are doing, for example, when solving a performance bottleneck.

• ES is somehow pretty easy to scale, and scaling an SQL database can be more complicated.

• Most likely, this is a solution that largely depends on your use case, the amount of processed data and the requests that you are going to run. So the best thing, of course, is - as always - to do some tests and compare these two solutions. But do not make any premature optimizations, as one big advantage of ORM is that it is simple to save: you do not need to take care of the integrity of your data and maintain an additional system.