Django - Effectively mass create legacy models

Question

Python 2.7.9 Django 1.7 MySQL 5.6

I would like to populate a whole bunch of instances of objects belonging to several classes, put them in a single create() request, open a database connection, execute the request, and then close. My main motivation is performance, but compact code is also a plus.

The functionality of bulk_create() seems to be exactly what I want, but I violate at least one of the warnings listed here , i.e.

He does not work with many relationships.

and

It does not work with child models in a multi-table inheritance scenario.

These restrictions are also described in the source code :

 # So this case is fun. When you bulk insert you don't get the primary # keys back (if it an autoincrement), so you can't insert into the # child tables which references this. There are two workarounds, 1) # this could be implemented if you didn't have an autoincrement pk, # and 2) you could do it by doing O(n) normal inserts into the parent # tables to get the primary keys back, and then doing a single bulk # insert into the childmost table. Some databases might allow doing # this by using RETURNING clause for the insert query. We're punting # on these for now because they are relatively rare cases. 

But the error was returned when I try to make it general

ValueError: unable to create an inherited model array

My models do not seem to contain many-to-many fields or foreign keys. I don’t quite understand what inheritance scenarios on several tables they mean, so I'm not sure if this is my problem. I was hoping I could slip into my structure, which looks like this, but then I got a general error, so no cubes:

 child class with OneToOneField---\ \ child class with OneToOneField----->---concrete parent class / child class with OneToOneField---/ 

As for the workarounds suggested in the source, # 1 is not an option for me, and # 2 does not look attractive, because I assume that this will entail a donation of performance gains for which I am going.

Are there other workarounds that could mimic bulk_create() when handling inheritance, like this, and not give up performance gains? Do I need to return to raw SQL? I would not mind creating a separate collection and doing a separate INSERT / create() for each type of child objects.

Answer 1

The workaround I stopped at was wrapping all my compiled create() in with transaction.atomic(): This significantly reduced the time spent by not opening any database connections or making any queries until all of Python returned.

The disadvantage may be that when any errors occur, all changes are rolled back, and the database remains untouched. This could be fixed by breaking create() into batches and opening and closing a transaction around each. (In my case, this was not desirable behavior because I wanted all the data or none of this.)