Google App Engine request memory usage

Question

When I run a query on a large set of small objects (15k objects with a few short string and logical properties) without doing anything with these objects, I see that the memory usage of the instance is constantly increasing (an increase of 70 MB). The increase in memory does not look proportional to the amount of data that it ever needs to store in memory just for the request.

I use the following loop:

cursor = None while True: query = MyModel.all() if cursor: query.with_cursor(cursor) fetched = 0 for result in query.run(batch_size = 500): fetched += 1 # Do something with 'result' here. Actually leaving it empty for # testing to be sure I don't retain anything myself if fetched == 500: cursor = query.cursor() break else: break 

To be sure this is not due to appstats, I call appstats.recording.dont_record() to not record any statistics.

Does anyone know what could happen? Or any pointers on how to debug / project this?

Update 1 : I included gc.set_debug(gc.DEBUG_STATS) in the production code, and I see the garbage collector called regularly, so it tries to collect garbage. When I call gc.collect() at the end of the loop (also the end of the request); it returns 0 and does not help.

Update 2 . I hacked a bit to get guppy working on dev_appserver, and it showed that after explicit gc.collect() at the end of the loop, most of the memory was consumed by "dict of google.appengine.datastore.entity_pb.Property".

Answer 1

I reported this to the application development team, and they seem to confirm that this is actually a problem (presumably with cursor processing).

Answer 2

Each model object has a head over.

You request return objects as Protobuf for starters.

So you get a series of batch protobuffs for a set of results.

Then it is decoded. Each decoded object includes property names, as well as data for each object. You have 15K objects. How big are your property names, for example.

Thus, you have at least two copies of the result set in memory in various forms (possibly more), not including anything you do with instances of the model class.

The code / loop is not possible for garbage collectors, and this may happen later.

Look at tools like apptrace to help profile memory.