What is the practical limit of NDB count () for App Engine?

Question

We are trying to query the App Engine datastore for some common statistics using NDB. They should not be 100% accurate (i.e. I am not interested in the final sequence); they simply should generally reflect the number of objects.

With NDB, we simply produce something like:

query = MyModel.query(MyModel.source==source, MyModel.created<=some_time).order(-MyModel.created) count = query.count(keys_only=True) 

This is a timeout after the 60s. We regularly use entity groups and transactions, but I hope that this does not affect these count requests. We currently have about 4.2M MyModel objects, although a source filter would limit this to 210,000.

Is there an alternative way to count numbers of this value without a heap of user-defined memcache-y logic? Remember that numbers do not have to be accurate, just "generally correct."

Answer 1

I believe that the limit, previously 1000, is now removed. Thus, the practical limit is how much you can count before the timeout.

Some similar questions were asked and usually Sharded Counters .

But I think that you will probably be better off working with your counter either as a task (time reaches 10 minutes) or on the backend (without a timeout).

As Guido notes in the comments, requests cannot take more than 60 seconds.

EDIT: The 1000 constraint was removed some time ago, 1.3.6: Release Notes:

Datastore no longer uses a limit of 1000 objects for counting and offsetting. Requests that use them will now be safely executed until they return, or your application reaches the request timeout limit.