Now that the appengine storage queries are no longer limited to 1000, there are performance issues if you have to use an offset, like 250000

Question

Say I have a million entries in a model in the appengine app store. I will pay the execution penalty with the following request.

records = MyModel.all().filter("words =", "foo").fetch(offset=250000, limit=20) 

Appengine docs say

 The query has performance characteristics that correspond linearly with the offset amount plus the limit. 

Or do I need to create an index and do something like

 records = MyModel.all().filter("words =", "foo").filter("pub_date >", last_date).fetch(20) 

I am trying to check if I can query StringListProperty without adding any indexes to the model.

Answer 1

Using an index will almost certainly give you better performance, especially as the number of objects increases.

If you specify an offset, the datastore scans offset objects before your application begins to receive results. Two points above the one you quote explain a little more:

The first offset results were not missed by the data warehouse itself.