This is copied from my answer in google group objectify-appengine: https://groups.google.com/forum/?fromgroups#!topic/objectify-appengine/LlOyRJZRbnk
There are three main options when you are dealing with “aggregation data”, like what you describe:
1) Calculate it when you need it
You have come to the conclusion that I think it is too expensive.
2) Calculate it at intervals and save this result
Not very satisfactory, as this is due to a delay. In addition, you do not want to comb your entire database every day.
3) Update aggregation when data changes
This approach requires a bit more work each time the data changes, but it is almost certainly what you want to do.
Create a collection of contacts for each user. When a message arrives, make sure that there is a sender contact for this recipient. You might also want to delete the contact when the recipient deletes the last message from the sender.
Be careful not to interfere with the transaction speed limits of a group of individuals (one record per second). I will carry out several options:
1) You can save the contact list in each recipient:
class Person { @Id Long id; Set<Key<Person>> contacts; }
This would be a particular problem if, say, the recipient received mail from 20 new people at once. This is almost certainly a bad idea. On the other hand, it is incredibly fast and efficient to find who your contacts are. A small improvement would be to move this to a separate human-born object, so you don't always load this data:
class Contacts { @Parent Key<Person> owner; @Id long id = 1; // there only ever one of these per person, and it should have a predictable key for fetching Set<Key<Person>> contacts; }
Of course, typing in one object gives you a limit of 50,000 words. It may be slightly smaller than this if you first click on the 1M entity size limit. If your keys are ~ 20 characters, it will be about the same. If this is a problem, you can resolve several Contact objects, and at this point you have something similar to the Entity Index Entity template from Brett Slatkin 2009. Input / output of Google messages: http://www.youtube.com/ watch? v = AgaL6NGpkB8
2) You can save your contact list in another direction
class Person { @Id Long id; @Index Set<Key<Peson>> contactOf; }
It's a little expensive to find out who your contacts are - you only need a request key, not a simple get-by-key. But you are no longer limited by the speed of writing an entity. Most likely, users do not send more than one message per second, and if they send 1000 messages in bulk, you can update contactOf in one transaction.
As above, you probably want to move this index to a separate object:
class Contacts { @Parent Key<Person> person; @Id long id = 1; // there only ever one of these per person, and it should have a predictable key for fetching Set<Key<Person>> of; }
3) You can also save these contacts in a completely separate entity
class Contact { @Parent Key<Person> person; @Id Long id; @Index Key<Person> owner; }
This is really just a less effective way to solve # 2.
It is important to update this structure when each message is sent or received.