GAE News Channel Implementation. Should I use Advanced Search?

Question

I have a problem I'm struggling with right now. I am trying to implement a news feed feature in my application using the endpoints of the GAE and java endpoints. The general concept consists of followers and followers, where his followers can observe the actions of the follower. A new follower must also see his followers in past actions, not only from the moment he begins.

I made several attempts with the following components. Each attempt worked fine, but something was missing:

• In each user action, I added a log object to the data store with the user id enabled. When the user showed his news feed, I simply requested for all these objects their user IDs according to the list of users. Everything was fine until I realized that the IN request could not be programmed. So this option was not.
• In this attempt, which is also the current state of the application, im uses the search API. With each user action, im not saving the essence of the log in the data warehouse, but the document in the search index. Complex queries can be cursed here, and the world is smiling again. But ... I'm not too sure that billing is wise, this is a smart decision. It seems that the costs of finding / adding / deleting documents along with documented daily restrictions make all this too fragmentary.
• The next attempt is a prospective search API. From what I read in the documents , it seems like the right component to choose for this purpose. Unfortunately, the documentation is really poor and provides very few examples. Also, payment information is unclear.

So, I ask for stackoverflow community advice. Could you advise me about this? and if prospective search is the right choice, can you provide some clear example of Java code that uses cloud endpoints?

EDIT: Just emphasize the basic design requirement here. The news function should be able to retrieve the sorted actions of subsequent users using the cursor (to avoid querying the entire batch).

Answer 1

Use the pull-aggregate-per-follower model: periodically (or upon request) request all subsequent actions once, and then cache them inside the selected successor entity. Remember the time of the last request, so the next time you simply request from this point (provided that the action cannot be added / changed in the past).

This will give you the following features (and limitations):

• If the request is on request, you do not need to request users who are inactive.
• Since the query is “only for new” (looking only for new actions), it would not cost anything if it returned zero results.
• You will only request one action from each follower per follower. After that, all the last actions will be cached inside one object and loaded into memory with one get . This should be a significant saving and time saving.
• You can sort / filter actions in memory in any way.

Limitations:

• Entities have a limit of 1 MB, so there is a maximum number of actions that can be cached in one object. Thus, you need to either limit the caching of recent actions for each user, or extend the caching of actions to several objects.
• You will need to use the IN follow followees query (maximum 30), as well as use parallel threads to achieve decent performance. This can easily hit 3-5 seconds when requesting more than 1000-2000 followers. In addition, you can easily push an RPC restriction (for example, simultaneous API calls) on a single instance while serving multiple users.

Answer 2

I hope I correctly understood the question - do you want to embed the news feed in your application and allow users to follow each other. New members should be able to see user actions. I'm sure there are several other ways to solve this problem, but I will try to help you by providing a solution that uses JAVA JDO to access the data store.

First, I developed entity relationships in JDO as follows:

 1 User to many actions. 1 User to many followers (User). 1 User to many following (User). 

Here are the simple JDO classes:

User Class:

 @PersistenceCapable(identityType=IdentityType.APPLICATION) public class User { @PrimaryKey @Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY) private Key key; @Persistent private String userId; // Google unique user ID, could also store user email. @Persistent private Set<Key> actions; @Persistent private Set<Key> followers; @Persistent private List<Key> following; public User(Key key, String userId) { this.key = key; this.userId = userId; this.actions = new HashSet<Key>(); this.followers = new HashSet<Key>(); this.following = new HashSet<Key>(); } public Key getKey() { return this.key; } public void addAction(Key actionKey) { this.actions.add(actionKey); } public void addActions(Set<Key> actionKeys) { this.actions.addAll(actionKeys); } public Set<Key> getActions() { return this.actions; } public void addFollower(Key followerKey) { this.followers.add(followerKey); } public void addFollowers(Set<Key> followerKeys) { this.followers.addAll(followerKeys); } public Set<Key> getFollowers() { return this.followers; } public void addFollowing(Key followingKey) { this.following.add(followingKey); } public void addAllFollowing(Set<Key> followingKeys) { this.following.addAll(followingKeys); } public Set<Key> getFollowing() { return this.following; } } 

Action class:

 @PersistenceCapable(identityType=IdentityType.APPLICATION) public class Action { @PrimaryKey @Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY) private Key key; @Persistent Date date; @Persistent private String title; public Action(Key key, String title) { this.key = key; this.title = title; this.date = new Date(); // date of creation (now). } public Key getKey() { return this.key; } public void setTitle(String title) { this.title = title; } public String getTitle() { return this.title; } } 

The Action class uses the Date property; you can refer to the documentation for the corresponding data types in the data warehouse. When an action is created, the Date object is allocated and initialized so that it represents the time at which it was assigned, measured to the nearest millisecond .

In my example above, I linked the objects with my keys, instead you could bind them according to your classes as follows:

 List<Action> actions; 

The relationship in my example is one of the many-to-one relationship that is irrelevant, perhaps it should be one-to-many. Read more here so you can look and possibly decide which one is best for your solution.

Once the relationships are defined, you can create your endpoint classes around the JDO model classes . This will create the basic api methods. You might want to change the methods of the endpoint class to suit your needs, for example, change the way you create an action. A basic example would be to create a key from an action name as follows (ActionEnpoint.java):

 ... @ApiMethod(name = "insertAction") public Action insertAction( @Named("title") String title ) { PersistenceManager pm = getPersistenceManager(); Key key = KeyFactory.createKey(Action.class.getSimpleName(), title); Action action = null; try { action = new Action(key, title); pm.makePersistent(action); } finally { pm.close(); } return action; } ... 

If you want, you can add a method to your UserEndpoint class to query the data warehouse and return all the actions that belong to this user and per day using the data warehouse query objects .

You need to add a method to your UserEndpoint class that allows you to add an action to this user, here is a simple example:

 ... @ApiMethod(name = "addActionToUser") public Achiever addActionToUser( @Named("userId") String userId, @Named("actionTitle") String actionTitle) { PersistenceManager pm = getPersistenceManager(); Key userKey = KeyFactory.createKey(User.class.getSimpleName(), userId); Key actionKey = KeyFactory.createKey(Action.class.getSimpleName(), actionTitle); User user = null; try { user = (User) pm.getObjectById(User.class, userKey); user.addAction(actionKey); pm.makePersistent(user); } catch (Exception e) { } return user; } ... 

Once all of the above is completed, you can easily get a list of actions for each user by calling the getUser method in your UserEndpoint class, which returns a User object. Then you can call [ReturnedUserObject] .getActions (). The new follower can now view all the followees actions by simply calling the api method to get the followees object and get his / her actions. Then you can simply sort the actions by date or as you imagine it.

I hope I understood your question correctly, I was not sure about the first component that you mentioned, but it seemed that you messed up your relationship. Hope this solution will point you in the right direction, at least :).

If you need further help or clarification, or my answer was completely incompatible with what you were looking for, please let me know.

Regards, Miki