Geek Answers Handbook

GoogleAppEngine JDO: How to Count and Group Using BigTable

I need to collect some statistics on my objects in a data warehouse.

As an example, I need to know how many objects I have, how many objects with some properties set for specific values, etc. In a regular relational DBMS, I can use

SELECT COUNT(*) ... WHERE property=<some value>

or

  SELECT MAX(*), ... GROUP BY property

etc .. But here I do not see any of these structures.

In addition, I cannot load all the objects in memory (for example, using pm.getExtent (MyCall.class, false)), because I have too many objects (more than 100k in details).

Do you know any trick to achieve my goal?

+4
source share
2 answers

Actually, it depends on your specific requirements.

Btw, there is a general way to prepare statistics data in the background.

For example, you can run several tasks using the Queue service, which will use a query like select x where x.property == some value + cursor + an sum variable . If you are in the first step, the cursor will be empty, and the amount will be zero. Then you will iterate over the query result by 1000 elements (query limit) or 9 minutes (task restriction), increasing the sum at each step, and then, if this does not end, call this task with a new cursor and sums, I mean that you add the request to the next step in the queue. The cursor is easily serialized to a string.

When you have the final step, you need to save the value of the result somewhere in the statistics results table.

Take a look at:

And also this statistics / aggregation material really depends on your actual task / requirements / project, there are several ways to achieve this, optimal for different tasks. There is no standard way like in SQL

+2
source

Support for aggregate functions is limited to GAE. This is, first of all, an artifact of the BigTable custom scheme. An alternative is to maintain aggregate functions as separate fields for quick access to them.

To make an account, you can do something like this -

 Query q = em.createQuery("SELECT count(p) FROM your.package.Class p"); Integer i = (Integer) q.getSingleResult();

but this will probably return you a total of 1000 lines, as GAE limits the number of lines received to 1000.

Some helpful tips on getting around these issues are

http://marceloverdijk.blogspot.com/2009/06/google-app-engine-datastore-doubts.html

Is there a way to do aggregate functions in the Google App Engine?

+1
source

More articles:


All Articles