Possible duplicate:
Warning boot loader loading boot file for application "" No descending index on the key performing sequential download "
My post is very similar to: Warning for loading the bootloader of the mass application loader "There is no descending index on __key__ performing sequential loading"
I really want to do the same.
Basically, I use the following to load all instances of one of my views:
appcfg.py download_data --config_file=bulkloader.yaml --kind=ModelName --filename=ModelName.csv --application=MyAppid --url=http:
If the type has more instances than the batch size, I get this warning:
No descending index on __key__, performing serial download
This results in loading only about 6,500 objects in 471.4 seconds (according to the bulkloader tool after its completion). Which is really slow because I have about 4 other species, each of which is even larger (about 15,000 entities)!
Also, according to my Mac activity monitor, I download only about 24 Kbps, as shown in the bandwidth in the bulkloader output:
[INFO ] Logging to bulkloader-log-20110514.011333 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 10
My questions:
1) How to get rid of this warning “There is no downward index on __key__ performing sequential loading” to get parallel loading speed?
I think the answer to my question is to add a descending index. Something like:
<datastore-index kind="Game" ancestor="false" source="manual"> <property name="id" direction="desc"/> </datastore-index>
I tried adding this to the datastore-indexes.xml file.
It rolled out successfully, but I looked at the Datastore indexes on the admin portal on Google, but I did not see it working or building. In any case, for the sake of this, I repeated the command below, and it was still slow ....
I also tried adding the same xml, but with source = "auto", to the datastore-indexes-auto.xml file. However, when I tried to expand my eclipse, I complained about the following error:
java.io.IOException: Error posting to URL: https://appengine.google.com/api/datastore/index/add?app_id=<My_APP_ID>&version=1& 400 Bad Request Creating a composite index failed: This index: entity_type: "Game" ancestor: false Property { name: "id" direction: 2 } is not necessary, since single-property indices are built in. Please remove it from your index file and upgrade to the latest version of the SDK, if you haven't already.
2) Removing this warning requires me to update my bulkloader.yaml generator? I have included the type of game below:
python_preamble: - import: base64 - import: re - import: google.appengine.ext.bulkload.transform - import: google.appengine.ext.bulkload.bulkloader_wizard - import: google.appengine.ext.db - import: google.appengine.api.datastore - import: google.appengine.api.users transformers: - kind: Game connector: csv connector_options:
Useful find
How I typed this question. I found this App Engine bootloader performance
It is mainly explained that increasing bandwidth_limit by something reasonable and increasing rps_limit can really help speed up the process.
So I tried:
appcfg.py download_data --config_file=bulkloader.yaml --kind=ModelName --filename=ModelName.csv --application=MyAppId --url=http:
Which reduced the boot time to 109.8 seconds. This is a massive reduction!
However, my goal is still focused on getting rid of "There is no downstream index on __ key__ performing sequential loading" for parallel loading.
Additional information may be relevant.
I am using objectify3.0.jar to manage my GAE datastore. So my type of game is as follows:
public class Game { @Id private Long id;