What response time can be expected from GAE / NDB?

Question

We are currently creating a small and simple central HTTP service that maps "external identifiers" (for example, facebook identifier) ​​to an "internal (uu) id" unique to all of our services to help with analytics.

The first prototype in "our stack" (bulb + postgresql) was completed during the day. But since we want the service to (almost) never interrupt or scale automatically, we decided to use the Google App Engine.

After a week of reading and testing and comparing, this question arises:

What is the “normal” response time in App Engine (with NDB)?

We get a response time that constantly exceeds 500 ms on average and well above 1 s in the 90th client.

I have attached the version of our code below, hoping that someone can point out an obvious flaw. We really like autoscaling and distributed storage, but we cannot imagine that 500 ms is the expected performance in our case. The sql-based prototype responded much faster (sequentially), hosted on a single Heroku single, using the free, non-cache postgresql (even with ORM).

We tried both synchronous and asynchronous code options below and looked at the appstats profile. These are always RPC calls (both memcache and datastore), which take a very long time (50 ms-100 ms), which has deteriorated due to the fact that there are always several calls (for example, mc.get () + ds.get () + ds.set () to write). We also tried to postpone the task to the task queue as much as possible without noticeable success.

import json import uuid from google.appengine.ext import ndb import webapp2 from webapp2_extras.routes import RedirectRoute def _parse_request(request): if request.content_type == 'application/json': try: body_json = json.loads(request.body) provider_name = body_json.get('provider_name', None) provider_user_id = body_json.get('provider_user_id', None) except ValueError: return webapp2.abort(400, detail='invalid json') else: provider_name = request.params.get('provider_name', None) provider_user_id = request.params.get('provider_user_id', None) return provider_name, provider_user_id class Provider(ndb.Model): name = ndb.StringProperty(required=True) class Identity(ndb.Model): user = ndb.KeyProperty(kind='GlobalUser') class GlobalUser(ndb.Model): uuid = ndb.StringProperty(required=True) @property def identities(self): return Identity.query(Identity.user==self.key).fetch() class ResolveHandler(webapp2.RequestHandler): @ndb.toplevel def post(self): provider_name, provider_user_id = _parse_request(self.request) if not provider_name or not provider_user_id: return self.abort(400, detail='missing provider_name and/or provider_user_id') identity = ndb.Key(Provider, provider_name, Identity, provider_user_id).get() if identity: user_uuid = identity.user.id() else: user_uuid = uuid.uuid4().hex GlobalUser( id=user_uuid, uuid=user_uuid ).put_async() Identity( parent=ndb.Key(Provider, provider_name), id=provider_user_id, user=ndb.Key(GlobalUser, user_uuid) ).put_async() return webapp2.Response( status='200 OK', content_type='application/json', body = json.dumps({ 'provider_name' : provider_name, 'provider_user_id' : provider_user_id, 'uuid' : user_uuid }) ) app = webapp2.WSGIApplication([ RedirectRoute('/v1/resolve', ResolveHandler, 'resolve', strict_slash=True) ], debug=False) 

For completeness of use (almost by default) app.yaml

 application: GAE_APP_IDENTIFIER version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: 2.5.2 - name: webob version: 1.2.3 inbound_services: - warmup 

Answer 1

In my experience, RPC performance varies by orders of magnitude, between 5 ms-100 ms to receive data. I suspect this is due to the GAE data center load. Sometimes it gets better, sometimes it gets worse.

Your operation looks very simple. I expect with 3 queries it will take about 20 ms, but it can be up to 300 ms. A steady average of 500 ms sounds very high though.

ndb performs local caching when retrieving objects by identifier. This should work if you are accessing the same users and these requests should be much faster.

I assume that you are doing initial production testing, not dev_appserver. The performance of dev_appserver is not representative.

You don’t know how many iterations you tested, but you can try increasing the number to see if 500ms is really your average.

When you are blocked by simple RPC calls, you cannot optimize your work.

Answer 2

The first obvious point that I see is: do you really need a transaction for each request?

I believe that if most of your queries do not create new entities, it is better to do .get_by_id () outside the transaction. And if the object is not found, start the transaction or even better delay the creation of the object.

 def request_handler(key, data): entity = key.get() if entity: return 'ok' else: defer(_deferred_create, key, data) return 'ok' def _deferred_create(key, data): @ndb.transactional def _tx(): entity = key.get() if not entity: entity = CreateEntity(data) entity.put() _tx() 

This should give a much better response time for requests addressed to the user.

The second and only optimization I see is to use ndb.put_multi () to minimize RPC calls.

PS Not 100% sure, but you can try to disable multithreading (threadsave: no) to get a more stable response time.