GAE app receives socket errors when communicating with BigQuery

Question

Our python application for GAE communicates with BigQuery using the Google Api Client for Python (we are currently using version 1.3.1 ) using special GAE-based authentication assistants. Very often, we get a socket error when communicating with BigQuery.

In particular, we create the Google python client API as follows

1. bq_scope = 'https://www.googleapis.com/auth/bigquery' 2. credentials = AppAssertionCredentials(scope=bq_scope) 3. http = credentials.authorize(httplib2.Http()) 4. bq_service = build('bigquery', 'v2', http=http) 

Then we interact with the BQ service and get the following error

File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/gae_override/httplib.py", line 536, in getresponse 'An error occurred while connecting to the server:% s'% e) Error: when An error occurred while connecting to the server: Unable to get url: [api url ...]

The above error is of type google.appengine.api.remote_socket._remote_socket_error.error , and not an exception that wraps the error.

Initially, we thought that this could be due to a timeout, so we also tried to set line 3 of the timeout change in the above snippet to

 3. http = credentials.authorize(httplib2.Http(timeout=60)) 

However, according to the client library log output, an API call takes less than 1 second to fail, and explicitly specifying a timeout did not change the behavior of the system.

Please note that the error occurs in different API calls, and not just in one, and this usually happens with very light operations, for example, we often see an error when polling BQ for job status and rarely when retrieving data. When we start the operation again, the system works.

Any idea why this might happen, and perhaps the best way to handle this?