Using gcloud-python in GAE

Question

Using gcloud-python in GAE

I have a bunch of little raspberry Pis that runs some python code that is saved directly to the Datastore (skips GAE) using the gcloud-python storage package. This works great. Now I want to present data through web and mobile clients using the Google App Engine. On my MacBook, I installed GAE using the installer and gcloud via pip. I can write a simple python script and execute it directly from the terminal, which can write and read from the data store via gcloud - this also works great.

However, when I try to include the same code in GAE, it fails. Based on my research, I expect this to be a PATH problem, but after several hours of various attempts, I cannot solve it. Suggestions will be appreciated.

I find this post to be similar to my problem: Google App Engine, change python version

Here are some details that may make a difference:

Python version

$ which python
/usr/bin/python

According to the indicated problem, I set GAE preferences to have Python Path / usr / bin / python. I tried it

$ PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

os.path from Python interpreter

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud installation

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python 2.7 Links

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

Corresponding code

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE Log File

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

+4

google-app-engine google-cloud-datastore gcloud-python google-cloud-python

Sheridan gray Nov 21 '14 at 9:22

source share

2

bossylobster · Answer 1 · 2015-01-22T18:01:36+0000

gcloud-python App Engine, . .

:

install_gcloud.sh, pip gcloud

pip install --target="application/vendor/" gcloud

( , App Engine .)

pip - --target pkg_resources.get_distribution ( ).

, pytz , App Engine, gae-pytz. pytz .

, ( ) oauth2client.client .

commit.

script vendor/ appengine_config.py Darth Vendor:

import darth
darth.vendor('vendor')

, protobuf google ( App Engine, . google.appengine.ext.ndb) __path__ :

import os
import google

curr_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(curr_dir, 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

CAVEAT ( 22 2015 .)

, gcloud-python App Engine 3-5x , App Engine db ndb. , RPC App Engine, gcloud-python HTTP App Engine API Datastore.

. , point .

Jeff O'Neill · Answer 2 · 2014-11-21T15:06:32+0000

GAE Mac Python, Mac. :

ln -s /Library/Python/2.7/site-packages/.../gcloud /Users/sheridangray/Projects/city-pulse-web/gcloud

( ... )

Using gcloud-python in GAE

CAVEAT ( 22 2015 .)

More articles: