How do I get "InternalError: table" dev ~ guestbook !! Are entities "already existing" when I just created a data warehouse?

Question

I am trying to figure out how to configure Test Driven Development for GAE.

I start the tests with:

nosetests -v --with-gae 

I keep getting the error:

 InternalError: table "dev~guestbook!!Entities" already exists 

The data warehouse does not exist until I create it in setUp (), but am I still getting an error that already exists?

I am using the code from the GAE tutorial .

Here is my test code in function_tests.py:

 import sys, os, subprocess, time, unittest, shlex sys.path.append("/usr/local/google_appengine") sys.path.append("/usr/local/google_appengine/lib/yaml/lib") sys.path.append("/usr/local/google_appengine/lib/webapp2-2.5.2") sys.path.append("/usr/local/google_appengine/lib/django-1.5") sys.path.append("/usr/local/google_appengine/lib/cherrypy") sys.path.append("/usr/local/google_appengine/lib/concurrent") sys.path.append("/usr/local/google_appengine/lib/docker") sys.path.append("/usr/local/google_appengine/lib/requests") sys.path.append("/usr/local/google_appengine/lib/websocket") sys.path.append("/usr/local/google_appengine/lib/fancy_urllib") sys.path.append("/usr/local/google_appengine/lib/antlr3") from selenium import webdriver from google.appengine.api import memcache, apiproxy_stub, apiproxy_stub_map from google.appengine.ext import db from google.appengine.ext import testbed from google.appengine.datastore import datastore_stub_util from google.appengine.tools.devappserver2 import devappserver2 class NewVisitorTest(unittest.TestCase): def setUp(self): # Start the dev server cmd = "/usr/local/bin/dev_appserver.py /Users/Bryan/work/GoogleAppEngine/guestbook/app.yaml --port 8080 --storage_path /tmp/datastore --clear_datastore --skip_sdk_update_check" self.dev_appserver = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE) time.sleep(2) # Important, let dev_appserver start up self.testbed = testbed.Testbed() self.testbed.setup_env(app_id='dermal') self.testbed.activate() self.testbed.init_user_stub() # Create a consistency policy that will simulate the High Replication consistency model. # with a probability of 1, the datastore should be available. self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=1) # Initialize the datastore stub with this policy. self.testbed.init_datastore_v3_stub(datastore_file="/tmp/datastore/datastore.db", use_sqlite=True, consistency_policy=self.policy) self.testbed.init_memcache_stub() self.datastore_stub = apiproxy_stub_map.apiproxy.GetStub('datastore_v3') # setup the dev_appserver APP_CONFIGS = ['app.yaml'] # setup client to make sure from guestbook import Author, Greeting if not ( Author.query( Author.email == "[e-mail]").get()): logging.info("create Admin") client = Author( email = "[e-mail]", ).put() Assert( Author.query( Author.email == "[e-mail]").get() ) self.browser = webdriver.Firefox() self.browser.implicitly_wait(3) def tearDown(self): self.browser.quit() self.testbed.deactivate() self.dev_appserver.terminate() def test_submit_anon_greeting(self): self.browser.get('http://localhost:8080') self.browser.find_element_by_name('content').send_keys('Anonymous test post') self.browser.find_element_by_name('submit').submit() Assert.assertEquals(driver.getPageSource().contains('Anonymous test post')) 

Here is the trace:

 test_submit_anon_greeting (functional_tests.NewVisitorTest) ... INFO 2015-05-11 14:41:40,516 devappserver2.py:745] Skipping SDK update check. INFO 2015-05-11 14:41:40,594 api_server.py:190] Starting API server at: http://localhost:59656 INFO 2015-05-11 14:41:40,598 dispatcher.py:192] Starting module "default" running at: http://localhost:8080 INFO 2015-05-11 14:41:40,600 admin_server.py:118] Starting admin server at: http://localhost:8000 WARNING 2015-05-11 14:41:45,008 tasklets.py:409] suspended generator _run_to_list(query.py:964) raised InternalError(table "dev~guestbook!!Entities" already exists) ERROR 2015-05-11 14:41:45,009 webapp2.py:1552] table "dev~guestbook!!Entities" already exists Traceback (most recent call last): File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__ rv = self.handle_exception(request, response, e) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__ rv = self.router.dispatch(request, response) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher return route.handler_adapter(request, response) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__ return handler.dispatch() File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch return self.handle_exception(e, self.app.debug) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch return method(*args, **kwargs) File "/Users/Bryan/work/GoogleAppEngine/guestbook/guestbook.py", line 50, in get greetings = greetings_query.fetch(10) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/utils.py", line 142, in positional_wrapper return wrapped(*args, **kwds) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/query.py", line 1187, in fetch return self.fetch_async(limit, **q_options).get_result() File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 325, in get_result self.check_success() File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along value = gen.throw(exc.__class__, exc, tb) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/query.py", line 964, in _run_to_list batch = yield rpc File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion result = rpc.get_result() File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_query.py", line 2870, in __query_result_hook self._batch_shared.conn.check_rpc_success(rpc) File "/Users/Bryan/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1342, in check_rpc_success raise _ToDatastoreError(err) InternalError: table "dev~guestbook!!Entities" already exists 

Answer 1

There seem to be a few things here.

First, it looks like you are using NoseGAE --with-gae . The plugin handles the setup and breaks your test bench, so you don’t need to. This means that you do not need any self.testbed code, and it can cause internal conflicts. Either switch to the NoseGAE mode, or do not use the --with-gae . If you use NoseGAE, it has the --gae-datastore option, which allows you to set the path to the data store that it will use for your tests. Then, inside your test class, set the property nosegae_datastore_v3 = True property that it is configured for you:

 class NewVisitorTest(unittest.TestCase): # enable the datastore stub nosegae_datastore_v3 = True 

Secondly, the dev_appserver / sqlite collaboration method, the application server loads the sqlite db file into memory and works with it there. When the application server shuts down, it flushes the contents of the database to disk. Since you use the same datastore for your tests as the dev_appserver.py process that you open for selenium, they may or may not see the binding data that you set in your test.

Here is an example from https://github.com/Trii/NoseGAE/blob/master/nosegae.py#L124-L140

 class MyTest(unittest.TestCase): nosegae_datastore_v3 = True nosegae_datastore_v3_kwargs = { 'datastore_file': '/tmp/nosegae.sqlite3', 'use_sqlite': True } def test_something(self): entity = MyModel(name='NoseGAE') entity.put() self.assertNotNone(entity.key.id()) 

Answer 2

I assume this line may be faulty:

 self.testbed.init_datastore_v3_stub(datastore_file="/tmp/datastore/datastore.db", use_sqlite=True, consistency_policy=self.policy) 

Setting datastore_file = "/tmp/datastore/datastore.db" means that you want to reuse this existing data store in your tests

The python code documentation says:

The argument to datastore_file can be the path to an existing data store file or No (default) to use a data store in memory that is initially empty.

Personaly I use them in my tests:

  def setUp(self): self.testbed = testbed.Testbed() self.testbed.activate() self.testbed.init_datastore_v3_stub( consistency_policy=datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0) ) self.testbed.init_memcache_stub() def tearDown(self): self.testbed.deactivate()