App Engine migration method

Question

Database migrations are a popular pattern, especially with Ruby on Rails. Since migrations indicate how to generate old data in accordance with the new scheme, they can be useful when you have production data that needs to be quickly and reliably converted.

But porting models to the App Engine is difficult, since processing all the objects is sequentially complex, and there is no offline operation to efficiently transfer all transactions into one large transaction.

What are your methods for modifying the "db.Model" schema and transferring data according to the new schema?

Answer 1

That's what I'm doing.

I have a MigratingModel class that inherits all my models. Here is migrating_model.py:

"""Models which know how to migrate themselves""" import logging from google.appengine.ext import db from google.appengine.api import memcache class MigrationError(Exception): """Error migrating""" class MigratingModel(db.Model): """A model which knows how to migrate itself. Subclasses must define a class-level migration_version integer attribute. """ current_migration_version = db.IntegerProperty(required=True, default=0) def __init__(self, *args, **kw): if not kw.get('_from_entity'): # Assume newly-created entities needn't migrate. try: kw.setdefault('current_migration_version', self.__class__.migration_version) except AttributeError: msg = ('migration_version required for %s' % self.__class__.__name__) logging.critical(msg) raise MigrationError, msg super(MigratingModel, self).__init__(*args, **kw) @classmethod def from_entity(cls, *args, **kw): # From_entity() calls __init__() with _from_entity=True obj = super(MigratingModel, cls).from_entity(*args, **kw) return obj.migrate() def migrate(self): target_version = self.__class__.migration_version if self.current_migration_version < target_version: migrations = range(self.current_migration_version+1, target_version+1) for self.current_migration_version in migrations: method_name = 'migrate_%d' % self.current_migration_version logging.debug('%s migrating to %d: %s' % (self.__class__.__name__, self.current_migration_version, method_name)) getattr(self, method_name)() db.put(self) return self 

MigratingModel intercepts the conversion from the raw data store object to the full db.Model instance. If current_migration_version behind the class of the last migration_version , then it runs a series of migrate_N() methods that do a heavy lift.

For example:

 """Migrating model example""" # ...imports... class User(MigratingModel): migration_version = 3 name = db.StringProperty() # deprecated: use first_name and last_name first_name = db.StringProperty() last_name = db.StringProperty() age = db.IntegerProperty() invalid = db.BooleanProperty() # to search for bad users def migrate_1(self): """Convert the unified name to dedicated first/last properties.""" self.first_name, self.last_name = self.name.split() def migrate_2(self): """Ensure the users' names are capitalized.""" self.first_name = self.first_name.capitalize() self.last_name = self.last_name.capitalize() def migrate_3(self): """Detect invalid accounts""" if self.age < 0 or self.age > 85: self.invalid = True 

On a busy site, the migrate () method should retry if db.put() fails, and possibly db.put() critical error if the migration does not work.

I have not received this yet, but at some point I will probably mix my migrations from a separate file.

Final thoughts

Hard to test App Engine. It is difficult to access your production data in a test environment, and it is currently difficult to make a coherent snapshot backup. Therefore, for major changes, consider creating a new version that uses a completely different model name that imports from the old model and migrates as needed. (For example, User2 instead of User ). Thus, if you need to revert to a previous version, you have an effective data backup.