How to move drops from BlobStore to Google Cloud Storage?

Question

Our app has been running the App Engine with Blobstore for many years. We would like to move our video files to Google Cloud Storage. What is the best practice for transferring large drops from Blobstore to GCS?

Is it just a matter of using BlobReader and writing bytes to GCS one at a time? Or are there other combinations / tools available?

As for writing to GCS from App Engine, there is no shortage of libraries:

• Blobstore API (can generate BlobKey for GCS objects, BlobStore file APIs are deprecated, but will be valid for one-time use)
• Google Cloud Storage API (deprecated again, but will be valid for one-time use)
• Google Cloud Storage Client Library
• Python library Google Cloud Storage
• Google Cloud Storage JSON API Library
• boto

Any reasons to use one over the other?

Answer 1

I did not have to do this, but I would say that there are no automatic migration tools. So yes, you have to collapse yourself.

My approach would be to batch migrate using cron.yaml , keeping track of which files were migrated so you can serve them differently (according to this page ). Cloud storage provides you with an md5 hash after creating an object that you could use to verify that each file’s migration was error-free before deleting a copy of Blobstore (you can calculate a similar hash while the file is in transit).

Regarding libraries:

• The Google Cloud Storage Client Library uses a revolving download feature to “stream” a file, which will make things smoother in memory. I believe it is fairly reliable (unlike the legacy API / Google Cloud Storage APIs / Files API).
• The JSON API client is at a lower level.
• boto not optimized for use in GAE, but rather on the desktop, and you don’t want to leave Google Cloud to migrate ($$$).
• As far as I know, Blobstore allows you to send files from GCS and upload files to GCS, but not write files from your application as such.