Geek Answers Handbook

Download Google App Engine and Amazon S3 files

I know this was asked before, but there really is no clear answer. My problem is that I created a script file upload for GAE and only after that I found out that you can only store files up to aprox. 1 MB in the data warehouse. I can stop you right here if you tell me that if I allow billing, the 1 MB limit is history, but I doubt it.

I need to be able to upload up to 20 MB per file, so I thought maybe I could use Amazon S3. Any ideas on how to do this?

I was told to use a combination of GAE + Ec2 and S3, but I have no idea how this will work.

Thanks, Max

+6
python google-app-engine amazon-s3 amazon-ec2
source share
5 answers

From Amazon S3 Documentation :

  • A user opens a web browser and accesses your web page.

  • Your web page contains an HTTP form containing all the information a user needs to upload content to Amazon S3.

  • A user uploads content directly to Amazon S3.

GAE prepares and maintains a web page, which speeds up the work. The user logs on to S3, a lengthy operation, but it is between your user's browser and Amazon; GAE is not involved.

Part of the S3 protocol is success_action_redirect, which allows you to tell S3 where the browser should be directed if it loads successfully. This redirection may be in GAE.

+13
source share

Google App Engine and EC2 are competitors. They do the same, although GAE provides an environment for your application to work with strict language restrictions, while EC2 provides you with a virtual machine (I think VMWare) that hosts your application.

S3, on the other hand, is a raw api store. You can use SOAP or REST api to access it. If you want to stick with GAE, you can just use the Amazon S3 Python Library to make REST calls from Python to S3.

You, of course, will have to pay for use on S3. It's amazing how granular their billing is. When I started, I was literally blamed for 4 cents a month.

+3
source share

For future use, Google added support for large file uploads (up to 50 MB): a new feature was released as part of the Blobstore API and is discussed here .

+2
source share

Thomas L. Goladay Answer - The correct answer, I suppose. Anyway, just in case, here is a link to the Amazon Web Services SDK for App Engine (Java), which you can use, for example. to download files from App Engine to Amazon S3. ( Edit: Oh, you just noticed - except for S3) http://apetresc.wordpress.com/2010/06/22/introducing-the-gae-aws-sdk-for-java/

Written by Adrian Petrescu. On its website:

[This is] the version of the Amazon Web Services SDK for Java, which will be launched internally from the Google App Engine. It wasn’t if you just turned on the JAR that AWS provides directly in the GAE WAR, because the GAEs security model does not allow Apache Commons HTTP Client for creating sockets and low-level network primitives required to establish an HTTP connection; instead, Google requires you to make all connections through its URLFetch utility

+1
source share

Some links for Google App Engine + S3:

Previous related message ... 10mb limit .

This link shows a small file downloads . I have not yet found an example of large downloads ...

This link shows a different approach, ( with a fix for a known issue )

0
source share

More articles:


All Articles