1MB quota limit for blobstore in Google App Engine?

Question

I use the App Engine (version 1.4.3) to write blobstore directly to save images. when I try to save an image larger than 1 MB, I get the following exception

com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call datastore_v3.Put() was too large. 

I thought the limit for each object is 2 GB

Here is the java code that saves the image

 private void putInBlobStore(final String mimeType, final byte[] data) throws IOException { final FileService fileService = FileServiceFactory.getFileService(); final AppEngineFile file = fileService.createNewBlobFile(mimeType); final FileWriteChannel writeChannel = fileService.openWriteChannel(file, true); writeChannel.write(ByteBuffer.wrap(data)); writeChannel.closeFinally(); } 

Answer 1

The maximum object size is 2 GB, but each API call can handle a maximum of 1 MB. At least for reading, but I suppose this might be the same for writing. So you can try to split the object record into 1 MB chunks and see if that helps.

Answer 2

This is how I read and write large files:

 public byte[] readImageData(BlobKey blobKey, long blobSize) { BlobstoreService blobStoreService = BlobstoreServiceFactory .getBlobstoreService(); byte[] allTheBytes = new byte[0]; long amountLeftToRead = blobSize; long startIndex = 0; while (amountLeftToRead > 0) { long amountToReadNow = Math.min( BlobstoreService.MAX_BLOB_FETCH_SIZE - 1, amountLeftToRead); byte[] chunkOfBytes = blobStoreService.fetchData(blobKey, startIndex, startIndex + amountToReadNow - 1); allTheBytes = ArrayUtils.addAll(allTheBytes, chunkOfBytes); amountLeftToRead -= amountToReadNow; startIndex += amountToReadNow; } return allTheBytes; } public BlobKey writeImageData(byte[] bytes) throws IOException { FileService fileService = FileServiceFactory.getFileService(); AppEngineFile file = fileService.createNewBlobFile("image/jpeg"); boolean lock = true; FileWriteChannel writeChannel = fileService .openWriteChannel(file, lock); writeChannel.write(ByteBuffer.wrap(bytes)); writeChannel.closeFinally(); return fileService.getBlobKey(file); } 

Answer 3

As Brummo suggested, if you split it into pieces <1MB works. Here is the code.

 public BlobKey putInBlobStoreString(String fileName, String contentType, byte[] filebytes) throws IOException { // Get a file service FileService fileService = FileServiceFactory.getFileService(); AppEngineFile file = fileService.createNewBlobFile(contentType, fileName); // Open a channel to write to it boolean lock = true; FileWriteChannel writeChannel = null; writeChannel = fileService.openWriteChannel(file, lock); // lets buffer the bitch BufferedInputStream in = new BufferedInputStream(new ByteArrayInputStream(filebytes)); byte[] buffer = new byte[524288]; // 0.5 MB buffers int read; while( (read = in.read(buffer)) > 0 ){ //-1 means EndOfStream ByteBuffer bb = ByteBuffer.wrap(buffer); writeChannel.write(bb); } writeChannel.closeFinally(); return fileService.getBlobKey(file); }