The exception "Content-MD5 you specified does not match what we received"

Question

I have an exception that I have never experienced before when testing my application that uploads a file from ec2 to s3. Content:

Exception in thread "Thread-1" com.amazonaws.services.s3.model.AmazonS3Exception: The Content-MD5 you specified did not match what we received. (Service: Amazon S3; Status Code: 400; Error Code: BadDigest; Request ID: 972CB8E04388AB20), S3 Extended Request ID: T7bmFnQ2RlGWlJD+aGYfTy97XZw88pbQrwNB8YCezSjyq6O2joxHRP/6ko+Q2zZeGewkw4x/90k= at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1383) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3676) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1439) at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131) at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123) at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139) at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 

What can I do to fix this error? In my application, I used the same code as before.

Answer 1

I think I solved my problem. Finally, I found that some of my files actually changed at boot time. Since the file is generated by another thread, loading and generation are performed at the same time. The file cannot be generated immediately, and during file generation it can be downloaded at the same time, the file actually changed during the download.

The md5 file is created at the beginning of the AmazonS3Client download, then the entire file is uploaded to S3, at this time the file is different from the file loaded at the beginning, so md5 actually changed. I changed my program to a single-threaded program, and the problem did not return again.

Answer 2

Another reason for this problem is to run such code (python)

 with open(filename, 'r') as fd: self._bucket1.put_object(Key=key, Body=fd) self._bucket2.put_object(Key=key, Body=fd) 

In this case, the file object (fd) points to the end of the file when it reaches line 3, so we get the "Content MD5" error, to avoid this, we will need to tell the file reader to return to the original position in the file

 with open(filename, 'r') as fd: bucket1.put_object(Key=key, Body=fd) fd.seek(0) bucket2.put_object(Key=key, Body=fd) 

Thus, we will not get the aforementioned Boto error.

Answer 3

I also ran into this error when I was doing something like this:

 InputStream productInputStream = convertImageFileToInputStream(file); InputStream thumbnailInputStream = generateThumbnail(productInputStream); String uploadedFileUrl = amazonS3Uploader.uploadToS3(BUCKET_PRODUCTS_IMAGES, productFilename, productInputStream); String uploadedThumbnailUrl = amazonS3Uploader.uploadToS3(BUCKET_PRODUCTS_IMAGES, productThumbnailFilename, thumbnailInputStream); 

The generateThumbnail method manipulated productInputStream using a third-party library. Since I could not change the third-party library, I just performed the download first:

 InputStream productInputStream = convertImageFileToInputStream(file); // do this first... String uploadedFileUrl = amazonS3Uploader.uploadToS3(BUCKET_PRODUCTS_IMAGES, productFilename, productInputStream); /// and then this... InputStream thumbnailInputStream = generateThumbnail(productInputStream); String uploadedThumbnailUrl = amazonS3Uploader.uploadToS3(BUCKET_PRODUCTS_IMAGES, productThumbnailFilename, thumbnailInputStream); 

... and added this line to my generateThumbnail method:

 productInputStream.reset();