How to programmatically get Amazon S3 MD5 checksum using boto

Question

Sent messages: Amazon S3 and checksum , How to encode base64 md5 in BASH

I need to load a tar file from a restricted S3 bucket. [Mostly access permissions granted only for download]

After downloading, I have to check the md5 verification amount of the downloaded file for the MD5-Check sum for the data presented in the metadata in S3

I am currently using the S3 file browser to manually mark the "x-amz-meta-md5" content header and check this value compared to the calculated md5 of the downloaded file.

I would like to know if there is a programmatic way to use boto to capture the value of the md5 hash of the S3 file, referred to as metadata.

from boto.s3.connection import S3Connection conn = S3Connection(access_key, secret_key) bucket=conn.get_bucket("test-bucket") rs_keys = bucket.get_all_keys() for key_val in rs_keys: print key_val, key_val.**HOW_TO_GET_MD5_FROM_METADATA(?)** 

Please correct if my understanding is incorrect. I am looking for a way to capture header data programmatically

Answer 1

When boto loads a file using any of the get_contents_to_* methods, it calculates the MD5 bytes checksum that it loads and makes it available as the md5 attribute of the Key object. In addition, S3 sends an ETag header in the response, which represents the idea of ​​the server about the MD5 checksum. This is available as an ETag attribute of the ETag object. So, after downloading the file, you can simply compare the value of these two attributes to see if they match.

If you want to know that S3 believes that MD5 is not loading the file (as shown in your example), you can simply do this:

 for key_val in rs_keys: print key_val, key_val.etag 

Answer 2

It seems that ETag is not md5sum if the file was built after running the multi-part download. I think that in this case you only need to download the file and execute the checksum locally. If the result is correct, a copy of S3 should be good. If the local checksum is wrong, the copy of s3 may be bad or the download may fail. If you no longer have the source file or records of its md5sum, I think you're out of luck. It would be great if the md5sum of the assembled file were available or if there was a way to locally calculate the expected etag of the file to be downloaded via multipart.