End multipart_upload with boto3?

Question

Tried this:

import boto3 from boto3.s3.transfer import TransferConfig, S3Transfer path = "/temp/" fileName = "bigFile.gz" # this happens to be a 5.9 Gig file client = boto3.client('s3', region) config = TransferConfig( multipart_threshold=4*1024, # number of bytes max_concurrency=10, num_download_attempts=10, ) transfer = S3Transfer(client, config) transfer.upload_file(path+fileName, 'bucket', 'key') 

Result: 5.9 gig file on s3. It does not seem to contain several parts.

I found this example , but part not defined.

 import boto3 bucket = 'bucket' path = "/temp/" fileName = "bigFile.gz" key = 'key' s3 = boto3.client('s3') # Initiate the multipart upload and send the part(s) mpu = s3.create_multipart_upload(Bucket=bucket, Key=key) with open(path+fileName,'rb') as data: part1 = s3.upload_part(Bucket=bucket , Key=key , PartNumber=1 , UploadId=mpu['UploadId'] , Body=data) # Next, we need to gather information about each part to complete # the upload. Needed are the part number and ETag. part_info = { 'Parts': [ { 'PartNumber': 1, 'ETag': part['ETag'] } ] } # Now the upload works! s3.complete_multipart_upload(Bucket=bucket , Key=key , UploadId=mpu['UploadId'] , MultipartUpload=part_info) 

Question: Does anyone know how to use multi-page loading with boto3?

Answer 1

I would advise you to use boto3.s3.transfer for this purpose. Here is an example:

 import boto3 def upload_file( filename ): session = boto3.Session() s3_client = session.client( 's3' ) try: print "Uploading file:", filename tc = boto3.s3.transfer.TransferConfig() t = boto3.s3.transfer.S3Transfer( client=s3_client, config=tc ) t.upload_file( filename, 'my-bucket-name', 'name-in-s3.dat' ) except Exception as e: print "Error uploading: %s" % ( e ) 

Answer 2

Why not use just the copy option in boto3?

 s3.copy(CopySource={'Bucket':sourceBucket, 'Key':sourceKey}, Bucket=targetBucket, Key=targetKey, ExtraArgs={'ACL':'bucket-owner-full-control'}) 

There is information on how to initialize the s3 object and, obviously, additional parameters for the call available here boto3 docs

Answer 3

Your code snippet should explicitly have part → part1 in the dictionary. Typically, you will have several parts (otherwise use multi-part loading), and the 'Parts' list will contain an element for each part.

You may also be interested in the new pythonic S3 interface: http://s3fs.readthedocs.org/en/latest/