Check if key exists in bucket in s3 using boto3

Question

I would like to know if a key exists in boto3. I can encode the contents of the bucket and check the key if it matches.

But it seems longer and redundant. Boto3 white papers clearly indicate how to do this.

Maybe I'm missing the obvious. Can someone please tell me how I can achieve this.

Answer 1

Boto 2 boto.s3.key.Key an object used to use the exists method, which checks if a key existed on S3 by executing a HEAD request and looking at the result, but it seems that this is no longer there. You have to do it yourself:

 import boto3 import botocore s3 = boto3.resource('s3') try: s3.Object('my-bucket', 'dootdoot.jpg').load() except botocore.exceptions.ClientError as e: if e.response['Error']['Code'] == "404": # The object does not exist. ... else: # Something else has gone wrong. raise else: # The object does exist. ... 

load() executes a HEAD request for a single key, which is fast, even if the object in question is large or you have many objects in your bucket.

Of course, you can check if an object exists because you plan to use it. If so, you can just forget about load() and do get() or download_file() directly, and then handle the error case there.

Answer 2

I'm not a big fan of using exceptions for control flow. This is an alternative approach that works in boto3:

 import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('my-bucket') key = 'dootdoot.jpg' objs = list(bucket.objects.filter(Prefix=key)) if len(objs) > 0 and objs[0].key == key: print("Exists!") else: print("Doesn't exist") 

Answer 3

The easiest way I've found (and probably the most efficient) is this:

 import boto3 from botocore.errorfactory import ClientError s3 = boto3.client('s3') try: s3.head_object(Bucket='bucket_name', Key='file_path') except ClientError: # Not found pass 

Answer 4

In Boto3, if you check a folder (prefix) or file using list_objects. You can use the existence of “Content” in a dict response as a check to see if an object exists. This is another way to avoid try / except catch, as @EvilPuppetMaster offers

 import boto3 client = boto3.client('s3') results = client.list_objects(Bucket='my-bucket', Prefix='dootdoot.jpg') return 'Contents' in results 

Answer 5

Not only client , but also bucket :

 import boto3 import botocore bucket = boto3.resource('s3', region_name='eu-west-1').Bucket('my-bucket') try: bucket.Object('my-file').get() except botocore.exceptions.ClientError as ex: if ex.response['Error']['Code'] == 'NoSuchKey': print('NoSuchKey') 

Answer 6

 import boto3 client = boto3.client('s3') s3_key = 'Your file without bucket name eg abc/bcd.txt' bucket = 'your bucket name' content = client.head_object(Bucket=bucket,Key=s3_key) if content.get('ResponseMetadata',None) is not None: print "File exists - s3://%s/%s " %(bucket,s3_key) else: print "File does not exist - s3://%s/%s " %(bucket,s3_key) 

Answer 7

You can use S3F , which are essentially a shell for boto3, which provides typical file system style operations:

 import s3fs s3 = s3fs.S3FileSystem() s3.exists('myfile.txt') 

Answer 8

FWIW, here are the very simple functions that I use

 import boto3 def get_resource(config: dict={}): """Loads the s3 resource. Expects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be in the environment or in a config dictionary. Looks in the environment first.""" s3 = boto3.resource('s3', aws_access_key_id=os.environ.get( "AWS_ACCESS_KEY_ID", config.get("AWS_ACCESS_KEY_ID")), aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY", config.get("AWS_SECRET_ACCESS_KEY"))) return s3 def get_bucket(s3, s3_uri: str): """Get the bucket from the resource. A thin wrapper, use with caution. Example usage: >> bucket = get_bucket(get_resource(), s3_uri_prod)""" return s3.Bucket(s3_uri) def isfile_s3(bucket, key: str) -> bool: """Returns T/F whether the file exists.""" objs = list(bucket.objects.filter(Prefix=key)) return len(objs) == 1 and objs[0].key == key def isdir_s3(bucket, key: str) -> bool: """Returns T/F whether the directory exists.""" objs = list(bucket.objects.filter(Prefix=key)) return len(objs) > 1 

Answer 9

Try it easy

 import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('mybucket_name') # just Bucket name file_name = 'A/B/filename.txt' # full file path obj = list(bucket.objects.filter(Prefix=file_name)) if len(obj) > 0: print("Exists") else: print("Not Exists") 

Answer 10

This is the easiest method I've found:

 response = s3_client.list_objects_v2( Bucket=bucket_name, MaxKeys=1, Prefix=key_name ) file_exists = response['KeyCount'] > 0: 

Answer 11

If you have less than 1000 in the directory or bucket, you can get their set even after checking if there is such a key in this set:

 files_in_dir = {d['Key'].split('/')[-1] for d in s3_client.list_objects_v2( Bucket='mybucket', Prefix='my/dir').get('Contents') or []} 

Such code works even if my/dir does not exist.

http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2

Answer 12

 S3_REGION="eu-central-1" bucket="mybucket1" name="objectname" import boto3 from botocore.client import Config client = boto3.client('s3',region_name=S3_REGION,config=Config(signature_version='s3v4')) list = client.list_objects_v2(Bucket=bucket,Prefix=name) for obj in list.get('Contents', []): if obj['Key'] == name: return True return False 

Answer 13

There is one simple way by which we can check if a file exists in the S3 bucket. We do not need to use an exception for this

 sesssion = boto3.Session(aws_access_key_id, aws_secret_access_key) s3 = session.client('s3') object_name = 'filename' bucket = 'bucketname' obj_status = s3.list_objects(Bucket = bucket, Prefix = object_name) if obj_status.get('Contents'): print("File exists") else: print("File does not exists") 

Answer 14

For boto3, ObjectSummary can be used to check if an object exists.

Contains a summary of the item stored in the Amazon S3 trash. This object does not contain the full metadata of the object or its contents.

 import boto3 from botocore.errorfactory import ClientError def path_exists(path, bucket_name): """Check to see if an object exists on S3""" s3 = boto3.resource('s3') try: s3.ObjectSummary(bucket_name=bucket_name, key=path).load() except ClientError as e: if e.response['Error']['Code'] == "404": return False else: raise e return True path_exists('path/to/file.html') 

In ObjectSummary.load

Invokes s3.Client.head_object to update the attributes of the ObjectSummary resource.

This shows that you can use ObjectSummary instead of Object if you plan not to use get() . The load() function does not retrieve the object, but only a summary.

Answer 15

If you are looking for a key that is equivalent to a directory, then you might need this approach

 session = boto3.session.Session() resource = session.resource("s3") bucket = resource.Bucket('mybucket') key = 'dir-like-or-file-like-key' objects = [o for o in bucket.objects.filter(Prefix=key).limit(1)] has_key = len(objects) > 0 

This works for a parent key or a key that matches the file, or a key that does not exist. I tried the preferred approach above and failed on the parent keys.

Answer 16

I noticed that in order for botocore.exceptions.ClientError to botocore.exceptions.ClientError thrown using botocore.exceptions.ClientError we need to set botocore. botocore occupies 36M of disk space. This is especially important if we use aws lambda functions. Instead, if we just use the exception, we can skip the extra library!

• I check that the file extension is .csv
• This will not throw an exception if the basket does not exist!
• This will not throw an exception if the cart exists, but the object does not exist!
• This throws an exception if the basket is empty!
• This throws an exception if the basket does not have permissions!

The code looks like this. Please share your thoughts:

 import boto3 import traceback def download4mS3(s3bucket, s3Path, localPath): s3 = boto3.resource('s3') print('Looking for the csv data file ending with .csv in bucket: ' + s3bucket + ' path: ' + s3Path) if s3Path.endswith('.csv') and s3Path != '': try: s3.Bucket(s3bucket).download_file(s3Path, localPath) except Exception as e: print(e) print(traceback.format_exc()) if e.response['Error']['Code'] == "404": print("Downloading the file from: [", s3Path, "] failed") exit(12) else: raise print("Downloading the file from: [", s3Path, "] succeeded") else: print("csv file not found in in : [", s3Path, "]") exit(12) 

Answer 17

Here is a solution that works for me. One caveat is that I know the exact key format in advance, so I only list one file

 import boto3 # The s3 base class to interact with S3 class S3(object): def __init__(self): self.s3_client = boto3.client('s3') def check_if_object_exists(self, s3_bucket, s3_key): response = self.s3_client.list_objects( Bucket = s3_bucket, Prefix = s3_key ) if 'ETag' in str(response): return True else: return False if __name__ == '__main__': s3 = S3() if s3.check_if_object_exists(bucket, key): print "Found S3 object." else: print "No object found." 

Answer 18

Departure

 bucket.get_key( key_name, headers=None, version_id=None, response_headers=None, validate=True ) 

Check if a specific key exists in the bucket. This method uses a HEAD request to check for a key. Return: an instance of a Key or None object

from Documents Boto S3

You can simply call bucket.get_key (key name) and check if the returned object is None.