Django storage with s3-boto browser

Question

I have a django project that uses django-storage over s3-boto.

The problem is that every file located on S3 cannot be cached because the URL is changed from every call.

here are two calls generated by django storage:

https://my.s3.amazonaws.com/cache/user_6/profile_pic/profile_profile_picture_thumbnail.jpg?Signature=HlVSayUIJj6dMyk%2F4KBtFlz0uJs%3D&Expires=1364418058&AWSAccessKeyId=[awsaccesskey] https://my.s3.amazonaws.com/cache/user_6/profile_pic/profile_profile_picture_thumbnail.jpg?Signature=xh2VxKys0pkq7yHpbJmH000wkwg%3D&Expires=1364418110&AWSAccessKeyId=[awsaccesskey] 

As you can see, the signature is different. What can I do to prevent it from breaking my browser cache?

Answer 1

In your settings just add the following:

 AWS_QUERYSTRING_AUTH = False 

This will ensure the creation of URLs for files without additional parameters. Your urls look like this:

 https://my.s3.amazonaws.com/cache/user_6/profile_pic/profile_profile_picture_thumbnail.jpg 

Answer 2

It is best to subclass the Boto S3 Storage Backend and override the URL method.

/project/root/storage.py

 from django.conf import settings from storages.backends.s3boto import S3BotoStorage class S3Storage(S3BotoStorage): def url(self, name): name = self._clean_name(name) return '{0}{1}'.format(settings.MEDIA_URL, name) 

/project/root/settings.py

 MEDIA_URL = 'https://my.s3.amazonaws.com/' DEFAULT_FILE_STORAGE = 'project.storage.S3Storage' AWS_ACCESS_KEY_ID = '*******' AWS_SECRET_ACCESS_KEY = '********' AWS_STORAGE_BUCKET_NAME = 'your-bucket' 

Just make sure your images are publicly available.

Answer 3

When AWS_QUERYSTRING_AUTH = True (default), django will generate a temporary url every time we pick up the URL.

If you do not want to generate a temporary URL:

Add AWS_QUERYSTRING_AUTH = False to your settings.py

If you still want a temporary URL:

Temporary URLs will be valid for AWS_QUERYSTRING_EXPIRE seconds (3600 by default). Therefore, we can cache this temporary url (until we cache it longer than it is valid). Thus, we can return the same URL for subsequent page requests, which allows the browser to retrieve the client from the cache.

settings.py

 # We subclass the default storage engine to add some caching DEFAULT_FILE_STORAGE = 'project.storage.CachedS3Boto3Storage' 

Project /storage.py

 import hashlib from django.conf import settings from django.core.cache import cache from storages.backends.s3boto3 import S3Boto3Storage class CachedS3Boto3Storage(S3Boto3Storage): """ adds caching for temporary urls """ def url(self, name): # Add a prefix to avoid conflicts with any other apps key = hashlib.md5("CachedS3Boto3Storage_%s" % name).hexdigest() result = cache.get(key) if result: return result # No cached value exists, follow the usual logic result = super(CachedS3Boto3Storage, self).url(name) # Cache the result for 3/4 of the temp_url lifetime. try: timeout = settings.AWS_QUERYSTRING_EXPIRE except: timeout = 3600 timeout = int(timeout*.75) cache.set(key, result, timeout) return result 

Answer 4

Protecting some file storage

Most of your media downloads are user avatars, for example, you want to be public. But if you have media that requires authentication before you can access it - say, a PDF summary that only members can access, then you don’t want S3BotoStorages to have a public S3 ACL by default. Here we do not need a subclass, because we can pass an instance, and not refer to the class.

so first remove protection for all file fields in your settings and add cache control

 AWS_HEADERS = { 'Cache-Control': 'max-age=86400', } # By default don't protect s3 urls and handle that in the model AWS_QUERYSTRING_AUTH = False 

Then make the desired protected field to use your protected path

 from django.db import models import storages.backends.s3boto protected_storage = storages.backends.s3boto.S3BotoStorage( acl='private', querystring_auth=True, querystring_expire=600, # 10 minutes, try to ensure people won't/can't share ) class Profile(models.Model): resume = models.FileField( null=True, blank=True, help_text='PDF resume accessible only to members', storage=protected_storage, ) 

But you also need to use your regular storage when you are in dev, and usually you use local storage, so I personally communicated with it

 if settings.DEFAULT_FILE_STORAGE == 'django.core.files.storage.FileSystemStorage': protected_storage = FileSystemStorage() logger.debug('Using FileSystemStorage for resumes files') else: protected_storage = S3BotoStorage( acl='private', querystring_auth=True, querystring_expire=86400, # 24Hrs, expiration try to ensure people won't/can't share after 24Hrs ) logger.debug('Using protected S3BotoStorage for resumes files') 

REF: https://tartarus.org/james/diary/2013/07/18/fun-with-django-storage-backends

Answer 5

If you want to use signed URLs, but still want to cache them before they expire, just use the daango built-in caching :

 from django.core.cache import cache class MyModel(models.Model): #using django-storages with AWS_QUERYSTRING_AUTH=True media = models.FileField() @property def url(self): """ Return signed url, re-using cached value if not expired. """ #Refresh url if within n seconds of expiry to give clients #time to retrieve content from bucket. #Make sure AWS_QUERYSTRING_EXPIRE is sufficiently larger than n n = 30 time_to_expiry = settings.AWS_QUERYSTRING_EXPIRE - n #Create a unique key for this instance url in the cache key = '{0}{1}'.format(self.__class__.__name__, self.pk) url = cache.get(key) if not url: url = self.media.url #refresh url via django-storages cache.set(key, url, time_to_expiry) return url