Sync S3 folders / buckets

Question

I have an S3 bucket that contains static content for all of my production customers. I also have an intermediate environment that I use for testing before I deploy to production. I also want the staging environment to point to S3 for testing downloads and other features. The problem is that I do not want the staging server to refer to the same production bucket / folder s3, because there is a risk of overriding production files.

My solution is to use another folder in the same bucket or create another bucket that I can periodically update with the contents of the production bucket. Is there a way to easily sync two folders or buckets on Amazon S3?

Any other suggestions for managing this type of scenario are also welcome.

Answer 1

CloudBerry Explorer comes with a PowerShell command line interface, and you can find out here how to use it for synchronization.

Answer 2

s3cmd is a good CLI utility that you can use in cronjob. It is even a sync function similar to * nix rsync.

Answer 3

There's also DragonDisk - like a browser for cloud browsers and other amazon s3 clients - with the exception of the free and multi-platform (QT).

It performs synchronization tasks, both local ↔ s3 and s3 ↔ s3. It also has a command line interface that can perform synchronization: http://www.dragondisk.com/faq.html

Answer 4

Here is an example on S3CMD for your reference https://www.admon.org/system-tuning/sync-two-amazon-s3-buckets/

Answer 5

Go to CloudBerry Explorer and you can sync data between your local computer and Amazon S3. Not what you want, but it will help you get started. More here