Backup to S3: symbolic links

Question

I am trying to figure out how to backup some data on S3.

We have a local backup system implemented using rsnapshot, and this works fine. We are trying to use s3cmd with the --sync option to simulate rsync to transfer files.

The problem we are facing is that symbolic links are not created as symbolic links, they seem to solve them in a physical file and load them. Does anyone have any suggestions as to why this will happen?

I am wondering if I am missing something obvious or that S3 is simply not suitable for this kind of operation. I could install an EC2 instance and attach some EBS, but it would be preferable to use S3.

Answer 1

There are no symbolic links in S3, but what you can use is a google solution that creates a S3-based file system (FUSE). Further information here:

https://code.google.com/p/s3fs/wiki/FuseOverAmazon

and here:

http://tjstein.com/articles/mounting-s3-buckets-using-fuse/

I hope this helps

Answer 2

Amazon S3 itself has no concept of symbolic links, so I suspect s3cmd is loading a physical file. Its limitation is S3, not s3cmd.

I assume you need the symlink itself? If so, can you gzip / tar your directory with a symlink and download it?

Answer 3

Try using -F, --follow-symlinks OPTiON when using sync . It worked for me.