Automatically detect file changes and sync via S3

Question

I have a local directory of media files on a Linux system that I sync with my Amazon S3 account using s3sync script. Currently, I manually run the s3sync script when I know that the media files have been modified.

How can I automatically run a script when changing files?

I was thinking of creating a cron job to run the script every few minutes, but this seems like over processing, because even if there are no changes, the script will still have to scan the entire directory structure, which is quite large.

I also considered incron / inotify , which allows you to run commands when a specific file or directory is changed, but these tools do not seem to automatically support monitoring changes in the entire subdirectory. Correct me if I am wrong, but it seems that incron / inotify can only track files that they clearly need to track. for example, if I wanted to track changes to all files at any level within a directory, I would have to write separate scripts to track changes and deletions of directory files and files to update the list of files and directories controlled by incron.

Are there more effective solutions?

Answer 1

For such tasks I use fssm gem.

create watcher.rb file

require 'fssm' FSSM.monitor('/dir_to_watch/', '**/*') do update {|base, relative| `your_script` } delete {|base, relative| `your_script` } create {|base, relative| `your_script` } end 

then

 ruby watcher.rb 

Of course, you can demonize it if you want.

Answer 2

Here is an example script that you can use instead, and use a simple rsync script.

http://andrewwilkinson.wordpress.com/2011/01/14/rsync-backups-to-amazon-s3/

It basically means using fuses and s3fs ( http://code.google.com/p/s3fs/ ) to set s3 share as a directory on your local file system and use rsync for synchronization 2. A simple cron job will do the trick.

Answer 3

Now there is an effective solution. It was just announced (long overdue):
http://aws.amazon.com/blogs/aws/s3-event-notification/

Very simple to implement - time to throw away all the ugly cron jobs and list loops.