Need Java library - local file cache for remote files (e.g. S3)

Question

My application needs a layer between Java and the file system in order to make transparent the fact that the file system contains only a subset of all files (stored on S3). A layer should do a lot of what a regular IO file does, which should open files, block them for reading / writing, etc., but when it opens, it should probably load files and exit closed ones. Another feature that I need is that if the file is locked for reading / writing, an open call can unlock the file and close the existing stream (i.e., disconnect another user). Another is managing temporary files.

Is there anything remotely similar to open source, or do I just need to roll up my sleeves? Should I start from scratch, or are there some hooks in java IO that I have to click?

Answer 1

I would suggest you check apache commons vfs. Even if this is not quite what you need, you can find useful ideas from it.

Answer 2

Perhaps AWS Storage Gateway is worth considering in your case.

Gateway Cached Volumes:. You can store your primary data in Amazon S3 and save locally available data. Gateway-cached volumes provide significant savings on primary storage, minimize the need to scale your storage in place, and maintain low latency access to your frequently accessed data.

Answer 3

This project aims to simplify the creation of a custom file system implementation for Java 7 NIO (java7-fs-base). The author implemented Dropbox FS (java7-fs-dropbox) and started working on S3 (java7-fs-amazon-s3). https://github.com/fge/java7-filesystems