Version control on large files

Question

We are happy to use SVN for SCM at work. I currently have our binary assets in the same SVN repository as our code. SVN supports very large files (it transfers them "thread-wise" to maintain memory performance), but it is SLOOWWWWW.

What asset management software do you recommend for about one billion (and growth) assets? We would prefer branching and merging (different assets and configuration files go to different clients).

Answer 1

Please remember that merging binary files is almost impossible! At least automatically. At least I have never heard of a program that supports tripartite merging in binary format. Not to mention conflict resolution.

This is why most asset management tools do not have branching, because it makes little sense, since you cannot merge again. Lock and linear history is better. If you want to branch out, make a copy of the history file.

Answer 2

Perforce is the only version control system I've heard for huge files and entire projects. It is free for two seats, but quite expensive for more users (about $ 900 per seat). I heard that it can handle repositories up to terabytes in size.

git might be another option. It behaves somewhat differently than SVN, but is intended for large projects (such as the Linux kernel). I'm not sure if this is good for large binaries.

Answer 3

At my company, we use Documentum as an ECMS, managing batches (and I really mean a lot) of binaries. Documentum (or other ECMS, such as Alfresco) should be the "right" document management solution. Documentum supports tagging and forking and can provide files in the form of WebDAV (so integration into a workflow can be more or less transparent).

This is a theory. In practice, we found Documentum to be slow, hard to configure, and manageable. And frankly, even if we have many documents, most of them are no more than a few moles.

This answer is more than what you should not do than what you should do ... sorry ...

Answer 4

Extending additional Mercurial files might help. It adds the ability to mark files as "large." Stores these files in a central repository separate from the rest of the repository. Uses hashes to determine which versions of large files you need, and downloads only the ones you need right now.

Answer 5

If you need to manage HUGE files, try Plastic SCM (www.plasticscm.com). AFAIK Perforce is also an option, but branching and merging are not so strong.