Memory exhausted: for large files using diff

Question

I am trying to create a patch using two large folders (~ 7 GB).

Here is how I do it:

$: # diff -Naurbw ... / other-folder> file.patch

But perhaps due to file sizes, the patch is not created and gives an error:

diff: memory exhausted

I tried to make the space more than 15 GB, but the problem still persists.

Can someone help me with the flags I should use?

Answer 1

I recently came across this when I needed to split two large files (> 5Gb each).

I tried using diff with various options, but even --speed-large-files had no effect. Other methods, such as splitting files into smaller ones using xdelta or sorting files according to this suggestion , did not help either. I even grabbed onto a very powerful virtual machine (> 72Gb RAM), but still got this memory exhausted error.

I finally got to work by adding the following parameter to sysctl.conf ( sudo vim /etc/sysctl.conf ):

 vm.overcommit_memory=1 

To make sure that the parameter is actually applied, you can run

 sudo sysctl -p 

Remember to change this setting when done!