Git gc uses a lot of memory, even I limited it

Question

I run git on a shared host (uberspace.de) and click on the memory limit when doing git push or git gc . The reported memory consumption is about 1,500 MB.

So, I followed the advice in

stack overflow

It seemed to work once, but after many commits, it no longer helps, and again the memory consumption is about 1500 MB, and the git pack process is deleted.

I take a 120 megabyte SQL dump with very little changes (about 5 lines each time) - so the packaged repository is only about 150 MB, but it is unpacked by about 3 GB (and I click on the bitpack limit).

What pmap do I see, it opens a lot of packages of objects - this is what adds to high memory? How can this be limited?

Finally, I get "object packets that died from signal 15" and mail from the hoster that my process was killed.

 sh-4.1$ git config --list|grep pack pack.windowmemory=25m pack.packsizelimit=25m pack.threads=1 pack.deltecachesize=25m sh-4.1$ git gc Counting objects: 2586, done. ^Zmpressing objects: 15% (163/1085) [1]+ Stopped(SIGTSTP) git gc sh-4.1$ bg [1] git gc & sh-4.1$ pmap 14190 14190: git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --delta-base-offset .git/objects/pack/.tmp-14187-pack 0000000000400000 1848K rx-- /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git 00000000007ce000 36K rw--- /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git 00000000007d7000 280K rw--- [ anon ] 0000000000da1000 1680K rw--- [ anon ] 00007f735dc5d000 161648K rw--- [ anon ] 00007f7377555000 149580K rw--- [ anon ] 00007f73915ee000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.pack 00007f73990b6000 125592K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.pack 00007f73a0b5c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.pack 00007f73a8624000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.pack 00007f73b00ec000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.pack 00007f73b7bb4000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.pack 00007f73bf67c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.pack 00007f73c7144000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.pack 00007f73cec0c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.pack 00007f73d66d4000 125348K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.pack 00007f73de13d000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.pack 00007f73e5c05000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.pack 00007f73ed6cb000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.pack 00007f73f5191000 120872K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.pack 00007f73fc79b000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.pack 00007f7404263000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.pack 00007f740bd29000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.pack 00007f74137ef000 120868K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.pack 00007f741adf8000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.pack 00007f74228c0000 484K rw--- [ anon ] 00007f7422939000 96840K r---- /usr/lib/locale/locale-archive 00007f74287cb000 8K rx-- /lib64/libdl-2.12.so 00007f74287cd000 2048K ----- /lib64/libdl-2.12.so 00007f74289cd000 4K r---- /lib64/libdl-2.12.so 00007f74289ce000 4K rw--- /lib64/libdl-2.12.so 00007f74289cf000 1576K rx-- /lib64/libc-2.12.so 00007f7428b59000 2048K ----- /lib64/libc-2.12.so 00007f7428d59000 16K r---- /lib64/libc-2.12.so 00007f7428d5d000 8K rw--- /lib64/libc-2.12.so 00007f7428d5f000 16K rw--- [ anon ] 00007f7428d63000 92K rx-- /lib64/libpthread-2.12.so 00007f7428d7a000 2048K ----- /lib64/libpthread-2.12.so 00007f7428f7a000 4K r---- /lib64/libpthread-2.12.so 00007f7428f7b000 4K rw--- /lib64/libpthread-2.12.so 00007f7428f7c000 16K rw--- [ anon ] 00007f7428f80000 28K rx-- /lib64/librt-2.12.so 00007f7428f87000 2044K ----- /lib64/librt-2.12.so 00007f7429186000 4K r---- /lib64/librt-2.12.so 00007f7429187000 4K rw--- /lib64/librt-2.12.so 00007f7429188000 1768K rx-- /usr/lib64/libcrypto.so.1.0.1e 00007f7429342000 2044K ----- /usr/lib64/libcrypto.so.1.0.1e 00007f7429541000 108K r---- /usr/lib64/libcrypto.so.1.0.1e 00007f742955c000 48K rw--- /usr/lib64/libcrypto.so.1.0.1e 00007f7429568000 16K rw--- [ anon ] 00007f742956c000 84K rx-- /lib64/libz.so.1.2.3 00007f7429581000 2044K ----- /lib64/libz.so.1.2.3 00007f7429780000 4K r---- /lib64/libz.so.1.2.3 00007f7429781000 4K rw--- /lib64/libz.so.1.2.3 00007f7429782000 128K rx-- /lib64/ld-2.12.so 00007f74297a6000 464K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.pack 00007f742981a000 52K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.pack 00007f7429827000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.pack 00007f7429828000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.pack 00007f7429829000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.pack 00007f742982a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.pack 00007f742982b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.pack 00007f742982c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.pack 00007f742982d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.pack 00007f742982e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.pack 00007f742982f000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.pack 00007f7429831000 20K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.pack 00007f7429836000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.pack 00007f7429838000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.pack 00007f742983a000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.pack 00007f742983c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.pack 00007f742983d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.pack 00007f742983e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.pack 00007f742983f000 20K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.pack 00007f7429875000 4K rw--- [ anon ] 00007f7429876000 400K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.pack 00007f74298da000 512K rw--- [ anon ] 00007f742995a000 44K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.idx 00007f7429965000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.idx 00007f7429966000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.idx 00007f7429967000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.idx 00007f7429968000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.idx 00007f7429969000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.idx 00007f742996a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.idx 00007f742996b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.idx 00007f742996c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.idx 00007f742996d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.idx 00007f742996e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.idx 00007f742996f000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.idx 00007f7429970000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.idx 00007f7429971000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.idx 00007f7429972000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.idx 00007f7429973000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.idx 00007f7429974000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.idx 00007f7429975000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.idx 00007f7429976000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.idx 00007f7429977000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.idx 00007f7429978000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.idx 00007f7429979000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.idx 00007f742997a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.idx 00007f742997b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.idx 00007f742997c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.idx 00007f742997d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.idx 00007f742997e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.idx 00007f742997f000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.idx 00007f7429980000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.idx 00007f7429981000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.idx 00007f7429982000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.idx 00007f7429983000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.idx 00007f7429984000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.idx 00007f7429985000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.idx 00007f7429986000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.idx 00007f7429987000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.idx 00007f7429988000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.idx 00007f7429989000 24K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.idx 00007f742998f000 20K rw--- [ anon ] 00007f7429998000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.idx 00007f7429999000 28K r--s- /usr/lib64/gconv/gconv-modules.cache 00007f74299a0000 4K rw--- [ anon ] 00007f74299a1000 4K r---- /lib64/ld-2.12.so 00007f74299a2000 4K rw--- /lib64/ld-2.12.so 00007f74299a3000 4K rw--- [ anon ] 00007ffe76939000 84K rw--- [ stack ] 00007ffe7698b000 4K rx-- [ anon ] ffffffffff600000 4K rx-- [ anon ] total 2809104K sh-4.1$ error: pack-objects died of signal 15 error: failed to run repack 

Answer 1

maybe a solution

I noticed that when I select packageSizeLimit very little, it is consistent - it increased the .git directory, not reduced it, and this can lead to reaching the bit-bit size limit (unverified)

1) rsync repo to another computer with a lot of RAM and correctly pack it

2) rsync back to memory-limited server

3) increase to pack.packSizeLimit=300m

4) Now after git gc repo does not grow like crazy

5) commit database dump

6) repo doubles its size

7) directly execute a git gc

8) the size becomes thin

9) now click

see how it goes in a few days ...

Answer 2

I have a large repository that I have to compress, and I did not notice any memrory related options. If git-pack-objects need to repackage some objects, it will have to display approximately the same large memory. The only way to manage it is to limit the number of objects that it needs to repack.

In your case, the object is already compressed in small package files (one for each click). So git gc should repack these small package files into large ones. You can try to create empty pack-<HASH>.keep files next to large package files that were created with the last successful repack. Then perhaps the next git gc will not try to share these objects again.