Java memory overuse

Question

In my project, I constantly compress small data blocks. Now I find out that jvm then grows to 6 GB of RAM (resident (RES) RAM, not shared or virtual or so), and then dies due to lack of memory. As if the garbage collector never works or so. I pulled out the appropriate code and pasted it below. When I run it (java6, 32-bit Linux), it grows to 1 GB of RAM. Has anyone figured out how to reduce memory usage?

import java.util.Random; import java.util.zip.Deflater; import java.util.zip.Inflater; class test { int blockSize = 4096; Random r = new Random(); public test() throws Exception { blockSize = 4096; byte [] data = new byte[blockSize]; for(int index=0; index<blockSize; index++) data[index] = (byte)r.nextInt(); for(long cnt=0; cnt<1000000; cnt++) { byte [] result = compress(data); if (result != null) data[0] = result[0]; } } byte [] compress(byte [] in) { assert in.length == blockSize; Deflater compresser = new Deflater(); compresser.setInput(in); compresser.finish(); byte [] out = new byte[in.length]; int outLen = compresser.deflate(out); if (outLen < blockSize) { byte [] finalOut = new byte[outLen]; System.arraycopy(out, 0, finalOut, 0, outLen); return finalOut; } return null; } public static void main(String [] args) throws Exception { new test(); } } 

Answer 1

Well, Volkert van Heusden solved his problem, but to summarize:

At the beginning of the compress(byte [] in) method, we create a java.util.zip.Deflater .

We use Deflater to perform some actions, and then leave the compress() method. We are losing our link to Deflater -variable. At this point, Deflater no longer in use and is waiting for the garbage collector to kill it.

Deflater allocates both a heap of Java memory and C / C ++ / heap internal memory. The native heap memory allocated by the Deflater symbol will be held until the garbage collector Deflater.finalize Deflater.finalize Deflater.finalize method. If the garbage collector does not work fast enough (there may be a lot of free java heap memory), we may end up with a bunch of C / C ++ memory. If this happens, we get "Out of memory" -errors.

Oracle Error Report JDK-4797189 is probably related. It contains a piece of code that illustrates and reproduces the problem:

 public class Bug { public static void main( String args[] ) { while ( true ) { /* If ANY of these two lines is not commented, the JVM runs out of memory */ final Deflater deflater = new Deflater( 9, true ); final Inflater inflater = new Inflater( true ); } } } 

The solution is to free up resources when you are done by calling the Deflater.end() method (or Inflater.end() ).

Answer 2

Well, it seems to me that there is no memory leak in the code, so it actually seems that the virtual machine is not GC-byte arrays.

"Has anyone figured out how to reduce memory usage?"
Well, I would try with

 byte firstByteOfDataWhichIsCompressedAndThenUncompressed(byte [] in) { ... } 

which specifically returns the first byte of an uncompressed array, not the entire array. I know this is a terrible method name, and I hope you find the best.

Following code

  for(long cnt=0; cnt<1000000; cnt++) { byte [] result = compress(data); if (result != null) data[0] = result[0]; } 

will become

  for(long cnt=0; cnt<1000000; cnt++) data[0] = firstByteOfDataWhichIsCompressedAndThenUncompressed(data);