Geek Answers Handbook

Why am I getting a heap outOfMemory exception?

I am trying to allocate a large matrix (about 10 GB). I work on a 64 bit machine with a 64 bit JVM. My process should have 2^64 bytes, and I set the JVM heap size to 128G (I have 16 GB of RAM on my machine, if that matters). I realized that I had to get memory from the OS and that unnecessary matrix cells would be replaced by the OS. However, I get the exception above.

Edit:

This is how I define the matrix:

 Jama.Matrix A = new Matrix(num_words, num_documents);

Where num_words is approximately 100k and num_documents is approximately 35k. It is also worth mentioning that type double

Edit2:

Relevant flags:

 -Xms40m -Xmx128g -d64
+6
source share
3 answers

JVM works as its own process, in this regard: JVM requests memory from the OS, which can allocate it either in RAM when swapping.

The memory that you can allocate in java does not depend on your RAM, but on the -Xmx command-line -Xmx that you specify when starting your JVM. If there is not enough memory in RAM, the JVM gets it from swap and (I think) I don’t even know about it.

but

  • on some operating systems (Windows XP), the limit is 4G.
  • the heap in the JVM is not designed to work with disk.
  • swap is limited.

If you need to work with big data, you need to work with BigMemory products (EhCache or Terracotta).

Finally, run jvisualvm or with prameter -verbose:gc to see the heap selection.

+2
source

Here are some descriptions:

 Xms -> the init memory that should be allocated in the start up in MB. Xmx -> the Max amount of memory that your application can get in MB eg Xmx2048m. -XX:MaxNewSize= -> the max S1 and S2 memory size -XX:NewSize= -> init S1 and S2 size

so in your case, if you want to allocate as much memory as you can, so you need to say, for example, 16 * 1024 = 16384 or 16g , and -XX:MaxNewSize= and -XX:NewSize= set it to 40% of your Xmx

+1
source

A few things you should know.

  • Managed memory does not change very well. In fact, this usually leads to your application dying or your OS being blocked (for example, windows that you should use if even 10% of your heap is swapped).
  • The heap is divided into several regions, and it will not change as you might expect. Usually it will scale these areas together, for example. this will make young and long spaces 1: 8, that is, you cannot easily capture almost the entire pile at once. I would try a maximum heap of 192 GB.
  • Most advanced processes support 48-bit address spaces or 256 TB of virtual memory and only 40-bit real memory (1 TB per processor).
  • -d64 only works on Solaris.

In short, you will need about 256 GB of main memory and a large heap that you are really considering what you are doing. Random access to memory is one million times faster than access to disk spaces, and this means that it is not only a million times slower, but is unlikely to work at all.

What you can do is use heap memory, and if you really know what you are doing, you can do this work and be perhaps only 100 times slower than having the memory you need. esp, if you use fast SSD for swap or your matrix is ​​sparse, these are the right ways.

+1
source

More articles:


All Articles