Geek Answers Handbook

Java.lang.OutOfMemoryError although a lot

I am trying to read a 2.5 GB TXT file in my application. I am running Win7 x64 and have 43 GB of available memory (out of 64 GB). I tried playing with -Xmx -XX: MaxParmSize -XX: ParmSize, etc. None of this affects the error. What else could I try? This mistake seems very strange, since I, of course, have enough space for heaps.

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Unknown Source) at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source) at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) at java.lang.AbstractStringBuilder.append(Unknown Source) at java.lang.StringBuilder.append(Unknown Source) at j.utilities.IO.loadString(IO.java:187) at j.utilities.IO.loadString(IO.java:169) at city.PreProcess.main(PreProcess.java:78)

I run

 java version "1.7.0_09" Java(TM) SE Runtime Environment (build 1.7.0_09-b05) Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

Thank you very much in advance.

=============== ANSWER ===============

Ok i just tested it with

 StringBuilder sb = new StringBuilder(); for ( int i=1; i<Integer.MAX_VALUE; i++ ) sb.append("x");

and received

 Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Unknown Source) ...

So this is really a StringBuilder that is trying to build an array larger than Integer.MAX_VALUE.

In case of interest

 StringBuilder sb = new StringBuilder(); int i=1; try { for ( ; i<Integer.MAX_VALUE; i++ ) sb.append("x"); } catch ( OutOfMemoryError e ) { System.out.println(i); // OUTPUT: 1207959551 System.out.println(Integer.MAX_VALUE); // OUTPUT: 2147483647 }

With StringBuilder you can copy 1 207 955 550 characters - much less Integer.MAX_VALUE.

+6
source share
4 answers

You are trying to allocate an array that is too large. This is because you are trying to create a very long string. Since arrays are indexed by an integer, an array cannot have more than Integer.MAX_VALUE elements. Even if your heap size is very large, you cannot allocate an array with more than Integer.MAX_VALUE elements, simply because you cannot index its elements with Integer . See Maximum Java Array Size for more details.

+8
source

You can create a new StringBuilder with a size, for example.

  StringBuilder sb = new StringBuilder(Integer.MAX_VALUE);

The problem is that you are trying to read a file that is larger than the StringBuilder in its array can have. You have several options, for example:

1) Do you really need to immediately read the entire file in memory? If so, you will have to read it in several StringBuilders.

2) Process the file sequentially.

3) Read it in a compressed structure and unpack the necessary parts when you need them.

+2
source

You should examine the -Xmsn option for the java command.

Indicates the initial size of the memory allocation pool.

Edit: I see you have already done this.

0
source

You can store data in stitch buffer data: List in define interval and clear the StringBuffer. hope this helps you. let me know if additional information is required.

0
source

More articles:


All Articles