We recently upgraded our large webapp (runs on jboss 5) from java 6 to java 7.
Within hours, we saw an OutOfMemory error, and it looks like it has ended.
We are launching a 32-bit JVM, so we limit it to 4 GB, and 2 GB is allocated to the JVM.
In java 6, the whole process took about 2.3 GB, but with java 7 it increased significantly, and we reached the limit of 4 GB without starting the GC completely, because the Java heap was still not full.
The stack trace showed that the unmarshalling XML code creates a new SAXParserFactory for each request, and the Inflater class for unpacking the jar file stores a lot of data in the native heap (~ 200,000 Inflater instances). This seems pretty inefficient to me - both the new SAXParserFactory, and several Inflaters. It may be red herring, but I don't have java 6 version to compare behavior.
By launching gc manually every few hours, the application without any problems quickly kisses for several days. But this is not a solution.
The question is why the memory allocation has increased by a whopping 1.5 GB, and how can I stop this?
I expect a newer version of java to be more efficient, faster, and so on. But instead, we have what looks like a memory leak in the JVM.
In any case, the stack trace looks like this:
# There is insufficient memory for the Java Runtime Environment to continue.
Some useful links I found on this topic:
http://practicalcloudcomputing.com/post/444939181/outofmemoryjnigzip
http://www.javacodegeeks.com/2013/01/java-heap-space-native-heap-and-memory-problems.html
JVM parameters are as follows:
-server -Xms2048m -Xmx2048m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:SurvivorRatio=4 -XX:MinHeapFreeRatio=11 -XX:PermSize=80m -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+DisableExplicitGC -Djava.awt.headless=TRUE -DUseSunHttpHandler=TRUE -Dsun.net.client.defaultConnectTimeout=25000 -Dsun.net.client.defaultReadTimeout=50000 -Dfile.encoding=UTF-8 -Xloggc:gc.log Dcom.sun.management.jmxremote.port=9003 -Dcom.sun.management.jmxremote.authenticate=FALSE -Dcom.sun.management.jmxremote.ssl=FALSE -Duser.language=de -Duser.region=DE -Duser.country=DE -Djboss.vfs.cache.TimedPolicyCaching.lifetime=1440000 -DVFjavaWL=er.core.de
Update
A little more analysis. Here is my naive interpretation of what happens with xml parsing:
- We have a singleton
JAXBHelper class with JAXBContext level JAXBContext (this is a thread safety object). - Each time a request arrives, we try to associate an incoming xml string with a Java object (previously generated JAXB from xsd).
- The bind method creates unmarshaller at the instance level (they are not thread safe, but cheap to create).
- The binding method then calls the unmarshaller
unmarshal(InputStream) method. - Now
javax.xml.bind.helpers.AbstractUnmarshallerImpl calls SAXParserFactory.newInstance , which in turn calls the class loader, which in turn requires Inflater .
Surely an implementation (we are using jaxb-ri 2.2.7) should save SAXParserFactory or SAXParser and reuse them? I note that they are not thread safe, so this can be done using ThreadLocal . Or could you reuse instances of Inflater ?
Does the fact that they are not reused mean that I am not dereferencing my objects? UnMarshaller does not seem to have any close method.