Why does the use of the JVM Heap Usage Max reported by the JMX change over time?

Question

My JVM heap max is set to 8 GB for the node name for one of my haop clusters. When I control the JVM using the JMX, the indicated maximum signal is constantly changing, as shown in the attached image.

http://highlycaffeinated.com/assets/images/heapmax.png

I see this behavior on only one (the most active) of my how-to clusters. On other clusters, the maximum message level is fixed at a given value. Any ideas why the maximum message size has changed?

Update:

The java version is "1.6.0_20"

The maximum heap value is specified in the hadoop-env.sh file with the following line:

export HADOOP_NAMENODE_OPTS="-Xmx8G -Dcom.sun.management.jmxremote.port=8004 $JMX_SHARED_PROPS"

ps shows:

hadoop 27605 1 99 Jul30 ? 11-07:23:13 /usr/lib/jvm/jre/bin/java -Xmx1000m -Xmx8G

Update 2:

Added the -Xms8G switch to the start command line last night:

export HADOOP_NAMENODE_OPTS="-Xms8G -Xmx8G -Dcom.sun.management.jmxremote.port=8004 $JMX_SHARED_PROPS"

As shown in the image below, the maximum value is still changing, although the pattern seems to have changed.

http://highlycaffeinated.com/assets/images/heapmax2.png

Update 3:

Here's a new graph that also shows Non-Heap max that remains constant:

http://highlycaffeinated.com/assets/images/heapmax3.png

Answer 1

According to MemoryMXBean documentation, memory usage is reported in two categories: heap and non-heap memory. The description of the category "Not a bunch" says:

The Java virtual machine manages memory other than heap (called memory without heap). The Java virtual machine has a method scope that is common to all threads. The scope of the method does not belong to cumulus memory. It stores the structures of each class, such as a constant runtime pool, field and method data, and code for methods and constructors. It is created when the Java virtual machine starts.

The method scope is logically part of the heap, but the implementation of the Java virtual machine may not collect or garbage collect. Like a heap, a method area can have a fixed size or it can expand and contract. The memory for the method area does not have to be contiguous.

This description is very similar to the Permanent Generation (PermGen), which is really part of the heap and is calculated against the memory allocated using the -Xmx flag. I'm not sure why they decided to report this separately, as this is part of the heap.

I suspect that the fluctuations you see are the result of shrinking JVMs and the growth of continuous generation, which will result in the indicated maximum heap space available for non-PermGen usage changing accordingly. If you could get the sum of the heap and non-heap maxima, as reported by JMX, and this sum remains constant at the 8G limit, to test this hypothesis.

Answer 2

One possibility is that the space of surviving JVMs fluctuates at maximum size.

The maximum JVM size reported by JMX through the HeapMemoryUsage.max attribute is not the actual maximum heap size (i.e. set with -Xmx)

The reported value is the maximum heap size minus the maximum survivor size,

To get the maximum heap size, add two jmx attributes:

 java.lang:type=Memory/HeapMemoryUsage.max + java.lang:type=MemoryPool,name=Survivor Space/Usage.max 

(tested on oracle jdk 1.7.0_45)