Java / Groovy - memory leak in GroovyClassLoader

Question

I download a large number of Groovy scripts (2.4.6) and run them using GroovyScriptEngineImpl in my Java 8, and after a while I have a problem.

There are a few things you need to know:

• I need to recreate a new GroovyScriptEngineImpl every time I run a script
• I have to recreate a new GroovyClassLoader every time I run the script

I need to do this in order to isolate each script in a separate "environment": I load some external JAR files into the class loader for some scripts, and I do not want other scripts to be able to use classes in these JARs when they are executed.

My problem is that for each script that I run, GroovyClassLoader will create a new ScriptXXXX class and load it, but never unload it.

As a result, the number of loaded classes increases indefinitely, and the memory ends up completely full.

I tried a huge number of different solutions, but no one works:

• Adding -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC to JVM Arguments
• Adding -Dgroovy.use.classvalue=true to JVM Arguments
• Removing the "metaclasses" created for each ScriptXXXX class, as shown below: Groovy Classes are not collected, but have no evidence of a memory leak
• Clearing the cache and closing GroovyClassLoader
• Using introspection to manually delete some fields that cache classes in GroovyScriptEngineImpl
• etc...

Here's the “Shortest Path to the GC” for one of the ScriptXXXX classes in Eclipse Memory Analyzer:

I clearly don't have enough solutions here, and none of them work, as the class loader always refers to classes that never get GCed.

If you want to reproduce the problem, here is a sample code:

 GroovyScriptEngineImpl se; while (true) { se = new GroovyScriptEngineImpl(new GroovyClassLoader()); CompiledScript script = se.compile("println(\"hello\")"); script.eval(se.createBindings()); } 

thanks

UPDATE . After reading pczeus answer, I tried to restrict metaspace, and some classes seem to really unload, and I think these are ScriptXXX classes.

However, after a few minutes I get Out of Metaspace during script execution.

Here is the profile that I get with VisualVM:

And the "Path to the GC" in the Eclipse Memory Analyzer for ScriptXXX classes ScriptXXX really empty (they are no longer an instance of classes), even if the class is still listed in the histogram.

Answer 1

This is definitely similar to the edge case, which requires special configuration and, possibly, the choice of the garbage collection scheme used.

Since you are using Java8, and know that you are loading thousands of time classes, you should try to configure and limit the number of MetaSpace available and adjust the frequency of cleaning. Straight from https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent :

JDK8: Metaspace

In JDK 8, class metadata is now stored in the native heap, and this space is called Metaspace. New flags added for Metaspace in JDK 8:

-XX: MetaspaceSize = where is the initial amount of space (initial height-water) allocated for class metadata (in bytes) that can cause garbage collection to offload classes. The amount is approximate. After the high water mark is first reached, the next high humidity mark is controlled by the garbage collector

-XX: MaxMetaspaceSize = where is the maximum amount of space for class metadata (in bytes). This flag can be used to limit the amount of space allocated for class metadata. This value is approximate. There is no limit by default. -XX: MinMetaspaceFreeRatio = where the minimum percentage of the class metadata is free after GC to avoid the increase in the amount of space (high water marks) allocated for class metadata that will cause garbage collection.

-XX: MaxMetaspaceFreeRatio = where the maximum percentage of the class metadata is free after GC to avoid reducing the amount of space (high water marks) allocated for class metadata that will cause garbage collection.

By default, the distribution of class metadata is limited by the amount of available internal memory. We can use the new MaxMetaspaceSize option to limit the amount of internal memory used for class metadata. This is similar to MaxPermSize. Garbage collection causes the collection of dead class and class loaders when the class metadata reaches MetaspaceSize (12 MB on a 32-bit client VM and 16 MB on a 32-bit server VM with large sizes on 64-bit virtual machines). Set MetaspaceSize to a higher value to delay the collection of induced garbage. After the induced garbage collection, the use of the class metadata needed to create the next garbage collection can be increased.