High memory usage when using Hibernate

Question

I am coding a server side application with running java on a linux server. I use hibernate to open a session for the database, use my own sql for the query, and always close this session with try, catch, finally.

Server-side firewall using sleep mode at a very high frequency.

I already define MaxHeapSize for 3000M, but usually it uses 2.7 GB in RAM, it can decrease, but slower than increase. Once it grows to 3.6 GB of memory, more than my MaxHeapSize determines at startup.

When the memory used is 3.6 GB, I try to reset it with the -jmap command and get a 1.3 GB heapdump.

Im using mcl for eclipse to parse it, here is the dominant tree from MAT I think the hibernation problem is a problem, I have so many org.apache.commons.collections.map.AbstractReferenceMap $ ReferenceEntry like this. It may not be disposed of by garbage collection, or it may slow down.

How can i fix this?

Answer 1

You have 250k entries in your IN query list. Even your own query will bring the database to its knees. Oracle limits the list of IN queries to 1000 for performance reasons, so you should do the same.

Providing more RAM will not solve the problem, you need to limit the selection / update packages to no more than 1000 entries, using pagination.

Streaming is an option , but for such a large set of keygation pagination results .

If you can do all the processing in the database, you don’t have to move 250k records from the database to the application. There is a very good reason why many RDBMSs offer advanced procedural languages ​​(e.g. PL / SQL, T-SQL).

Answer 2

Please note that even if the number of objects in a PlanCache request can be configured and limited, this is probably not so much.

In our case, we wrote queries in hql like this:

hql = String.format("from Entity where msisdn='%s'", msisdn); 

This led to N different requests going to queryPlanCache. When we changed this request to:

 hql = "from Blacklist where msisnd = :msisdn"; ... query.setParameter("msisdn", msisdn); 

The queryPlanCache size has been significantly reduced from 100 MB to almost 0. This second request translates into one single prepared Stabilizer, resulting in only one object in the cache.

Answer 3

Thanks Vlad Mihalcea , referring to the Hibernate issue , this is a bug in sleep mode, version 3.6 has been fixed. I just upgrade hibernate version 3.3.2 to version 3.6.10, use the default value of "hibernate.query.plan_cache_max_soft_references" (2048), "hibernate.query.plan_cache_max_strong_references" (128), and my problem disappeared. Using higher memory will not.