100% use of Java cpu server after two days of continuous work with approximately 110 users

Question

I have tomcat 6.0.20, apr 1.2, jdk 1.6.0_15 with mysql 5.1.38 working on a rhel box with 4 GB of RAM. It has one jsp / servlet application with 5 users, one struts 1.2.0.9 with 64 users on it and one struts 2.0 application with 35 users on it. Users of struts 2.0 record every second, about 900 records per day. I also use toplink to save in the last two applications. I announced that all objects that do not have a null reference in the code applied production values ​​for the configuration files from struts 2 and tomcat. Applied thread caching in mysql, shortening wait_timeout and interactive_timeout is equivalent to tomcat session timeout. Enlarged file descriptors in Linux. Recycled requests. I studied the stream dump, looked at the gc statistics, the changes applied above based on this,

YET is still encountering the java.lang.OutOfMemoryError error.

at different times it is for different things, sometimes its Servlet.service (), sometimes its image.servlet, sometimes it is jasper that calls it.

extremely frustrating, as errors are not constant, but continue to change over time

Any help please would be greatly appreciated !!!

JAVA_OPTS = -server -XX: + UseConcMarkSweepGC -XX: + CMSClassUnloadingEnabled -XX: + CMSPermGenSweepingEnabled -XX: + CMSParallelRemarkEnabled (tomcat manager reports 34 mb empty, so mx didn’t use perms

persistence.xml

<property name="toplink.jdbc.url" value="jdbc:mysql://localhost:3306/dbname?autoReconnect=false"/> 

server.xml

 <Connector port="80" protocol="HTTP/1.1" connectionTimeout="2000" redirectPort="8443" compression="on" compressableMimeType="application/octet-stream,text/html,text/xml,text/plain,application/x-javascript,image/gif,text/css,image/gif,application/vnd.ms-excel,application/pdf" enableLookups="false"/> 

context.xml

 <Context reloadable="false" delegate="false" privileged="false"> 

Answer 1

First, you must make sure that some process uses the entire processor. Is this really a Java program? Try top(1) to figure it out if you haven't already.

If you have one, and you are sure that it is a Java program, enable remote debugging . When the processor boils next time, connect and make sure that the thread is not in an infinite loop.

If this is not the case, you have lost memory (no matter what the tomcat manager says). Start creating a JMX console and check out the various memory spaces. I assume that the perm is pretty full (is there a lot of big JARs in your class path? Or are you using cglib somewhere?) When this happens (and since you have perm gen GC enabled), the Java VM will try to free memory in perm gen space all time.

If you do not use something that creates class files at runtime (for example, a scripting language such as Groovy), this will not work: in normal Java programs, classes can never be GC'd. If you still do this, the GC will start up and start up and achieve nothing but burn all the processor power.

Answer 2

Perhaps this is the same problem referenced by 1 and 2 . Try to create only small transactions.

Answer 3

I would recommend using jstat to monitor tomcat memory and garbage collection patterns.

You say that you encounter primary errors, but you did not increase the perm gen count. Rebooting tomcat applications may cause perm gen to leak, but you probably watch your use before you open all of your pages, resulting in more classes being loaded in perm gen.

Answer 4

Your problem is that you have something like this in your code:

  new Thread(){ public void run(){ //something } }.start(); 

which, in turn, creates each time a new class definition file in the Tomcat cache - and that eats space, so you have a PermGenSpace exception ...

rewrite some of your critical classes so as not to create custom classes on the fly, make separate inner classes and things will change ...

  class MyJobDoer extends Thread{ public void run(){ //something } } MyJobDoer mjd=new MyJobDoer(); mjd.start();