Geek Answers Handbook

Tomcat JDBC pool connections are all stuck in Socket Read and no new connection is created

I have a Tomcat 7 application hosted on Amazon ElasticBeanstalk and a MySQL 5.5 database hosted on Amazon RDS. The database server serves only one Tomcat application and has a maximum connection limit of 10,000.

Strange things happen to database connections after a few hours of operation.

The MySQL server reports that only 3 connections were created by the Tomcat JDBC connection pool, all of them were "sleeping" (sample output):

| 228 | root | ip-10-240-xx-xxx.ap-southeast-2.compute.internal:33270 | xxxxx | Sleep | 13 | | NULL |

The Tomcat thread dump assumes that all 3 connections read (block) from IO network sockets:

 at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114) at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161) at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189) - locked <0x00000000bc349cc0> (a com.mysql.jdbc.util.ReadAheadInputStream) at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3036)

Since there are only 3 DB connections in the connection pool, and they are all stuck in IO reading, Tomcat refuses to serve more HTTP requests that require database data.

The Tomcat JDBC pool settings are assigned:

 maxActive="500" maxIdle="100" minIdle="50" initialSize="50" maxWait="15000" timeBetweenEvictionRunsMillis="10000" minEvictableIdleTimeMillis="30000" removeAbandoned="true" removeAbandonedTimeout="120000" logAbandoned="true" testOnBorrow="true" testWhileIdle="true" validationQuery="select 1" driverClassName="com.mysql.jdbc.Driver" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"

Still not sure why all 3 database connections are stuck in the IO read network, however, given the configuration above, I would expect Tomcat to create additional connections when all existing connections are busy.

I have 8 other Beanstalk applications using the same setup, however only this particular application has such strange behavior in pools and a problem with the IO network.

What are you offering?

Thank you very much.

+4
source share
1 answer

As stated in http://bugs.mysql.com/bug.php?id=31353 , a problem with unsafe threads may occur.

+1
source

More articles:


All Articles