Limiting the flow of single processes in Java on Red Hat Linux

Question

I had a problem on a test machine running Red Hat Linux (kernel version 2.4.21-37.ELsmp) using Java 1.6 (1.6.0_02 or 1.6.0_04). The problem is that after creating a certain number of threads in one thread group, the operating system does not want or cannot create more.

This seems to be typical of Java creating threads, as the C thread stream program was able to create about 1.5k threads. Also, this does not happen with the Java 1.4 JVM ... it can create threads over 1.4k, although they are obviously handled differently with respect to the OS.

In this case, the number of threads that it disconnects is only 29 threads. This can be verified with a simple Java program that simply creates threads until it receives an error message and then prints the number of threads it created. Mistake:

  java.lang.OutOfMemoryError: unable to create new native thread 

This does not seem to be affected by such things as the number of threads used by other processes or users, or the total amount of memory that the system uses at that time. JVM settings like Xms, Xmx, and Xss do not seem to change anything (which is expected, given that the problem is with creating threads of your own OS).

The output of "ulimit -a" is as follows:

 core file size (blocks, -c) 0
 data seg size (kbytes, -d) unlimited
 file size (blocks, -f) unlimited
 max locked memory (kbytes, -l) 4
 max memory size (kbytes, -m) unlimited
 open files (-n) 1024
 pipe size (512 bytes, -p) 8
 stack size (kbytes, -s) 10240
 cpu time (seconds, -t) unlimited
 max user processes (-u) 7168
 virtual memory (kbytes, -v) unlimited

User process limitation does not seem to be a problem. Finding information about what might be wrong is not much, but this post seems to indicate that at least some Red Hat cores limit the process to 300 MB of memory allocated for the stack, and 10 MB per thread for the stack seems to that the problem may be there (although it seems strange and unlikely).

I tried resizing the stack using "ulimit -s" to check this, but any value other than 10240 and the JVM does not start with an error:

  Error occurred during initialization of VM
 Cannot create VM thread.  Out of system resources. 

I can get around Linux at all, but I really know little about the system configuration, and I could not find anything specific to deal with this situation. Any ideas on which system settings or JVMs might trigger this would be appreciated.

Editing Running the thread restriction program mentioned by the plinth didn't crash until he tried to create thread 1529.

The problem also did not occur using 1.4 JVM (occurs with JVM 1.6.0_02 and 1.6.0_04, cannot test with 1.5 JVM at the moment).

The code for the stream test that I use is as follows:

public class ThreadTest { public static void main(String[] pArgs) throws Exception { try { // keep spawning new threads forever while (true) { new TestThread().start(); } } // when out of memory error is reached, print out the number of // successful threads spawned and exit catch ( OutOfMemoryError e ) { System.out.println(TestThread.CREATE_COUNT); System.exit(-1); } } static class TestThread extends Thread { private static int CREATE_COUNT = 0; public TestThread() { CREATE_COUNT++; } // make the thread wait for eternity after being spawned public void run() { try { sleep(Integer.MAX_VALUE); } // even if there is an interruption, dont do anything catch (InterruptedException e) { } } } } 

If you run this using the 1.4 JVM, it freezes when it cannot create more threads and requires kill -9 (at least for me).

More Edit:

It turns out that the system that has the problem uses the LinuxThreads streaming model, while the other system that works fine uses the NPTL model.

Answer 1

Updating the kernel to a newer version (2.6.something) using NPTL threading has been fixed.

Answer 2

Have you viewed this resource ? It says that you should be able to run a thread-limit to find the maximum number of threads and you can configure it by compiling glibc.

Answer 3

This is with Ubuntu Linux (1GB RAM)

 dsm@localhost:~$ javac ThreadTest.java dsm@localhost:~$ java ThreadTest 8113 dsm@localhost:~$ java -version java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06) Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing) dsm@localhost:~$ 

Answer 4

Can you try it with JRM JRockit? IIRC, he had a different thread model than the Sun JVM.

Answer 5

The settings /etc/security/limits.d/90-nproc.conf can override your settings /etc/security/limits.conf . This can cause the system to act differently, as shown in ulimit -u .

https://bugzilla.redhat.com/show_bug.cgi?id=823030