"java.lang.OutOfMemoryError: unable to create new native thread"

Question

We get "java.lang.OutOfMemoryError : unable to create new native Thread " on 8 GB of RAM RAM after 32k threads (ps -eLF | grep -c java)

However, "top" and "free -m" shows 50% free memory available . JDk has a 64-bit version and works with both HotSpot and JRockit.Server with Linux 2.6.18

We also tried OS stack size (ulimit -s) settings and max process (ulimit -u), limit.conf, but all in vain.

We also tried almost all possible combinations of heap sizes, keeping it low, high, etc.

script that we use to run the application,

 /opt/jrockit-jdk1.6/bin/java -Xms512m -Xmx512m -Xss128k -jar JavaNatSimulator.jar /opt/tools/jnatclients/natSimulator.properties 

Thanks for the answer.

We tried to edit /etc/security/limits.conf and ulimit, but still the same

 [root@jboss02 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 72192 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 72192 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

Answer 1

This is not a memory problem, although the exception name strongly recommends it, but a problem with the operating system resources. You are running out of threads of your own, i.e. how many threads the operating system will allow your JVM to use.

This is an unusual problem because you rarely need so much. Do you have many unconditional threads where threads should, but don't end?

You may consider rewriting in Callable / Runnables under the control of the artist, if at all possible. There are many standard artists with different behaviors that your code can easily manage.

(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)

Answer 2

Your OS probably does not allow the number of threads you are trying to create, or you push some limit in the JVM. Especially if such a round number equal to 32k, the limit of one kind or another is a very likely culprit.

Do you really need 32k themes? Most modern languages ​​have some support for reusable thread pools - I'm sure Java has something in place (e.g. ExecutorService , as Jesper mentioned). Perhaps you could request threads from such a pool rather than manually create new ones.

Answer 3

I encountered the same problem during a load test, the reason is that the JVM cannot continue to create a new Java thread. Below is the JVM source code

 if (native_thread->osthread() == NULL) { // No one should hold a reference to the 'native_thread'. delete native_thread; if (JvmtiExport::should_post_resource_exhausted()) { JvmtiExport::post_resource_exhausted( JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS, "unable to create new native thread"); } THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), "unable to create new native thread"); } Thread::start(native_thread);' 

Main reason: The JVM throws this exception when JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR (resources are exhausted (means that memory is exhausted)) or JVMTI_RESOURCE_EXHAUSTED_THREADS (threads are exhausted).

In my case, Jboss creates too many threads to serve the request, but all threads are blocked. Because of this, the JVM is exhausted by threads as well as memory (each thread contains memory that is not freed because each thread is blocked).

It was analyzed that Java thread dumps found that almost 61K threads were blocked by one of our methods, which caused this problem. The following is part of the theme dump

 "SimpleAsyncTaskExecutor-16562" #38070 prio=5 os_prio=0 tid=0x00007f9985440000 nid=0x2ca6 waiting for monitor entry [0x00007f9d58c2d000] java.lang.Thread.State: BLOCKED (on object monitor) 

Answer 4

I would also recommend looking at the size of the stack and see if you have more threads created. The default stack size for JRockit 1.5 / 1.6 is 1 MB for a 64-bit Linux virtual machine. To fulfill this requirement, 32K threads will require a significant amount of physical and virtual memory.

Try reducing the stack size to 512KB as a starting point and see if it helps create more threads for your application. I also recommend exploring horizontal scaling, for example. sharing application processing on more physical or virtual machines.

When using a 64-bit virtual machine, the true limit will depend on the availability of physical and virtual OS memory and OS settings such as ulimitc. I also recommend the following article as a reference:

OutOfMemoryError: Unable to Create New Own Stream - Demystified Problem

Answer 5

I had the same problem due to ghost processes that didn't show up when using top in bash. This prevented the JVM from appearing more threads.

For me, this was resolved when listing all java processes using jps (just execute jps in your shell) and killed them separately using the kill -9 pid bash command for each ghost process.

This may help in some scenarios.

Answer 6

If your job fails due to OutOfMemmory on nodes, you can tweek the number of maximum cards and reducers, and choose a JVM for each. mapred.child.java.opts (the default is 200Xmx) should usually be increased based on your specific data nodes.

This link may be useful ... pls check

Answer 7

your JBoss configuration has some problems, / opt / jrockit -jdk1.6 / bin / java -Xms512m -Xmx512m Xms and Xmx limit the use of your JBoss memory to a given value, so from 8Gb you have a server that uses only 512M + for some other goals, increasing this number, do not forget to leave some for free for the OS and other things work there, and you can run it, despite the unpleasant code. Correcting the code would also be nice if possible.

Answer 8

You have the opportunity to encounter java.lang.OutOfMemoryError: Unable to create new native thread when the JVM requests a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be selected. The exact limit for native threads is very platform dependent, so it’s recommended to find out these limitations by running a test similar to the link example below. But, in general, the situation causing java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:

• A new Java thread is requested by an application running inside the JVM
• JVM native code proxies a request to create a new native thread in the OS OS is trying to create a new native thread, which requires memory allocation for the thread
• The OS will refuse native memory allocation either because the 32-bit size of the Java process has depleted the memory address space - for example, (2-4) the GB process size limit has been deleted - or the OS’s virtual memory has been completely exhausted
• java.lang.OutOfMemoryError: Unable to create a new native thread error.

Link: https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

Answer 9

I had the same problem and it turned out that this is a misuse of the java API. I initialized the builder in a batch processing method that should not be initialized more than once.

Basically I was doing something like:

 for (batch in batches) { process_batch(batch) } def process_batch(batch) { var client = TransportClient.builder().build() client.processList(batch) } 

when i had to do this:

 for (batch in batches) { var client = TransportClient.builder().build() process_batch(batch, client) } def process_batch(batch, client) { client.processList(batch) } 

Answer 10

To find out which processes create threads, try:

 ps huH 

I usually redirect the output to a file and analyze the file offline (the number of threads for each process matches the expected or not)

Answer 11

This error may appear due to the following two reasons:

• There is no room in memory to post new topics.

• The number of threads exceeds the operating system limit.

I doubt the number of threads exceeded the limit for the Java process

So the problem is probably due to memory

threads are not created on the JVM heap. They are created outside the JVM heap. Therefore, if there is less space left in RAM, after allocating the JVM heap, the application will encounter "java.lang.OutOfMemoryError: Failed to create a new native thread".

A possible solution is to reduce the heap memory or increase the total RAM size.

Answer 12

First of all, I would not blame OS OS / VM on it .. rather, the developer who wrote the code that creates sooo many Threads . Basically, somewhere in your code (or third-party) a lot of threads are created without control .

Take a close look at stacktraces / code and control the number of threads created. Typically, your application does not require a large number of threads if this creates another problem.