Dag-scheduler-event-loop java.lang.OutOfMemoryError: unable to create new native thread

After starting for 5-6 hours, I get the following error from the intrinsic safety program. I am using Ubuntu 16.04 LTS and open-jdk-8.

Exception in thread "ForkJoinPool-50-worker-11" Exception in thread "dag-scheduler-event-loop" Exception in thread "ForkJoinPool-50-worker-13" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672) at scala.concurrent.forkjoin.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1795) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:117) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672) at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.push(ForkJoinPool.java:1072) at scala.concurrent.forkjoin.ForkJoinTask.fork(ForkJoinTask.java:654) at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.start(Tasks.scala:377) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.start(Tasks.scala:443) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$$anonfun$spawnSubtasks$1.apply(Tasks.scala:189) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$$anonfun$spawnSubtasks$1.apply(Tasks.scala:186) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.spawnSubtasks(Tasks.scala:186) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.spawnSubtasks(Tasks.scala:443) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.internal(Tasks.scala:157) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:443) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:149) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443) at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341) at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673) at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443) at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426) at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56) at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:958) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51) at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:953) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443) at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672) at scala.concurrent.forkjoin.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1795) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:117) 

This is a bug created by Spark Driver, which by default runs in client mode, so some people simply increase the heap size by passing the --driver-memory 3g flag or something similar, but the message "unable to create new native thread" does say that the JVM asks the OS to create a new thread, but the OS can no longer allocate it, and the number of threads that the JVM can create using the OS request depends on the platform, but usually it is 32K threads in a 64-bit OS and JVM.

when i did ulimit -a i get the following

 core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 120242 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 120242 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

cat / proc / sys / kernel / pid_max

 32768 

cat / proc / sys / kernel / threads-max

 240484 

“Unable to create a new own thread” clearly means that it has nothing to do with the heap. so I think this is more of an OS problem.

+7
java apache-spark
source share
3 answers

It seems that a bug in using ForkJoinPool in Spark 2.0.0 creates too many threads. In particular, in UnionRDD.scala, which is used when you invoke a window operation in Dstream.

https://issues.apache.org/jira/browse/SPARK-17396 , so according to this ticket I upgraded to 2.0.1 and it fixed the problem.

+5
source share

Root cause analysis:

The threads in the JVM need some ulnar room in order to do the work they called for a solution. When there are more threads than there is room in the memory, we created the foundation for the problem: enter image description here The message java.lang.OutOfMemoryError: Unable to create new native thread means that the Java application has reached the limit of how many threads it can start .

What causes it?

You have the opportunity to encounter java.lang.OutOfMemoryError: Unable to create new native thread when the JVM requests a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryErro r will be selected. The exact limit for native threads is very platform dependent, so we recommend that you find out these limitations by running a test similar to the example below. But, in general, the situation causing java.lang.OutOfMemoryError: Unable to create a new native thread, goes through the following steps:

  • A new Java thread is requested by an application running inside the JVM
  • JVM native code proxies a request to create a new native thread in the OS
  • The OS is trying to create a new native thread that requires memory allocation for the thread
  • The OS will refuse to allocate its own memory either because the 32-bit size of the Java package has exhausted its memory address space - for example, (2-4) the GB process size limit has been deleted - or the OS’s virtual memory has been exhausted.
  • java.lang.OutOfMemoryError: Failed to create a new native thread error.

Give an example

The following example creates and starts new threads in a loop. When you run the code, the operating system limits are reached quickly and the message java.lang.OutOfMemoryError: Unable to create new native thread displayed.

 while(true){ new Thread(new Runnable(){ public void run() { try { Thread.sleep(10000000); } catch(InterruptedException e) { } } }).start(); } 

The exact restriction on the native thread depends on the platform, for example, tests on Windows, Linux and Mac OS X show that:

  • Mac OS X 10.9 64-bit, Java 1.7.0_45 - JVM dies after thread creation # 2031
  • 64-bit Ubuntu Linux, Java 1.7.0_45 - JVM dies after # 31893 threads were created
  • 64-bit version of Windows 7, Java 1.7.0_45 - due to the different threading model used by the OS, this error does not seem to be thrown to this particular platform. At thread # 250,000, the process was still alive, although the swap file had grown to 10 GB, and the application was faced with extreme performance issues. Therefore, make sure you know your limitations by calling a small test and know when java.lang.OutOfMemoryError will be launched: Unable to create a new native thread.

What's the solution?

Sometimes you can bypass Unable to create a new problem with your own thread, increasing restrictions at the OS level. For example, if you are limited by the number of processes that the JVM can appear in user space, you should check and possibly increase the limit:

 [ root@dev ~]# ulimit -a core file size (blocks, -c) 0 --- cut for brevity --- max user processes (-u) 1800 

More often than not, restrictions on new native threads that fall into OutOfMemoryError indicate a programming error. When your application generates thousands of threads, the likelihood that something went horribly wrong, there are not many applications that will benefit from such a huge number of threads. One way to solve the problem is to start accepting dump streams to understand the situation.

Link to the resource:

https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

There are more other answers here:

0
source share

In Java, you can stumble upon two types of errors in memory:

  • java.lang.OutOfMemoryError Java heap space error : This exception will be thrown when the application tries to allocate more data to the heap space, but this is not enough room for it. Although there may be a lot of memory available on your machine, you have reached the maximum amount of memory allowed by your JVM, which can be set via the -Xmx option
  • java.lang.OutOfMemoryError: Unable to create new native thread occurs when the JVM requests a new thread from the OS. If the main OS cannot allocate a new native thread, this OutOfMemoryError will be selected.

1) Checking the system settings of the Thread system

The file / proc / sys / kernel / threads-max provides a system-wide limit on the number of threads. The root user can change this value if they want:

 $ echo 100000 > /proc/sys/kernel/threads-max 

You can check the current number of threads running through the / proc / loadavg file system:

 $ cat /proc/loadavg 0.41 0.45 0.57 3/749 28174 

See the fourth field! This field consists of two numbers separated by a slash (/). The first of these is the number of currently running kernel planning objects (processes, threads); this will be less than or equal to the number of processors. The value after the slash is the number of kernel planning objects that currently exist in the system. In this case, you start 749 threads

2) Check the number of processes per user

In a Linux box, threads are essentially just processes with a common address space. Therefore, you need to check if your OS allows enough processes for the user. This can be checked with:

 ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515005 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

The default for each user is 1024 by default. At this stage, we will calculate the number of running processes. The number of running processes can be calculated using the ps output:

 $ ps -elf | wc -l 220 

However, this number does not take into account threads that may be caused by the process. If you try to start ps using -T, you will also see all the threads:

 $ ps -elfT | wc -l 385 

As you can see, the number of processes has increased significantly due to threads. This is usually never a problem, but in Java based applications this can cause your system to work within system limits! We continue our investigation. See how many threads are created by your JBoss process. You can do this in at least two ways:

 $ ps -p JBOSSPID -lfT | wc -l 

The above shell will return the number of lightweight processes created for the process indicated by the PID. This should match the thread dump count generated by jstack:

 $ jstack -l JBOSSPID | grep tid | wc -l 

You should now have evidence or not, that you need to increase the number of processes for the user. This can be done using the following command:

 $ ulimit -u 4096 

3) Check PID flow limits

After you have counted the number of threads, you must make sure that you do not press the system limits set by the kernel.pid_max limit parameter. You can verify this value by doing:

 $ sysctl -a | grep kernel.pid_max kernel.pid_max = 32768 

4) Reduce the thread stack size

Another option that you can use if you cannot change the OS settings reduces the stack size. The JVM has an interesting implementation, thanks to which more memory is allocated for the heap (not necessarily used by the heap), the less memory is available on the stack, and since threads are made from the stack, in practice this means more “memory” in the sense of heap (which people usually say ) leads to the fact that fewer threads can work simultaneously.

First of all, check the default stack size, depending on your operating system:

 $ java -XX:+PrintFlagsFinal -version | grep ThreadStackSize intx ThreadStackSize = 1024 {pd product} 

As you can see, the default stack stack size is 1024 kb on our computer. To reduce the size of the stack, add the -Xss option to the JVM options. In JBoss EAP 6 / WildFly, the minimum stack stack size is 228kb. You can change it offline by changing JAVA_OPTS, as in the following example:

 JAVA_OPTS="-Xms128m -Xmx1303m -Xss256k" 

In domain mode, you can configure the jvm element at different levels (Host, Server Group, Server). There you can set the required stack size, as in the following section:

 <jvm name="default"> <heap size="64m" max-size="256m"/> <jvm-options> <option value="-server"/> <option value="-Xss256k"/> </jvm-options> </jvm> 

Link to the resource:

How to solve java.lang.OutOfMemoryError: unable to create new native thread

0
source share

All Articles