Geek Answers Handbook

How many threads should jenkins execute?

I have a Jenkins server that does not run out of memory and cannot create its own threads. I increased the amount of memory and installed Plugin Monitoring .

There are about 150 projects on the server, and I observe how the thread flow works all day. Now about 990. I expect that when it reaches 1024, which is the limit of user limits for threads, Jenkins will run out of memory again.

[edit]: I have hit 1016 threads and am now getting the out of memory error

Is this a suitable number of threads for Jenkins? How can I tell Jenkins to destroy threads when they are done with them?

+4
source share
2 answers

TL; DR:

A post-build action was taken with a bash script that returned nothing using stderr or stdout in Jenkins. Therefore, at each start of the assembly, threads will be created and wait ing stuck. I solved this problem by getting bash script exit status.

long answer

I run Jenkins on CentOS and install via RPM. Regarding modifying the Winstone servlet container, you can change this in the Jenkin init script in /etc/sysctrl/jenkins . However, the above parameters control only the number of HTTP streams created, and not the total number of streams.

This would be a solution if my threads would freeze when accessing the Jenkins HTTP API as part of the post-commit action. However, using the always convenient monitoring plugin mentioned in my question, I checked for stuck threads.

Threads are stuck on something in the com.trilead.ssh2.channel package. The getChannelData method has a while(true) that looks for output in the stderr or stdout of the ssh stream. The thread was getting sucked in this cycle because nothing was happening. I found out about it on GrepCode .

This was because the post-build action was to execute the command through SSH on the server and execute a bash script that will check the git repository. However, the git repository was not configured correctly and the git command was a mistake, but the exit 1 status did not break through the bash script (partly due to the incorrectly formulated if-elif-else statement).

the script is complete and the build is deemed successful, but somehow the thread processing the Jenkins SSH connection remained hanging due to this git error.

But thanks for your help on this!

+2
source

If you run Jenkins out of the box, it uses the Winstone servlet container. You can pass command line arguments to it as described here . Some of these options may limit the number of threads:

  --handlerCountStartup = set the no of worker threads to spawn at startup. Default is 5 --handlerCountMax = set the max no of worker threads to allow. Default is 300 --handlerCountMaxIdle = set the max no of idle worker threads to allow. Default is 50

Now I tried this some time ago and was not 100% sure that it worked, so there are no guarantees, but it's worth a try.

+2
source

More articles:


All Articles