Puma stops Rails application launch for EC2 instance with Nginx (using Capistrano / Capistrano Puma)

Question

My question is at the top level: how can I make Puma stop failing. But it really consists of many small questions. I will celebrate and boldly each of them in order to try to answer this question.

I host a Rails application on an EC2 instance which is t2.nano. This is admittedly a very small box, but I do not expect my site to receive any kind of traffic. I have successfully constructed everything with Nginx and Puma using Capistrano and Capistrano Puma. Everything was great until one day I went to my site and saw the Nginx 504 message.

I opened the Nginx error log and saw that it could not connect to Puma:

connect() to unix:/home/deploy/myapp/shared/tmp/sockets/puma.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost, request: "GET / HTTP/1.0", upstream: "http://unix:/home/deploy/myapp/shared/tmp/sockets/puma.sock:/500.html", host: "myapp.com"

Having debugged this, I found out that Puma had stopped working. This is why Nginx was unable to connect to it. I think that there are two problems: firstly, the Puma should not stop. The server is tiny, but there is no traffic. secondly, that when Puma does not work, it should restart gracefully. However, right now I'm just focusing on the first question. Because if Puma constantly restarts, it seems reasonable that sometimes it kills the process in a harsh way.

To debug this, I opened htop. Of course, the machine worked without unnecessary memory. It makes sense - I am running a database, rails, webserver and memcache application on one tiny machine. He continues to run out of memory and kill Puma.

I looked at the Puma configuration I installed with Capistrano. In config / deploy.rb I had these lines -

 set :puma_threads, [0, 8] set :puma_workers, 0 

I read all about puma_workers and puma_threads. I also found out that Nginx has its own workers. Puma processes are very expensive. What makes Puma cool is that it is properly mule-threaded, so the independent processes are awesome. It seems that each worker has his own set of threads, so if there are 4 employees with 8 threads, there will be 32 processes. But in my case, I want to use very little memory. 2 processes sound good to me. <B> 1. Is my understanding of workers and threads correct?

I updated the config / deploy.rb file and deployed it with 0 puma_workers and min = 0, max = 2 threads.

The configuration for Nginx seems to live here: /etc/nginx/nginx.conf. And the configuration for Puma lives here: /home/deploy/myapp/shared/puma.rb. I would expect my updates in config / deploy.rb to force Capistano to edit the configuration files. No luck - my minimum threads were still set to 0.8. 2. Is it correct to try updating these values ​​through config / deploy.rb when using Capistano?

Also - I opened nginx.conf and saw worker_processes 4; . 3. Was it installed four times when I installed Nginx or did Capistano by default?

I opened htop and, of course, I had many Puma processes. So I manually edited my configuration files and restarted Puma and Nginx.

I changed the number of Nginx workers from 4 to 1. Looking at htop, this worked. I now have only 1 working Nginx. However, Nginx workers have never been very expensive (compared to Puma threads). Therefore, I do not think this is of great importance.

However, there were even more than two Puma threads - there were 6. In the lark I changed the minimum number of threads from 0 to 1 - thinking 0 is not a possible number, so maybe he set the default value, This increased the number of Puma processes to 9 I also tried changing the number of puma_workers to 1 for the same reason, and the number of processes increased. 4. What does it mean to have 0 threads and / or workers?

Then I tried to kill one of the puma processes manually (sudo kill xxxxx), and then all the Puma processes died.

5. What do I need to do to have only 2 puma processes?

As you can see, my understanding of Puma is not very great, and the lines between Puma vs Nginx vs Capistano are not clear. Any help is appreciated. I could not find great resources on this issue.

Answer 1

This is what I found out -

• If Puma stops working, make sure you have enough memory to handle the number of workers and threads that you specified. Each Puma process is quite expensive.
• If you set 0 workers, Puma will not work in cluster mode. It is recommended that you start MRI using cluster mode.
• flows are set for each cluster. if you have 2 jobs and 0.8 threads, this means that you will have two jobs, each of which will have from 1 to 8 threads.
• Puma uses processes in addition to threads. Puma has a PID for the parent process. if you use cluster mode, it has PID for cluster management. if you use cluster mode, it also has a PID for each cluster. then there is a fixed number of PIDs for performing other tasks (for each cluster). without cluster mode, there are 5 fixed PIDs. with cluster mode, there are 7 fixed PIDs.
• That's all you can say - if you see more processes than you expect, that's why. also - when you add a new employee, you add a significant amount of expensive processes. make sure you have a place. I have a small application and it seems to work well with 1 worker and min = 1, max = 4 threads. having a maximum of 8 threads, this seems to be what continued to kill cougar for me.

To answer my original questions -

• Yes, the correct explanation of workers and flows.
• capistrano-puma apparently only installs the puma configuration with the first deployment.
• I think nginx config is created when nginx is installed.
• 0 workers means you are using puma without cluster mode. It is impossible to have 0 threads. I find that 0.8 is the same as 1.8.
• Puma needs to run processes in addition to the requests you request. It is not possible for puma to work with only 2 or 3 PIDs. These processes perform additional tasks.

Answer 2

Suspect for Puma Hanging

The thing with Puma is that this is the only core project that encourages the use of threads in MRI Ruby (well, anyway, recommends Heroku).

That's why we sometimes see statements from people working on Puma about how people think that Puma has various problems, while the problem is elsewhere, and it is, and it affects only Puma: P

"We discovered and documented some very bizarre and unpleasant problems with Ruby GC in the past when heavily using threads in Ruby MRI with some bizarre corner cases (remember http://blog.skylight.io/hunting-for-leaks-in-ruby/ ), and who will say that this is not the last of such bizarre problems that people attribute to Puma?

Try turning the threads off for a while, see how this happens, and let us know, maybe the rabbit is there, again

Documents explaining flows against cluster mode and workers

• Thread pool docs: https://github.com/puma/puma#thread-pool
• Cluster docs mode: https://github.com/puma/puma#clustered-mode
• puma.rb options: https://github.com/puma/puma/blob/master/examples/config.rb

In the Thread pool section, the documents explain how to configure the number of worker threads. Remember that Puma is primarily a JRuby thing, and MRI and forking support were added only later, as an afterthought, because of this, streamlining configuration entries in documents (how to configure streams before setting up forking).

Status of documents:

Puma uses a dynamic thread pool that you can change. You can set the minimum and maximum number of threads available in the pool with the -t (or -threads) flag:

Puma 2 offers cluster mode, which allows you to use forked processes to process multiple incoming requests simultaneously in addition to the threads already provided.

Meaning, Puma will always be a thread, this is what it does, if you tell it to make 0/1 thread, it will make 1 thread so that it can serve requests.

In addition, if you set the number of workers (processes) to> 1, Puma will start in "Clustered mode", which means that it will be a fork, and each fork will be a thread,

i.e. -w 3 -t4:4 will cause three processes to process 4 threads each, which will allow you to simultaneously request server requests.

Puma documents do not indicate which and how many processes Puma will use internally for it, but only the educated guess is that, at a minimum, it needs to start all workers + 1 master process to manage them, deliver data to them, start them, stop, direct their magazines, etc.