How to manage outdated passenger workers that lead to downtime

Question

I have a rails application running on an ubuntu 14.04 machine and it is served by Nginx and passenger . There are times when an application shuts down, giving the following error:

 [ agents/LoggingAgent/Main.cpp:338 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown) [ agents/LoggingAgent/Main.cpp:400 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected... [ ServerKit/Server.h:453 ]: [LoggerAdminServer] Shutdown finished [ agents/LoggingAgent/Main.cpp:425 ]: PassengerAgent logger shutdown finished [ ServerKit/Server.h:453 ]: [ServerThr.1] Shutdown finished [ ServerKit/Server.h:453 ]: [ServerThr.2] Shutdown finished [ ServerKit/Server.h:453 ]: [AdminServer] Shutdown finished [ agents/HelperAgent/Main.cpp:724 ]: Disconnecting long-running connections for process 19431, application /public#default [ agents/HelperAgent/Main.cpp:724 ]: Disconnecting long-running connections for process 19440, application /public#default [ agents/Watchdog/Main.cpp:1255 ]: Starting PassengerAgent watchdog... [ agents/HelperAgent/Main.cpp:883 ]: Starting PassengerAgent server... [ agents/HelperAgent/Main.cpp:232 ]: PassengerAgent server running in multi-application mode. [ agents/HelperAgent/Main.cpp:637 ]: PassengerAgent server online, PID 19637 [ agents/LoggingAgent/Main.cpp:431 ]: Starting PassengerAgent logger... [ agents/LoggingAgent/Main.cpp:312 ]: PassengerAgent logger online, PID 19645 [ agents/HelperAgent/Main.cpp:868 ]: PassengerAgent server shutdown finished 

And the application remains on, unless Nginx restarts. This is probably due to outdated workers who no longer serve and eat memory (correct me if I am wrong).

I set the max pool size for the passenger to 6 , since my application instance accepts 150-200 Mb memory using 2GB RAM .

I was thinking of writing a script that regularly checks for obsolete workers, and if it starts restarting Nginx or doing touch tmp/restart.txt . I also want to know how to check outdated workers or if passenger is not working as expected.

Another solution that I thought was to restart Nginx regularly or do touch tmp/restart.txt at a regular interval of time, say, one hour.

I also read the documentation on three ways to restart the application

• passenger-config restart-app

• restart.txt

• Restart nginx

The first two cases will not drop the request when the application is restarted, but may have zero downtime when the application is restarted, since restarting the application may take some time depending on the application.

I want to know what would be a good way, or is there another way to get rid of this problem.

Answer 1

First, you really have to check the passenger configuration parameters . Secondly, you must really understand what your passenger worker really is. There is a passenger log parameter for this (which you can control), because it seems that you do not know what happens in these processes. :)

Now, regarding the first problem - your desire to force a restart of the workflow - if this happens due to the process freezing when it does nothing: you can combine

 passenger_min_instances passenger_max_instances 

from

 passenger_pool_idle_time / passenger_max_requests 

Read a little about these commands on this link, it will be in more detail.