Unicorn restarts by itself - No memory - dies

Question

I am running two Rails applications on DigitalOcean with 512 MB of RAM and with 4 nginx processes.

Rails applications use Unicorn.

One has 2 workers and the other 1.

My problem is related to the second application, in which 1 unicorn person works (the same problem was where there were 2 employees). What is happening, suddenly my application throws an error of 500. When I connect SSH to the server, I find that the application unicorn process is not working!

When I start the unicorn again, everything will be fine.

This is my log file. As you can see, the worker gets reaping, and then he cannot fork it, and the reason is that it has no memory.

, [2014-01-24T04:12:28.080716 #8820] INFO -- : master process ready I, [2014-01-24T04:12:28.110834 #8824] INFO -- : worker=0 ready E, [2014-01-24T06:45:08.423082 #8820] ERROR -- : reaped #<Process::Status: pid 8824 SIGKILL (signal 9)> worker=0 E, [2014-01-24T06:45:08.438352 #8820] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM) /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `fork' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `spawn_missing_workers' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:538:in `maintain_worker_count' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:303:in `join' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>' I, [2014-01-24T08:43:53.693228 #26868] INFO -- : Refreshing Gem list I, [2014-01-24T08:43:56.283950 #26868] INFO -- : unlinking existing socket=/tmp/unicorn.hmd.sock I, [2014-01-24T08:43:56.284840 #26868] INFO -- : listening on addr=/tmp/unicorn.hmd.sock fd=11 I, [2014-01-24T08:43:56.320075 #26868] INFO -- : master process ready I, [2014-01-24T08:43:56.348648 #26872] INFO -- : worker=0 ready E, [2014-01-24T09:10:07.251846 #26868] ERROR -- : reaped #<Process::Status: pid 26872 SIGKILL (signal 9)> worker=0 I, [2014-01-24T09:10:07.300339 #27743] INFO -- : worker=0 ready I, [2014-01-24T09:18:09.992675 #28039] INFO -- : executing ["/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn", "-D", "-c", "/home/vocp/projects/hmd/config/unicorn.rb", "-E", "production", {11=>#<Kgio::UNIXServer:/tmp/unicorn.hmd.sock>}] (in /home/vocp/projects/hmd) I, [2014-01-24T09:18:10.426852 #28039] INFO -- : inherited addr=/tmp/unicorn.hmd.sock fd=11 I, [2014-01-24T09:18:10.427090 #28039] INFO -- : Refreshing Gem list E, [2014-01-24T09:18:13.456986 #28039] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM) /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `fork' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `spawn_missing_workers' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:153:in `start' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load' /home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>' E, [2014-01-24T09:18:13.464982 #26868] ERROR -- : reaped #<Process::Status: pid 28039 exit 1> exec()-ed 

This is my unicorn.rb

 root = "/home/vocp/projects/hmd" working_directory root pid "#{root}/tmp/pids/unicorn.pid" stderr_path "#{root}/log/unicorn.log" stdout_path "#{root}/log/unicorn.log" listen "/tmp/unicorn.hmd.sock" worker_processes 1 timeout 30 preload_app true # Force the bundler gemfile environment variable to # reference the capistrano "current" symlink before_exec do |_| ENV["BUNDLE_GEMFILE"] = File.join(root, 'Gemfile') end before_fork do |server, worker| defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect! old_pid = Rails.root + '/tmp/pids/unicorn.pid.oldbin' if File.exists?(old_pid) && server.pid != old_pid begin Process.kill("QUIT", File.read(old_pid).to_i) rescue Errno::ENOENT, Errno::ESRCH puts "Old master alerady dead" end end end after_fork do |server, worker| defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection child_pid = server.config[:pid].sub('.pid', ".#{worker.nr}.pid") system("echo #{Process.pid} > #{child_pid}") end 

I have monit or god or any monitoring tools. I find this very strange because the server’s commonly used memory will be 380/490. And no one uses these two apps separately from me! They are under development.

Am I misconfigured something? Why is this happening? please help. Should I configure God to restart the unicorn when it crashes?

Answer 1

Unicorn is unfortunately the only way to use memory. The unicorn will allocate more memory if you need a rails app. But he does not let him out, even if he is no longer needed. For example, if you load many records for the index page at once, the unicorn will increase memory usage. Now this is compounded by the fact that 512 MB is not a huge amount of memory for 2-rail applications with 3 workers.

In addition, there are memory leaks that also increase memory usage. See this article https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app

At the end of the article, they refer to the unicorn-worker-killer gem to restart unicorn workers based on either maximum connections or maximum memory, which looks pretty simple.

Personally, I used bluepill to monitor individual unicorn processes and, if necessary, restarted them.

In your case, I will monitor all unicorn processes and restart them if they reach a certain memory size.

Answer 2

First check the memory using the "df -H" command on your server. If the memory is ok, reboot the sudo su reboot system and it will work fine.