Why didn't my heroku rails app restart itself?

Question

I run a rails app for my client and he has recently gone down. The place was about 9 hours before I noticed. I checked the logs and every request in the last 9 hours is added with the following code:

at=error code=H10 desc="App crashed" 

Before that, I see the following logs:

 2012-11-16T00:55:46+00:00 heroku[web.1]: Idling 2012-11-16T00:55:50+00:00 heroku[web.1]: Stopping all processes with SIGTERM 2012-11-16T00:55:51+00:00 app[web.1]: [2012-11-16 00:55:51] ERROR SignalException: SIGTERM 2012-11-16T00:55:51+00:00 app[web.1]: /usr/local/lib/ruby/1.9.1/webrick/server.rb:90:in `select' 2012-11-16T00:56:00+00:00 heroku[web.1]: Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM 2012-11-16T00:56:00+00:00 heroku[web.1]: Stopping remaining processes with SIGKILL 2012-11-16T00:56:02+00:00 heroku[web.1]: State changed from up to down 2012-11-16T00:56:02+00:00 heroku[web.1]: Process exited with status 137 2012-11-16T01:03:55+00:00 heroku[web.1]: Unidling 2012-11-16T01:03:55+00:00 heroku[web.1]: State changed from down to starting 2012-11-16T01:03:59+00:00 heroku[web.1]: Starting process with command `bundle exec rails server -p 4303` 2012-11-16T01:04:00+00:00 heroku[nginx]: 98.139.241.251 - - [16/Nov/2012:01:04:00 +0000] "GET / HTTP/1.1" 499 0 "-" "YahooCacheSystem" domain.com 2012-11-16T01:04:22+00:00 app[web.1]: => Ctrl-C to shutdown server 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Dispatcher: webrick 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Application: acsolar 2012-11-16T01:04:22+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:21 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : New Relic Ruby Agent 3.4.0.1 Initialized: pid = 2 2012-11-16T01:04:22+00:00 app[web.1]: => Booting WEBrick 2012-11-16T01:04:22+00:00 app[web.1]: => Rails 3.1.1 application starting in production on http://0.0.0.0:4303 2012-11-16T01:04:22+00:00 app[web.1]: => Call with -d to detach 2012-11-16T01:04:25+00:00 app[web.1]: [DEPRECATION] Your applications public directory contains an assets/products and/or assets/taxons subdirectory. 2012-11-16T01:04:25+00:00 app[web.1]: Run `rake spree:assets:relocate_images` to relocate the images. 2012-11-16T01:04:34+00:00 app[web.1]: ** [NewRelic][11/16/12 01:04:32 +0000 b8af98a1-2246-4b34-9dfe-61b9d4b747bc (2)] INFO : Reporting performance data every 60 seconds. 2012-11-16T01:04:34+00:00 app[web.1]: Connected to NewRelic Service at collector-5.newrelic.com 2012-11-16T01:05:00+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch 2012-11-16T01:05:00+00:00 heroku[web.1]: Stopping process with SIGKILL 2012-11-16T01:05:02+00:00 heroku[web.1]: Process exited with status 137 2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from crashed to down 2012-11-16T01:05:02+00:00 heroku[web.1]: State changed from starting to crashed 

I suppose it might have turned around and had a backup of the error, but why did it stay in a broken state without a reboot? Is there anything I can do so that it automatically restarts if this happens again in the future?

NewRelic also works for me, and it did not tell me about it, but this is another problem that I will have to investigate.

Answer 1

Heroku support answer suggests restarting your application manually using heroku restart . They are fixing the problem right now.

Hi, A process control error on our side caused some failed applications to only launch 1 web dino, which will appear as "inactive", although they were actually crashed. This means that the broken dyno has never been restarted, which leads to the failure of subsequent requests. We have identified these issues and are implementing the fix. If your application still does not respond, try restarting it with the heroku reload command. Please let us know if you need more help. Thanks Heroku Support