I have a rails application (4.1.5) supported by the postgres database (9.2.13), using pg gem (0.17.1), a unicorn server (1.1.0) with two worker processes.
The rails app launches jobs using sidekiq (2.17.7)
At some point, postgres db went into recovery mode. The following error was caused by several tasks:
PG::ConnectionBad:FATAL: the database system is in recovery mode FATAL: the database system is in recovery mode
The database returned, but the jobs continued to cause the following two errors:
PG::Error:invalid encoding name: utf8 /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:908:in `set_client_encoding' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:908:in `configure_connection' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `reconnect!' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_adapter.rb:313:in `verify!' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:453:in `block in checkout_and_verify' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:82:in `run_callbacks'
and
ActiveRecord::ConnectionTimeoutError:could not obtain a database connection within 5.000 seconds (waited 5.000 seconds) /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll' /home/ruby/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize' /home/ruby/data42/shared/bundle/ruby/2.2.0/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
It seems to me that the rails notice that the connection is inactive and is trying to connect to reset. In the active postgresql_adapter.rb record, the following method is called:
I assume that connection.reset does not actually work, so when gg gg is sent to set the encoding (the first part of the configure_connection method), it masks the fact that the connection is missing, throwing a specific encoding error.
Below is the method from pg gem (.17.1) ext / pg_connection.c / 2804
static VALUE pgconn_set_client_encoding(VALUE self, VALUE str) { PGconn *conn = pg_get_pgconn( self ); Check_Type(str, T_STRING); if ( (PQsetClientEncoding(conn, StringValuePtr(str))) == -1 ) { rb_raise(rb_ePGerror, "invalid encoding name: %s",StringValuePtr(str)); } return Qnil; }
So, if these guesses are correct, why does the reset connection not work?
Restarting the application correctly restores the connection to the database, but I hope for a non-manual solution to this problem.