Geek Answers Handbook

Correct pool size of activerecord connections with sidekiq and postgres for multiple third-party processes?

I run 7 third-party processes (currency set to 40), plus a passenger web server connecting to the postgres database. The Rails pool setting is set to 100, and the post_res max_connections setting is also 100 by default.

I just added a new work class where each job executes several postgres requests, and I started getting this error on many sidekiq sites, and sometimes on my web server: PG::ConnectionBad: FATAL: remaining connection slots are reserved for non-replication superuser connections

I tried increasing max_connections postgres to 200, and the error still occurs. Then I tried to reduce the setting of the activerecord pool to 25 (25 connections for each process = 200 shared connections), believing that I can start to receive database connection timeout errors, but at least it will stop the "missing remaining connection slots" errors "

But I still get the error remaining connection slots are reserved.

A smarter way to deal with this problem could be to download important postgres data, which I continue to reuse in redis, and then access it from redis - which, animatedly, plays much better and nicer with sidekiq. But even when I do this, I would like to understand what happens here with postgres connections:

  • Am I probably leaking connections, and is that what I should be managing sidekiq?

(see Disabling an ActiveRecord Connection Until the End of a Sidekiq Job )

  • Should I look for more obscure things like lock / parsing issues or thread issues with the PG driver?

(see https://github.com/mperham/sidekiq/issues/594 . I think I use ActiveRecord quite simply without any obscure or abnormal logic for the rails application ..)

  • Or maybe I just don’t understand how to configure the ActiveRecord pool and postgres max_connection work together ...?
+4
source share
1 answer

, , , , , .

, , , , sidekiq?

, . Sidekiq , . , , , , , , tl; dr: Sidekiq , .

, / PG?

, , - .

, , , ActiveRecord max_connection postgres ...?

, , , , max_connections sidekiq:

DB = sidekiq concurrency

* = postgres max_connections/total sidekiq (+ -)

* , , , , 95% postgres , max_connections, .

:

Ubuntu vm.overcommit_memory 1 redis, .

, OOM ( ) Killer, . , postgres , kill OOM Killer.

postgres, sidekiq . - , . OOM, , sidekiqs, postgres.

+6

More articles:


All Articles