Geek Answers Handbook

Mongodb server sockets closed on replSet

We have a problem with mongodb sockets closing on us when we make a large volume of connections in parallel.

Here's the test script:

var mongodb = require("mongodb"); var async = require("async"); mongodb.MongoClient.connect("mongodb://mongo-dev1:27017/test", function(err, db) { if (err) { throw err; } var calls = []; var col = db.collection("test"); var count = 10000; for(var i = 0; i < count; i++) { (function(i) { calls.push(function(cb) { console.time("update_" + i); col.update({ i : i }, { i : i }, { upsert : true }, function(err) { console.timeEnd("update_" + i); cb(err); }); }); })(i); } async.parallel(calls, function(err) { if (err) { throw err; } console.log("done"); }); });

If I run this script, it will complete the error with the following MongoError: server mongo-dev1:27017 sockets closed error MongoError: server mongo-dev1:27017 sockets closed

The output of the magazine from mongodb itself

SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.1.111:53556]

I can’t understand what mechanism causes the socket to close. I believe the Node side of the equation hangs because my moment of events shows a Node close event that occurs milliseconds before a SocketException in mongodb logs. I went to the mongodb package in mongodb-core and did some console.log , and the initiator for the event is at TCP.close (net.js:485:12) . This tells me that the socket itself is closing. Based on this, it really seems that linux itself is closing the mongoDB socket or host server and that it is not running Node or MongoDB. I'm not sure how to prove it.

Here is the first set of options that I reviewed but excluded:

  • Socket timeout. If it was a timeout, the error message is different, I checked this by passing the socketTimeoutMS parameter when building the connection. If I miss something small, I will get a timeout error.

  • MongoDB disconnected from connections. If I track connections on Mangodb replet using db.serverStatus().connections , I still have many connections available.

  • This behavior is not replicated when I communicate with a non-replica localhost. It could be a localhost issue, or it could be a replica issue.

  • If I change the parallel with a parallel limit of 100, it ends without problems. Since Node uses the connection pool, whether I sent 1000 in parallel or 100 at the same time in parallel, it should equal the volume of MongoDB traffic, because all of them are forced to enter the same 10 sockets. This helps me determine that this is a Node problem.

Using Node 10, Node 12, and MongoDB 2.6

+4
source share
2 answers

I had the same problem under heavy load, however, when transcoding to the mongodb driver, I found out that the default values ​​for socketTimeout and connectTimeout are set to 30000 ms.

Raising them, I solved the problem:

 mongodb://m1.url.xyz:27017,m2.url.xyz:27017/test?replicaSet=myset&connectTimeoutMS=300000&socketTimeoutMS=300000&readPreference=secondary

(Make sure ulimit and net.ipv4.tcp_keepalive_time settings are optimized for mongodb) http://docs.mongodb.org/manual/faq/diagnostics/

+2
source

I had the same problem when connecting to a replica set over the network and quickly releasing a large number of queries in a database script initialization.

When connecting to the same mongodb instance on the same computer, this was good, but over the network with a minimum delay of 2-3 ms to the replica set, this would cause closed socket errors.

I also changed the save time on both computers to 120 in the registry. MongoDB 3.0.6 server is located on Windows Server 2012, the application computer is on Windows 10.

Then I tried and had no effect (restarted the mongod services in the replica set first)

Then I added the ?connectTimeoutMS=120000&socketTimeoutMS=120000 part to the query string, and I had success over the network with a set of replicas.

When I looked at the documentation for the parameters, I found that the default timeouts for one instance server are 30,000 ms, but for the default replica set, 0ms

in: http://mongodb.imtqy.com/node-mongodb-native/2.1/reference/connecting/connection-settings/

see replication level options

socketOptions.connectTimeoutMS {Number, default: 0} TCP Connection timeout setting. socketOptions.socketTimeoutMS {Number, default: 0} TCP Socket timeout setting.

This is the documentation for the node.js. driver I could not find mongoose docs information on timeouts for sockets and connections for only 6-7 other options. I am currently assuming that mongoose will pass parameters that you specify verbatim, and not a whitelist of only certain options. then you can specify timeouts in the options object instead of the connection string.

+1
source

More articles:


All Articles