Mongo database takes up much more disk space than it should

Question

In my mongo database, I have one collection limited to 5 GB, one at 10 MB and several without limits. None of the unrestricted contains more than 20 small documents.

After a long (4 hours) stress test, which is recorded in only 5 GB, the database collection uses 18 GB.

This is what my db.stats says (values ​​in MB):

data-db:PRIMARY> db.stats(1024*1024) { "db" : "data", "collections" : 9, "objects" : 8723395, "avgObjSize" : 208.8405255064112, "dataSize" : 1737, "storageSize" : 5130, "numExtents" : 12, "indexes" : 19, "indexSize" : 2534, "fileSize" : 18423, "nsSizeMB" : 16, "ok" : 1 } 

And this is 5 GB of collection statistics (values ​​in MB):

 data-db:PRIMARY> db.sms_message_event.stats(1024*1024) { "ns" : "data.sms_message_event", "count" : 8723300, "size" : 1737, "avgObjSize" : 0.00019912189194456226, "storageSize" : 5120, "numExtents" : 3, "nindexes" : 6, "lastExtentSize" : 1026, "paddingFactor" : 1, "systemFlags" : 1, "userFlags" : 0, "totalIndexSize" : 2534, "indexSizes" : { "_id_" : 395, "t_1_when_-1" : 475, "smsc_message_id_1" : 185, "user_id_1_t_1_when_1" : 481, "message_id_1" : 318, "virtual_number_recipient_when_index" : 678 }, "capped" : true, "max" : 2147483647, "ok" : 1 } 

So why is fileSize so much bigger than storageSize? I can’t even start repairDatabase (), but I tried compact () for each collection with no limits, no result. In fact, this was expected when db was clean before the stress test. I mean, the files were deleted, and not only the collections fell.

From the logs, I see that during the stress test additional data files were created within ~ 1 hour.

Some magazines: http://pastie.org/private/t8u9caxstafbjdybgwtsfw

UPDATE: After one more night and another run of 4 stress tests, these are 28GBs :(

 data-db:PRIMARY> db.stats(1024*1024) { "db" : "data", "collections" : 9, "objects" : 8724995, "avgObjSize" : 208.840894006243, "dataSize" : 1737, "storageSize" : 5130, "numExtents" : 12, "indexes" : 19, "indexSize" : 2590, "fileSize" : 28658, "nsSizeMB" : 16, "ok" : 1 } 

Answer 1

This is due to an error in MongoDB when reusing the space allocated to limited collections. It has been registered as SERVER-9489 and will be streamlined and hopefully fixed soon.

How can you continue to run stress tests without running out of disk space by deleting the test DB directory after testing is complete, and then create a new one when you start a new test (this assumes that you do not need to reuse the same data). If you need the same data, you can use mongodump to keep it from run to run, although there may be other simpler options that depend on your exact use.