Mongo DB Invariant Failure

Question

Our DB + - 400Gb stops on our one server.

From the magazines:

2015-07-07T09:09:51.072+0200 I STORAGE [conn10] _getOpenFile() invalid file index requested 8388701 2015-07-07T09:09:51.072+0200 I - [conn10] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201 2015-07-07T09:09:51.082+0200 I CONTROL [conn10] 

Any idea what I should start looking for? Storage problem?

Answer 1

I just answer this question if some people again make the same non-technical mistake:

I tried scp all the files in the /data/db directory on the server. Since there are many files (from dbname.1 to dbname.55 , about 100 GB), it was interrupted in the middle (the last successful dbname.22 file), and I restarted and loaded dbname.23 in dbname.55 . And when I ran queries in the mongo client, it worked in some cases and failed for some others displaying the error message in the same way as in the question. I thought it could be a file that was damaged in file transfer, but the md5 check was ok. Only after I completed the md5 check for a long time did I find the reason.

It turned out that scp loads dbname.21 into dbname.29 after loading dbname.2 , so dbname.3 before dbname.9 never loaded onto the server. I am going to download them and this should solve the problem.

Answer 2

I came across a variant of this today. Mysteriously, one of my data files disappeared (or didn’t make it when switching from another server). None of the restore / restore procedures will work unless you complete the same error that you indicated. Fortunately, I have a separate mongod that has a collection with the same name, since a cheap one hacked, I copied the (true, incorrect) data file to another server, and although I knew that I would not receive any data, the recovery tools (e.g. mongod --repair ) were able to work on their magic, but, as expected, they recovered some data from the file I copied, so I had to weed out some documents. Fortunately, it was the file "mycollection.1", which is only 128 MB.

I do not think this applies in your case, since the index of the missing data file referred to in the log is ridiculously high. Your journal essentially says that it cannot find /data/dbname/mycollection.8388701 . You said that your data set is only 400 GB, so an index that is high just doesn't make sense. You should only have about 200 data files, since by default most of them are 2 GB. What is the result of db.stats() (specifically the fileSize attribute)?

This mongolab entry helped me figure out the structure of the data file.

My advice is where you should start looking:

• run the db.stats() command to get an idea of ​​how big your data on disk actually is.
• Does it make sense that your server is looking for a crazy high index data file? If not, the problem is not with storage, but with the extents and metadata of your collection / database.
• Do your repair tools work? If you have at least enough free disk space as the size of your data set (disk), try the mongod --repair or db.repairDatabase() tools to start the repair. I assume this will not work as my recovery attempts crashed with the same invalid file index requested error.
• Try to copy the “bad” file, as I did, which roughly corresponds to what the missing file will look like (bearing in mind that the size of the data file files is not the same, make every effort to combine it and try to repair). If this works, your data files will be cleared (but it takes up a lot of disk space).

We hope this helps you in the right direction.