How to find if I have a memory leak

Question

I wrote a crunch algorithm. The idea is this:

• Small main programs require very little memory (starts at 2 MB).
• Then, in a loop, it calls a function that needs quite some memory (about 100 MB), which must be released at the end of the function. To understand what is happening, the function is now always called with the same parameters.

It seems that the program is slowly eating memory, so I suspect a memory leak. I tried the Sanitizer address from Clang and the Pointer Checker from Intel, but they didn't find anything.

Now I look at the memory consumption in my activity monitor (I start OSX, but I get the same memory usage from the Unix "top" command), and just before calling a large function, the program takes 2 MB. When the function starts, the program takes 120 MB. The strange thing is that when the program ends with a large function and returns inside the loop, now it takes 37 MB! Then, when it returns to a large function, it takes 130 MB. Again, returning to the loop, it takes 36 MB, and then in a large function it takes 140 MB ...

Therefore, he drifts slowly, but not with a regular pattern. How can I trust the use of memory in the "top"?

Can memory fragmentation increase memory usage without memory leak?

---

I run the program in one night, and here is the data I get:

• In the first cycle, the program takes 150 MB
• 2 hours after 68 cycles, the program takes 220 MB
• After one night and 394 cycles, the program takes 480 MB

Thus, it seems that the function that allocates and frees memory (about 120 MB) seems to “leak” 1 MB every time it is called.

Answer 1

First, make sure that for a long period of time (for example, if one iteration takes a minute, perform a couple of hours), growth continues. If the growth of asymptotes then does not occur. Then I will try valgrind . Then, if this does not help, you will have to binary search for your code: Comment out the bit until the growth stops. I would start by completely removing the MKL library (leave the stubs if you want) and see what happens. Then change the vector to std::vector to see if this helps. After that you will have to use your opinion.

Answer 2

I think I found the culprit: MKL (latest version to date). I use Pardiso, and the following example is very slow: about 0.1 MB every 13 seconds, resulting in 280 MB overnight. These are the numbers that I get from modeling.

If you want to try, you can compile it with:

 icpc -std=c++11 pardiso-leak.cpp -o main -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl -lpthread -lm 

Thank you all for your help. I reported an error for Intel.

 #include <iostream> #include <vector> #include "mkl_pardiso.h" #include "mkl_types.h" int main (int argc, char const *argv[]) { const auto n = std::size_t{1000}; auto m = MKL_INT{n * n}; auto values = std::vector<double>(); auto column = std::vector<MKL_INT>(); auto row = std::vector<MKL_INT>(); row.push_back(1); for(std::size_t j = 0; j < n; ++j) { column.push_back(j + 1); values.push_back(1.0); column.push_back(j + n + 1); values.push_back(0.1); row.push_back(column.size() + 1); } for(std::size_t i = 1; i < n - 1; ++i) { for(std::size_t j = 0; j < n; ++j) { column.push_back(n * i + j - n + 1); values.push_back(0.1); column.push_back(n * i + j + 1); values.push_back(1.0); column.push_back(n * i + j + n + 1); values.push_back(0.1); row.push_back(column.size() + 1); } } for(std::size_t j = 0; j < n; ++j) { column.push_back((n - 1) * n + j - n + 1); values.push_back(0.1); column.push_back((n - 1) * n + j + 1); values.push_back(1.0); row.push_back(column.size() + 1); } auto y = std::vector<double>(m, 1.0); auto x = std::vector<double>(m, 0.0); auto pardiso_nrhs = MKL_INT{1}; auto pardiso_max_fact = MKL_INT{1}; auto pardiso_mnum = MKL_INT{1}; auto pardiso_mtype = MKL_INT{11}; auto pardiso_msglvl = MKL_INT{0}; MKL_INT pardiso_iparm[64]; for (int i = 0; i < 64; ++i) { pardiso_iparm[i] = 0; } pardiso_iparm[0] = 1; pardiso_iparm[1] = 2; pardiso_iparm[3] = 0; pardiso_iparm[4] = 0; pardiso_iparm[5] = 0; pardiso_iparm[7] = 0; pardiso_iparm[8] = 0; pardiso_iparm[9] = 13; pardiso_iparm[10] = 1; pardiso_iparm[11] = 0; pardiso_iparm[12] = 1; pardiso_iparm[17] = -1; pardiso_iparm[18] = 0; pardiso_iparm[20] = 0; pardiso_iparm[23] = 1; pardiso_iparm[24] = 0; pardiso_iparm[26] = 0; pardiso_iparm[27] = 0; pardiso_iparm[30] = 0; pardiso_iparm[31] = 0; pardiso_iparm[32] = 0; pardiso_iparm[33] = 0; pardiso_iparm[34] = 0; pardiso_iparm[59] = 0; pardiso_iparm[60] = 0; pardiso_iparm[61] = 0; pardiso_iparm[62] = 0; pardiso_iparm[63] = 0; void* pardiso_pt[64]; for (int i = 0; i < 64; ++i) { pardiso_pt[i] = nullptr; } auto error = MKL_INT{0}; auto phase = MKL_INT{11}; MKL_INT i_dummy; double d_dummy; PARDISO(pardiso_pt, &pardiso_max_fact, &pardiso_mnum, &pardiso_mtype, &phase, &m, values.data(), row.data(), column.data(), &i_dummy, &pardiso_nrhs, pardiso_iparm, &pardiso_msglvl, &d_dummy, &d_dummy, &error); phase = 22; PARDISO(pardiso_pt, &pardiso_max_fact, &pardiso_mnum, &pardiso_mtype, &phase, &m, values.data(), row.data(), column.data(), &i_dummy, &pardiso_nrhs, pardiso_iparm, &pardiso_msglvl, &d_dummy, &d_dummy, &error); phase = 33; for(size_t i = 0; i < 10000; ++i) { std::cout << "i = " << i << std::endl; PARDISO(pardiso_pt, &pardiso_max_fact, &pardiso_mnum, &pardiso_mtype, &phase, &m, values.data(), row.data(), column.data(), &i_dummy, &pardiso_nrhs, pardiso_iparm, &pardiso_msglvl, y.data(), x.data(), &error); } phase = -1; PARDISO(pardiso_pt, &pardiso_max_fact, &pardiso_mnum, &pardiso_mtype, &phase, &m, values.data(), row.data(), column.data(), &i_dummy, &pardiso_nrhs, pardiso_iparm, &pardiso_msglvl, &d_dummy, &d_dummy, &error); return 0; }